<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Traefik &#8211; hackanooga</title>
	<atom:link href="/category/traefik/feed/" rel="self" type="application/rss+xml" />
	<link>/</link>
	<description>Confessions of a homelab hacker</description>
	<lastBuildDate>Sat, 11 May 2024 13:44:01 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=6.6.2</generator>

<image>
	<url>/wp-content/uploads/2024/03/cropped-cropped-avatar-32x32.png</url>
	<title>Traefik &#8211; hackanooga</title>
	<link>/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Traefik 3.0 service discovery in Docker Swarm mode</title>
		<link>/traefik-3-0-service-discovery-in-docker-swarm-mode/</link>
		
		<dc:creator><![CDATA[mikeconrad]]></dc:creator>
		<pubDate>Sat, 11 May 2024 13:44:01 +0000</pubDate>
				<category><![CDATA[Ansible]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Docker]]></category>
		<category><![CDATA[Software Engineering]]></category>
		<category><![CDATA[Traefik]]></category>
		<category><![CDATA[Blog Post]]></category>
		<guid isPermaLink="false">/?p=564</guid>

					<description><![CDATA[I recently decided to set up a Docker swarm cluster for a project I was working on. If you aren&#8217;t familiar with Swarm mode, it is similar in some ways to k8s but with much less complexity and it is built into Docker. If you are looking for a fairly straightforward way to deploy containers [&#8230;]]]></description>
										<content:encoded><![CDATA[
<p>I recently decided to set up a Docker swarm cluster for a project I was working on.  If you aren&#8217;t familiar with Swarm mode, it is similar in some ways to k8s but with much less complexity and it is built into Docker.  If you are looking for a fairly straightforward way to deploy containers across a number of nodes without all the overhead of k8s it can be a good choice, however it isn&#8217;t a very popular or widespread solution these days.  </p>



<p>Anyway, I set up a VM scaling set in Azure with 10 Ubuntu 22.04 vms and wrote some Ansible scripts to automate the process of installing Docker on each machine as well as setting 3 up as swarm managers and the other 7 as worker nodes.  I ssh&#8217;d into the primary manager node and created a docker compose file for launching an observability stack.</p>



<p>Here is what that <code>docker-compose.yml</code> looks like:</p>



<pre class="wp-block-code"><code>---
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.88.0
    volumes:
      - /home/user/repo/common/devops/observability/otel-config.yaml:/etc/otel/config.yaml
      - /home/user/repo/log:/log/otel
    command: --config /etc/otel/config.yaml
    environment:
      JAEGER_ENDPOINT: 'tempo:4317'
      LOKI_ENDPOINT: 'http://loki:3100/loki/api/v1/push'
    ports:
      - '8889:8889' # Prometheus metrics exporter (scrape endpoint)
      - '13133:13133' # health_check extension
      - '55679:55679' # ZPages extension
    deploy:
      placement:
        constraints:
          - node.hostname==dockerswa2V8BY4
    networks:
      - traefik
  prometheus:
    container_name: prometheus
    image: prom/prometheus:v2.42.0
    volumes:
      - /home/user/repo/common/devops/observability/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - '9090:9090'
    deploy:
      placement:
        constraints:
          - node.hostname==dockerswa2V8BY4
    networks:
      - traefik
  loki:
    container_name: loki
    image: grafana/loki:2.7.4
    ports:
      - '3100:3100'
    networks:
      - traefik
  grafana:
    container_name: grafana
    image: grafana/grafana:9.4.3
    volumes:
      - /home/user/repo/common/devops/observability/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: 'false'
      GF_AUTH_ANONYMOUS_ORG_ROLE: 'Admin'
    expose:
      - '3000'
    labels:
      - traefik.constraint-label=traefik
      - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
      - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true
      - traefik.http.routers.grafana-http.rule=Host(`swarm-grafana.mydomain.com`)
      - traefik.http.routers.grafana-http.entrypoints=http
      - traefik.http.routers.grafana-http.middlewares=https-redirect
      # traefik-https the actual router using HTTPS
      # Uses the environment variable DOMAIN
      - traefik.http.routers.grafana-https.rule=Host(`swarm-grafana.mydomain.com`)
      - traefik.http.routers.grafana-https.entrypoints=https
      - traefik.http.routers.grafana-https.tls=true
      # Use the special Traefik service api@internal with the web UI/Dashboard
      - traefik.http.routers.grafana-https.service=grafana
      # Use the "le" (Let's Encrypt) resolver created below
      - traefik.http.routers.grafana-https.tls.certresolver=le
      # Enable HTTP Basic auth, using the middleware created above
      - traefik.http.services.grafana.loadbalancer.server.port=3000
    deploy:
      placement:
        constraints:
          - node.hostname==dockerswa2V8BY4
    networks:
      - traefik
  # Tempo runs as user 10001, and docker compose creates the volume as root.
  # As such, we need to chown the volume in order for Tempo to start correctly.
  init:
    image: &amp;tempoImage grafana/tempo:latest
    user: root
    entrypoint:
      - 'chown'
      - '10001:10001'
      - '/var/tempo'
    volumes:
      - /home/user/repo/tempo-data:/var/tempo
    deploy:
      placement:
        constraints:
          - node.hostname==dockerswa2V8BY4

  tempo:
    image: *tempoImage
    container_name: tempo
    command: &#91;'-config.file=/etc/tempo.yaml']
    volumes:
      - /home/user/repo/common/devops/observability/tempo.yaml:/etc/tempo.yaml
      - /home/user/repo/tempo-data:/var/tempo
    deploy:
      placement:
        constraints:
          - node.hostname==dockerswa2V8BY4
    ports:
      - '14268' # jaeger ingest
      - '3200' # tempo
      - '4317' # otlp grpc
      - '4318' # otlp http
      - '9411' # zipkin
    depends_on:
      - init
    networks:
      - traefik
networks:
  traefik:
    external: true
</code></pre>



<p>Pretty straightforward so I proceed to deploy it into the swarm</p>



<pre class="wp-block-code"><code>docker stack deploy -c docker-compose.yml observability</code></pre>



<p>Everything deploys properly but when I view the Traefik logs there is an issue with all the services except for the grafana service.  I get errors like this:</p>



<pre class="wp-block-code"><code>traefik_traefik.1.tm5iqb9x59on@dockerswa2V8BY4    | 2024-05-11T13:14:16Z ERR error="service \"observability-prometheus\" error: port is missing" container=observability-prometheus-37i852h4o36c23lzwuu9pvee9 providerName=swarm
</code></pre>



<p>It drove me crazy for about half a day or so.  I couldn&#8217;t find any reason why the grafana service worked as expected but none of the others did.  Part of my love/hate relationship with Traefik stems from the fact that configuration issues like this can be hard to track and debug.  Ultimately after lots of searching and banging my head against a wall I found the answer in the Traefik docs and thought I would share here for anyone else who might run into this issue.  Again, this solution is specific to Docker Swarm mode.</p>



<p></p>



<p><a href="https://doc.traefik.io/traefik/providers/swarm/#configuration-examples">https://doc.traefik.io/traefik/providers/swarm/#configuration-examples</a></p>



<p>Expand that first section and you will see the solution:</p>



<figure class="wp-block-image size-full is-resized"><img fetchpriority="high" decoding="async" width="634" height="181" src="/wp-content/uploads/2024/05/image.png" alt="" class="wp-image-565" style="width:720px;height:auto" srcset="/wp-content/uploads/2024/05/image.png 634w, /wp-content/uploads/2024/05/image-300x86.png 300w" sizes="(max-width: 634px) 100vw, 634px" /></figure>



<p>It turns out I just needed to update my <code>docker-compose.yml</code> and nest the labels under a deploy section, redeploy and everything was working as expected.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Traefik with Let’s Encrypt and Cloudflare (pt 2)</title>
		<link>/traefik-with-lets-encrypt-and-cloudflare-pt-2/</link>
		
		<dc:creator><![CDATA[mikeconrad]]></dc:creator>
		<pubDate>Thu, 15 Feb 2024 20:19:12 +0000</pubDate>
				<category><![CDATA[Automation]]></category>
		<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[Docker]]></category>
		<category><![CDATA[Traefik]]></category>
		<category><![CDATA[Blog Post]]></category>
		<guid isPermaLink="false">https://wordpress.hackanooga.com/?p=425</guid>

					<description><![CDATA[In this article we are gonna get into setting up Traefik to request dynamic certs from Lets Encrypt. I had a few issues getting this up and running and the documentation is a little fuzzy. In my case I decided to go with the DNS challenge route. Really the only reason I went with this [&#8230;]]]></description>
										<content:encoded><![CDATA[
<p>In this article we are gonna get into setting up Traefik to request dynamic certs from Lets Encrypt.  I had a few issues getting this up and running and the documentation is a little fuzzy.  In my case I decided to go with the DNS challenge route.  Really the only reason I went with this option is because I was having issues with the TLS and HTTP challenges.  Well as it turns out my issues didn&#8217;t have as much to do with my configuration as they did with my router.</p>



<p>Sometime in the past I had set up some special rules on my router to force all clients on my network to send DNS requests through a self hosted DNS server.  I did this to keep some of my &#8220;smart&#8221; devices from misbehaving by blocking there access to the outside world.  As it turns out some devices will ignore the DNS servers that you hand out via DHCP and will use their own instead.  That is of course unless you force DNS redirection but that is another post for another day.</p>



<p>Let&#8217;s revisit our current configuration:</p>



<pre class="wp-block-code"><code>version: '3'

services:
  reverse-proxy:
    # The official v2 Traefik docker image
    image: traefik:v2.11
    # Enables the web UI and tells Traefik to listen to docker
    command:
      - --api.insecure=true
      - --providers.docker=true
      - --providers.file.filename=/config.yml
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      # Set up LetsEncrypt
      - --certificatesresolvers.letsencrypt.acme.dnschallenge=true
      - --certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.letsencrypt.acme.email=mikeconrad@onmail.com
      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
      - --entryPoints.web.http.redirections.entryPoint.to=websecure
      - --entryPoints.web.http.redirections.entryPoint.scheme=https
      - --entryPoints.web.http.redirections.entrypoint.permanent=true
      - --log=true
      - --log.level=INFO
#      - '--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory'

    environment:
      - CF_DNS_API_TOKEN=${CF_DNS_API_TOKEN}
    ports:
      # The HTTP port
      - "80:80"
      - "443:443"
      # The Web UI (enabled by --api.insecure=true)
      - "8080:8080"
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./letsencrypt:/letsencrypt
      - ./volumes/traefik/logs:/logs
      - ./traefik/config.yml:/config.yml:ro
    networks:
      - traefik
  ots:
    image: luzifer/ots
    container_name: ots
    restart: always
    environment:
      # Optional, see "Customization" in README
      #CUSTOMIZE: '/etc/ots/customize.yaml'
      # See README for details
      REDIS_URL: redis://redis:6379/0
      # 168h = 1w
      SECRET_EXPIRY: "604800"
      # "mem" or "redis" (See README)
      STORAGE_TYPE: redis
    depends_on:
      - redis
    labels:
      - traefik.enable=true
      - traefik.http.routers.ots.rule=Host(`ots.hackanooga.com`)
      - traefik.http.routers.ots.entrypoints=websecure
      - traefik.http.routers.ots.tls=true
      - traefik.http.routers.ots.tls.certresolver=letsencrypt
    networks:
      - traefik
  redis:
    image: redis:alpine
    restart: always
    volumes:
      - ./redis-data:/data
    networks:
      - traefik
networks:
  traefik:
    external: true

</code></pre>



<p>Now that we have all of this in place there are a couple more things we need to do on the Cloudflare side:</p>



<h3 class="wp-block-heading">Step 1: Setup wildcard DNS entry</h3>



<p>This is pretty straightforward.  Follow the Cloudflare <a href="https://developers.cloudflare.com/dns/manage-dns-records/reference/wildcard-dns-records/" data-type="link" data-id="https://developers.cloudflare.com/dns/manage-dns-records/reference/wildcard-dns-records/">documentation</a> if you aren&#8217;t familiar with setting this up. </p>



<h3 class="wp-block-heading">Step 2: Create API Token</h3>



<p>This is where the Traefik documentation is a little lacking.  I had some issues getting this set up initially but ultimately found this <a href="https://go-acme.github.io/lego/dns/cloudflare/">documentation</a> which pointed me in the right direction.  In your Cloudflare account you will need to create an API token.  Navigate to the dashboard, go to your profile -> API Tokens and create new token.  It should have the following permissions:</p>



<pre class="wp-block-code"><code>Zone.Zone.Read
Zone.DNS.Edit</code></pre>



<figure class="wp-block-image size-full"><img decoding="async" width="1015" height="770" src="https://wordpress.hackanooga.com/wp-content/uploads/2024/03/image-9.png" alt="" class="wp-image-426" srcset="/wp-content/uploads/2024/03/image-9.png 1015w, /wp-content/uploads/2024/03/image-9-300x228.png 300w, /wp-content/uploads/2024/03/image-9-768x583.png 768w" sizes="(max-width: 1015px) 100vw, 1015px" /></figure>



<p>Also be sure to give it permission to access all zones in your account.  Now simply provide that token when starting up the stack and you should be good to go:</p>



<pre class="wp-block-code"><code>CF_DNS_API_TOKEN=&#91;redacted] docker compose up -d</code></pre>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Traefik with Let&#8217;s Encrypt and Cloudflare (pt 1)</title>
		<link>/traefik-with-lets-encrypt-and-cloudflare-pt-1/</link>
		
		<dc:creator><![CDATA[mikeconrad]]></dc:creator>
		<pubDate>Thu, 01 Feb 2024 19:35:00 +0000</pubDate>
				<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[Docker]]></category>
		<category><![CDATA[Self Hosted]]></category>
		<category><![CDATA[Traefik]]></category>
		<category><![CDATA[Blog Post]]></category>
		<guid isPermaLink="false">https://wordpress.hackanooga.com/?p=422</guid>

					<description><![CDATA[Recently I decided to rebuild one of my homelab servers. Previously I was using Nginx as my reverse proxy but I decided to switch to Traefik since I have been using it professionally for some time now. One of the reasons I like Traefik is that it is stupid simple to set up certificates and [&#8230;]]]></description>
										<content:encoded><![CDATA[
<p>Recently I decided to rebuild one of my homelab servers.  Previously I was using Nginx as my reverse proxy but I decided to switch to Traefik since I have been using it professionally for some time now.  One of the reasons I like Traefik is that it is stupid simple to set up certificates and when I am using it with Docker I don&#8217;t have to worry about a bunch of configuration files.  If you aren&#8217;t familiar with how Traefik works with Docker, here is a brief example of a <code>docker-compose.yaml</code></p>



<pre class="wp-block-code"><code>version: '3'

services:
  reverse-proxy:
    # The official v2 Traefik docker image
    image: traefik:v2.11
    # Enables the web UI and tells Traefik to listen to docker
    command:
      - --api.insecure=true
      - --providers.docker=true
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      # Set up LetsEncrypt
      - --certificatesresolvers.letsencrypt.acme.dnschallenge=true
      - --certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.letsencrypt.acme.email=user@example.com
      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
      # Redirect all http requests to https
      - --entryPoints.web.http.redirections.entryPoint.to=websecure
      - --entryPoints.web.http.redirections.entryPoint.scheme=https
      - --entryPoints.web.http.redirections.entrypoint.permanent=true
      - --log=true
      - --log.level=INFO
    # Needed to request certs via lets encrypt
    environment:
      - CF_DNS_API_TOKEN=&#91;redacted]
    ports:
      # The HTTP port
      - "80:80"
      - "443:443"
      # The Web UI (enabled by --api.insecure=true)
      - "8080:8080"
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
      # Used for storing letsencrypt certificates
      - ./letsencrypt:/letsencrypt
      - ./volumes/traefik/logs:/logs
    networks:
      - traefik
  ots:
    image: luzifer/ots
    container_name: ots
    restart: always
    environment:
      REDIS_URL: redis://redis:6379/0
      SECRET_EXPIRY: "604800"
      STORAGE_TYPE: redis
    depends_on:
      - redis
    labels:
      - traefik.enable=true
      - traefik.http.routers.ots.rule=Host(`ots.example.com`)
      - traefik.http.routers.ots.entrypoints=websecure
      - traefik.http.routers.ots.tls=true
      - traefik.http.routers.ots.tls.certresolver=letsencrypt
      - traefik.http.services.ots.loadbalancer.server.port=3000
    networks:
      - traefik
  redis:
    image: redis:alpine
    restart: always
    volumes:
      - ./redis-data:/data
    networks:
      - traefik
networks:
  traefik:
    external: true


</code></pre>



<p>In part one of this series I will be going over some of the basics of Traefik and how dynamic routing works.  If you want to skip to the good stuff and get everything configured with Cloudflare, you can skip to <a href="https://wordpress.hackanooga.com/traefik-with-lets-encrypt-and-cloudflare-pt-2/" data-type="post" data-id="425">part 2</a>.</p>



<p>This example set&#8217;s up the primary Traefik container which acts as the ingress controller as well as a  handy One Time Secret sharing <a href="https://github.com/Luzifer/ots">service</a> I use.  Traefik handles routing in Docker via labels.  For this to work properly the services that Traefik is trying to route to all need to be on the same Docker network.  For this example we created a network called traefik by running the following:</p>



<pre class="wp-block-code"><code>docker network create traefik
</code></pre>



<p>Let&#8217;s take a look at the labels we applied to the <code>ots</code> container a little closer:</p>



<pre class="wp-block-code"><code>    labels:
      - traefik.enable=true
      - traefik.http.routers.ots.rule=Host(`ots.example.com`)
      - traefik.http.routers.ots.entrypoints=websecure
      - traefik.http.routers.ots.tls=true
      - traefik.http.routers.ots.tls.certresolver=letsencrypt
      - traefik.http.services.ots.loadbalancer.server.port=3000</code></pre>



<p><code>traefik.enable=true</code> &#8211; This should be pretty self explanatory but it tells Traefik that we want it to know about this service.</p>



<p><code>traefik.http.routers.ots.rule=Host('ots.example.com') - This is where some of the magic comes in.</code>  Here we are defining a router called <code>ots</code>.  The name is arbitrary in that it doesn&#8217;t have to match the name of the service but for our example it does.  There are many rules that you can specify but the easiest for this example is host.  Basically we are saying that any request coming in for ots.example.com should be picked up by this router.  You can find more options for routers in the Traefik <a href="https://doc.traefik.io/traefik/routing/routers/">docs</a>.</p>



<p>&#8211; traefik.http.routers.ots.entrypoints=websecure<br>&#8211; traefik.http.routers.ots.tls=true<br>&#8211; traefik.http.routers.ots.tls.certresolver=letsencrypt<br><br>We are using these three labels to tell our router that we want it to use the websecure entrypoint, and that it should use the letsencrypt certresolver to grab it&#8217;s certificates.  websecure is an arbitrary name that we assigned to our :443 interface.  There are multiple ways to configure this, I choose to use the cli format in my traefik config:</p>



<p>&#8220;`</p>



<pre class="wp-block-code"><code>    command:
      - --api.insecure=true
      - --providers.docker=true
      # Our entrypoint names are arbitrary but these are convention.
      # The important part is the port binding that we associate.
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443

</code></pre>



<p>These last label is optional depending on your setup but it is important to understand as the documentation is a little fuzzy.</p>



<p>      &#8211; traefik.http.services.ots.loadbalancer.server.port=3000<br><br>Here&#8217;s how it works.  Suppose you have a container that exposes multiple ports.  Maybe one of those is a web ui and another is something that you don&#8217;t want exposed.  By default Traefik will try and guess which port to route requests to.  My understanding is that it will try and use the first exposed port.  However you can override this functionality by using the label above which will tell Traefik specifically which port you want to route to inside the container.<br><br>The service name is derived automatically from the definition in the docker compose file:</p>



<pre class="wp-block-code"><code><br>  ots: # This will become the service name<br>    image: luzifer/ots<br>    container_name: ots<br>    </code></pre>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
