How to Host Headscale on a Linux Server with Podman Quadlets (Part 2)

Headscale

This is part 2 of a two-part series. I’m assuming you already have traefik up and running on your cloud VM and are ready to get headscale setup for an awesome, access-anywhere homelab that only you control.

If you haven’t already, consider checking out Part 1 of this blog series for some tips and tricks regarding setup of an openSUSE VPS and getting Traefik up and running with socket activation instead of forwarding ports, all using podman!

The Headscale Container

We’ll be deploying headscale in podman, via a systemd quadlet, just like we did with traefik. I’m assuming you’re at least somewhat familiar with this methodology and any troubleshooting steps necessary, based on the content in Part 1.

We’re first going to create the .container file and a couple .volume files for container storage. In this case, I chose not to create a specific network for headscale, as it needs to share a network namespace with traefik in order for the isolated reverse proxy action to work. If this was a service that required a separate backend or database, I’d give it its own network and only allow traefik to talk to the frontend, but that’s not the case here.

cd back over to your friendly quadlet directory: /etc/containers/systemd and create a file for our headscale container: headscale.container:

[Unit]
Description=Headscale Container
After=network-online.target traefik.service
Requires=traefik.service

[Container]
ContainerName=headscale
Exec=serve
HostName=headscale
Image=ghcr.io/juanfont/headscale:stable
Network=traefik.network
Volume=headscale-config.volume:/etc/headscale:Z
Volume=headscale-data.volume:/var/lib/headscale:Z
Volume=/etc/localtime:/etc/localtime:ro
AutoUpdate=registry

[Service]
Restart=always

[Install]
WantedBy=default.target

You should recognize a lot of these options likely, and be able to extrapolate what we’re doing here. As a reminder, this is very tightly based on the community docker documentation for headscale deployment, as highlighted in the documentation. In this case though, rather than doing container orchestration with docker-compose, we’re opting for podman quadlets to provide systemd service units.

Do you see the two .volume references in the container file? We’ll create those next.

The Headscale Volumes

In the same working directory, we’re going to define two very rudimentary storage volumes for our headscale container. One will store the state (/var/lib/headscale) and one will store our config (/etc/headscale).

First we’ll create the headscale-config.volume file:

[Volume]

and finally the headscale-data.volume file:

[Volume]

They look awfully similar, eh? All we need to do is provide systemd a method of defining and referencing these volumes by their names. There’s nothing special about their content. It’s just going to create directories in /var/lib/containers/storage/volumes.

Now that we have these manifests defined, do execute your trusty reload command to have systemd reread unit files and build the necessary volumes:

sudo systemctl daemon-reload

Now we’ll start the newly created units to define our volumes so we have a place to put our config file in a moment…

sudo systemctl start headscale-config-volume.service headscale-data-volume.service

You should now have the two directories created under /var/lib/containers/storage/volumes/systemd-headscale-config and /var/lib/containers/storage/volumes/systemd-headscale-data.

First cd over to /var/lib/containers/storage/volumes/systemd-headscale-config/_data/ and we’re going to create our headscale config file.

The config file I’m about to show you is based on the example configuration in the headscale GitHub repo. Look there if you want to see template values and my “source of truth” for this file.

Create a file called config.yaml with the following contents:

database:
  sqlite:
    path: /var/lib/headscale/db.sqlite # This directory needs to match a volume definition for your container
    write_ahead_log: true
    wal_autocheckpoint: 1000
  type: sqlite

derp:
  auto_update_enable: true
  server:
    region_code: "headscale"
    region_name: "Headscale Embedded DERP"
    enabled: true
    private_key_path: /etc/headscale/derp_server_private.key
    region_id: 999
    stun_listen_addr: "0.0.0.0:3478" # Notice how this port matches our DERP entrypoint and socket defined for Traefik?
    automatically_add_embedded_derp_region: true
    ipv4: 123.456.789.000 # Subsitute your server's public, routable IP address here.
    ipv6: 1234:5678:90ab:cdef:ff:ff # Subsitute your server's public, routable IPv6 address here, or comment out if you don't have one.
  update_frequency: 24h
  urls: []
  paths:
    - /etc/headscale/derp.yaml # This directory should also match your volume definition for config

disable_check_updates: true

dns:
  base_domain: tailnet.mydomain.tld
  magic_dns: true
  nameservers:
    global:
      - 100.64.7.177 # I run a tailscale client on my home OPNSense firewall and have chosen to use it as DNS here. You can use your own instance, or a public DNS server if you wish.
      - 1.2.3.4
  override_local_dns: true
  search_domains:
    - yourHomeDomain.tld # You can also define this if you're using a private DNS resolver like I am on OPNSense, otherwise comment out the entire key
ephemeral_node_inactivity_timeout: 30m

prefixes:
  v4: 100.64.7.0/24 # Choose a prefix here. It MUST be in the CGNAT range though. # IPv4: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#L33
  v6: fd7a:115c:a1e0::/48 # Same thing here: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#LL81C52-L81C71
  allocation: random

listen_addr: 0.0.0.0:8080

log:
  format: text
  level: info

logtail:
  enabled: false

metrics_listen_addr: 0.0.0.0:9090

noise:
  private_key_path: /etc/headscale/noisekey

## We're not going to use OIDC in this tutorial, but if you run an IdP (AD or Keycloak or Authentik, for example) you can set that up here.
#oidc:
#  allowed_domains: []
#  allowed_users: []
#  client_id: ''
#  client_secret_path: null
#  extra_params: {}
#  issuer: ''
#  scope:
#  - openid
#  - profile
#  - email
#  strip_email_domain: true

policy:
  mode: file
  path: /etc/headscale/acls.json # This should match a container file volume definition

private_key_path: /etc/headscale/privkey

server_url: https://tailscale.domain.tld

## We'll comment out the TLS definition here as Traefik is going to terminate TLS for us
#tls_cert_path: null
#tls_key_path: null
#tls_letsencrypt_cache_dir: /var/lib/headscale/.cache
#tls_letsencrypt_challenge_type: HTTP-01
#tls_letsencrypt_hostname: ''
#tls_letsencrypt_listen: null

unix_socket: /var/run/headscale/headscale.sock
unix_socket_permissions: 0770

Okay, so we’ve got a fully fledged configuration file. It references a few things in our Volumes that we need to handle. First we’ll address ACLs.

Headscale ACLs

Tailscale has a very neat permission system that headscale implements 1-to-1. We’re in the business of security here, so we’ll create some ACLs to limit what certain users/groups can access. Create a file called acls.json with something similar to the following:

{
  // groups are collections of users having a common scope. A user can be in multiple groups
  // groups cannot be composed of groups
  "groups": {
    "group:admin": ["starkzarn"],
    "group:roaming": ["starkzarn", "user2"],
    "group:opnsense": ["opnsense"],
    "group:needDNS": ["sam", "user2", "tag:dns"]
  },
  // tagOwners in tailscale is an association between a TAG and the people allowed to set this TAG on a server.
  // This is documented [here](https://tailscale.com/kb/1068/acl-tags//defining-a-tag)
  // and explained [here](https://tailscale.com/blog/rbac-like-it-was-meant-to-be/)
  "tagOwners": {
    // the administrators can add servers in production
    "tag:workstation": ["group:admin"],
    "tag:privileged": ["group:admin"],
    "tag:phones": ["group:admin"],
    "tag:home": ["group:admin"],
    "tag:container": ["group:admin"],
    "tag:dns": ["group:admin"]
  },
  "acls": [
    // admin access to all servers
    {
      "action": "accept",
      "src": ["group:admin"],
      "dst": [
        "tag:workstation:*",
        "tag:privileged:*",
        "tag:phones:*",
        "tag:home:*",
        "tag:container:*",
        "tag:dns:*",
        "10.0.0.0/8:*",
        "192.168.0.0/16:*",
        "opnsense:0" // Defining "port 0" is a method of allowing a user to relay THROUGH a node without having direct access to it -- i.e. if you're using it as a subnet router
      ]
    },

    {
      "action": "accept",
      "src": ["group:roaming"],
      "dst": [
        "192.168.99.100/32:443",
        "192.168.99.100/32:80",
        "tag:home:80,443",
        "opnsense:0"
      ]
    },

    {
      "action": "accept",
      "proto": "udp",
      "src": ["group:needDNS"],
      "dst": ["192.168.99.1/32:53", "192.168.6.1/32:53", "opnsense:53"]
    },

    {
      "action": "accept",
      "src": ["tag:dns"],
      "dst": ["1.2.3.4:8443", "opnsense:0", "opnsense:8443"]
    },

    // We still have to allow internal users communications since nothing guarantees that each user have
    // their own users.
    { "action": "accept", "src": ["starkzarn"], "dst": ["starkzarn:*"] },
    { "action": "accept", "src": ["user2"], "dst": ["user2:*"] },
    { "action": "accept", "src": ["opnsense"], "dst": ["opnsense:*"] },
    { "action": "accept", "src": ["tag:dns"], "dst": ["tag:dns:*"] },
    {
      "action": "accept",
      "src": ["group:roaming"],
      "dst": ["autogroup:internet:*"]
    } // autogroup:internet is a destination for general internet that isn't explicitly defined RFC1918 addresses. This is necessary if you plan on running an exit node.
  ]
}

You’ll probably end up tweaking this file several times as you change services and add users. A general rule of thumb is only to assign permissions to groups and tags by functional need, then assign users to those groups as necessary.

DERP Config

Okay, it’s time for DERP… First of all what is DERP?

What is DERP?

As always, I’ll reference the authoritative source here: the Tailscale DERP documentation.

The TL;DR though is that Designated Encrypted Relay for Packets (DERP) servers work as method to achieve the NAT holepunching that I mentioned in Part 1. DERP servers provide a public endpoint that a client can negotiate a stateful connection with, and then hand off that connection to Tailscale so that the underlying wireguard client has a publicly accessible port (an ephemeral port via the stateful connection) to use for the mesh VPN connection. Headscale is cool because it provides us a method of running our own DERP server, which is the ultimate reason that we’re running it, rather than using Tailscale’s own servers. This is the pinch point that we’re replacing with our own self-hosted infrastructure.

Assuming you’re still in the /var/lib/containers/storage/volumes/systemd-headscale-config/_data/ directory, create a file called derp.yaml with the following contents:

regions:
  999:
    regionid: 999
    regioncode: headscale
    regionname: Headscale Region
    nodes:
      - name: 999
        regionid: 999
        hostname: yourtailscalehost.domain.tld
        ipv4: 123.456.789.000 # Subsitute your server's public, routable IP address here.
        ipv6: 1234:5678:90ab:cdef:ff:ff # Subsitute your server's public, routable IPv6 address here, or comment out if you don't have one.
        stunport: 0
        stunonly: false
        derpport: 0

This file is quick and simple. It’s referenced in the headscale config file and provides a mapping of derp parmeters for the integrated DERP server in headscale. This is going to give us the desired configuration of our controlplane only offering one DERP server — the one that we control. Make sure you make the necessary substitutions for IP addresses and domain.

Start-up

The other files referenced are cryptographic key files and will be autogenerated by headscale on first run. So… let’s do that; time to run headscale!

If you haven’t already, tell systemd to reread your unit files: sudo systemctl daemon-reload, and finally, let’s start headscale!

sudo systemctl enable --now headscale

With any luck your headscale container should be running! Check logs if you need to, again with journalctl here.

sudo journalctl -u headscale -n50 -f

Traefik Routing for Headscale

Okay, so if you’ve come this far, you have a working traefik instance, with your first router setup (this was the dashboard subdomain we setup in Part 1); and you have a working headscale instance. Now we’re going to plug them in and make headscale accessible from the web!

Just like we did for our dashboard, we’re going to create a dynamic configuration stub for this service. cd back over to your traefik-config volume:

cd /var/lib/containers/storage/volumes/systemd-traefik-config/_data/conf.d

Now we’re going to create a new file called headscale.yml:

http:
  services:
    headscale:
      loadBalancer:
        servers:
          - url: "http://headscale:8080/"
    headscale-metrics:
      loadBalancer:
        servers:
          - url: "http://headscale:9090/"

  middlewares:
    cors:
      headers:
        accessControlAllowHeaders: "*"
        accessControlAllowMethods:
          - "GET"
          - "POST"
          - "PUT"
        accessControlAllowOriginList:
          - "https://tailscale.yourDomain.tld"
        accessControlMaxAge: 100
        addVaryHeader: true

  routers:
    headscale:
      tls:
        options: modern
      rule: Host(`tailscale.yourDomain.tld`)
      middlewares:
        - default
        - cors
      service: headscale
      entrypoints:
        - https

    headscale-metrics:
      tls:
        options: modern
      rule: Host(`tailscale.yourDomain.tld`) && PathPrefix(`/metrics`)
      middlewares:
        - default
        - promAuth
      service: headscale-metrics
      entrypoints:
        - https

udp:
  services:
    derp:
      loadBalancer:
        servers:
          - address: "headscale:3478"

  routers:
    derp:
      service: derp
      entryPoints:
        - derp

There are a few things to unpack here…

1. First we have two major sections for HTTP and UDP traffic, respectively.
2. In our HTTP section we define three types of things: services, routers, and middlewares.
3. Our services define the “backend” where traefik will send requests that match whatever rule you define.
4. Our routers are the logic blocks behind our entrypoints and serve as a place to evaluate rules and match up requests to one of our services.

5. Middlewares are optional bits that plugin between a router and a service and do things like add headers.

So how does this apply to our file? Well, it should be relatively self-explanatory at this point, but specifically we’re setting up blocks to handle logic for headscale in general, specific requests for the /metrics endpoint, which is destined for a Prometheus exporter; and a special UDP router for DERP.

At this point, upon saving the file your traefik instance should immediately pick it up (thanks to the watch: True we setup in our static config for the file provider).

Testing

Okay, so headscale doesn’t exactly serve web content, but there is at least one special endpoint: /windows that serves installation instructions for a Windows host, so let’s try that! In your browser, navigate to https://tailscale.yourDomain.tld/windows and you should be beautifully greeted with this!

Using Your Custom Server For Your Client Devices

You made it through and have a working headscale instance that’s publicly accessible on the internet, and is running it’s own DERP server — all with automatic TLS termination via traefik. Nice. Now what?

The hard part is over obviously, but telling your client devices to use your headscale server does require a little hoop jumping. We’ll cover cases here for Linux, Android, and iOS.

Conclusion

You made it through my first two-part blog, and through how I’ve chosen to deploy headscale behind traefik. Thanks for sticking around. Feel free to leave comments to point out what I did wrong, what worked and didn’t for you, and what you’d like clarification (or less detail) on. I may very well make changes to the post due to it. This is meant to be a community thing, effectively my “public notes.” Join in the community with a GitHub account for comments below.

EDIT: If you want some bonus material, I have a follow-up post that covers setting up OIDC and two-factor authentication with Authelia!

Support

If you’ve enjoyed this, consider helping support the infrastructure to run it.

Buy me a coffee or three with the button below.