The need for speed

After my previous, more technical, post on home networking this one is more of like an update on gear and how my approach to “homelabbing” in general has changed in the last year.

I use my homelab primarily to run services that I don’t have to pay for (other than in setup time and utility bills) and to experiment with new technologies in self-hosting and networking. I’ve always been interested in networking every since my sysadmin days at my university. Recently, some of the network has (finally?) been upgraded to 2.5Gb Ethernet, which does feel like it’s having a slow rollout in general. I mean, I bought a 2005 PowerBook G4 17” this week (to add to my worryingly large collection of PowerPC Macs) and it already had gigabit Ethernet twenty years ago! I guess for most people gigabit is enough, but with my NASs full of movi training data for my models and backups of multiple PCs it’s very good to go from ~125 MB/s to nearly 300MB/s. Sadly, home 2.5Gb gear is still expensive, so I just covered the “hot path” in my network which is basically between the NASs and my workstation, with the rest like girlfriend’s Mac Mini and living room Android set-top box still being on single gigabit.

Gear

I’m currently rocking 2 Synology NASs and 2 other smaller PCs (there, behind the IKEA lamp). This IKEA Kallax is my “rack” and the UPS is one I could find that is supported by the Synology OS via USB, which just means that when the power goes out, the server does a graceful shutdown. The UPS is a Cyberpower Value Pro VP1000ELCD that seems to work just fine, and gives me a 25 minute window on battery. I went with it because replacement batteries are easy to find, and had good reviews. Other gear in the picture:

• Synology DS1621+ with 32GB of RAM, 4x8TB and 2x6TB NAS HDDs with 2xSSDs for read-write cache
• Synology DS923+ with 16GB of RAM, 4x6TB NAS HDDS and SSDs for caching too
• Beelink ME mini with 12GB of RAM (standard config), 3x1TB SSDs and 1x 500GB SSD
• Beelink Mini S12 Pro with 16GB of RAM and a single 500GB SSD
• QNAP QSW-1108-T unmanaged 8-port 2.5Gb switch
• Tapo C200 camera (to keep an eye on the 3D printer right next to the rack)
• some LTO-5 tapes

Yes, I still have LTO tapes - the drive is from 2017 and it still works fine, and some of the tapes are from that time too and they still read and write just fine. The advantage of LTO drives is that they’re all made for enterprise customers so they’re super reliable. And yes, expensive too - but the media is cheap for what it does. I wrote a post on the drive a while back.

The QNAP switch is one of the two I bought to upgrade the network, along with a pair of 5-port ASUStor ASW205T, also unmanaged. They’re reliable and the cheapest options I could find. No VLANs or anything, but I don’t have them anyway - everything is shoved onto a /20 network, with separate subnets for different devices, IoT stuff, VMs, etc.

The Synologys are reliable, but the last gear I’ll buy from them giving some enshittification as of late. As for the Beelinks, they’re part of this new breed of MiniPCs with low-power CPUs that have sprung up everywhere in the last few years. I have another S12 Pro not on this picture that runs as a Proxmox host with Pi-Hole and OPNSense workloads. For a long time, I’d have problems connecting to the network interface I use to manage them remotely because it would just stop responding after a day or so. But it seems that issue has been fixed with Debian 13’s new kernel, as it’s been a week or so now and I can still access the Proxmox UI. The BIOS for these devices has too many options and updates are almost nowhere to be seen, so caveat emptor if you buy one of them. The ME Mini is a more recent model that I just couldn’t pass up - 6 NVME bays, 12GB of RAM, Intel N150 (basically a refresh of the N100) and 2x 2.5Gb NICs for around 250 bucks. Incredible value, so I snatched one up as soon as it came out for pre-order.

Usage

As for the use I give to them, the Synologys are basically for storage (with the smaller one long-term, important stuff like photos, home videos and so on) with the ME Mini being where I keep short-term media to watch that I don’t mind losing, and the S12 Pro starting to take over as the main workhorse for services - some of them still run on the big DS1621+, but I really want to keep compute and storage separated moving forward.

I used to run services on an Intel Mac Mini that was replaced with an M4 one, but it’s a bit of a hassle requiring a fork of the normal Debian kernel and having driver issues with some of the hardware, so now it’s retired for good. As a bonus, the S12 is less power-hungry too.

In keeping with the experimental nature of the homelab, the ME Mini is running ZimaOS which is a beginner-friendly distro that has some bugs that require some tweaks but overall is more simple and polished than something like OMV with Portainer - I just need it to run a Samba share to the living room set-top box and sabNzbd / qBitTorrent which it does quite well, with much greater performance than HDDs.

Since the goal is experimentation, I want every “node” in the homelab to stand on its own, so what I do is have different subdomains for my main domain X (ie n100.x.com minime.X.com) and have them all terminate on a special VM just running nginx on the Proxmox host that also runs OPNSense and Pi-Hole. I call it traffic-cop 🤓 and it has a very basic nginx config. In the main nginx.conf I just include /etc/nginx/stream.conf that contains:

stream {
  upstream https_minime {
    server 192.168.2.14:443;
  }

  upstream https_s12 {
    server 192.168.2.13:443;
  }

  map $ssl_preread_protocol $upstream {
    default $name;
    "TLSv1.3" $name;
    "TLSv1.2" $name;
    "TLSv1.1" $name;
    "TLSv1" $name;
  }

  map $ssl_preread_server_name $name {
    hostnames;

    minime.domain.tld https_minime;
    *.minime.domain.tld https_minime;

    s12.domain.tld https_s12;
    *.s12.domain.tld https_s12;

    default https_minime;
  }

  server {
    listen 443;
    proxy_pass $upstream;
    ssl_preread on;
  }
}

This allows me to manage the certificates on each individual node and not deal with cumbersome TLS termination issues, as nginx running on traffic-cop just snoops at the headers and forwards traffic to the correct node based on its host name. It doesn’t try to open it up, which would require having certificates everywhere. The downside is that I need a DNS A records for each node with name and *.name pointing to my home IP, but that’s all the setup I need when adding a new node to the network that will receive outside traffic. For this to work in Debian, you need to install the libnginx-mod-stream package and maybe nginx-extras too. More info on the nginx docs for ssl_preread. Since this is all the VM does, it can run with just 256MB of RAM and a fraction of a CPU core.

Services

I was using Dockge for services on the Synology, but moved to Dokploy now on the S12. Feels like Dokploy is still being worked on, and the cluster possibilities are interesting since it can use Docker Swarm. I’ve also looked at Dockman which shows promise, and ZaneOps which is a bit more complex than my preference. What I like about Dokploy is how easy it is to get started.

These tools make it a bit hard to reproduce the setup, though. I’m still hoping for some tool that would make managing a Kubernetes cluster easier, something like edka but free to experiment with more nodes. Just saying Kubernetes because it seems to be the only alternative to “plain” Docker/Swarm or something less free like Nomad. Unfortunately most of the homelabbing Kubernetes stuff all seems to focus on GitOps and “do this, then this” instead of actually explaining any of it, making it hard for me to engage. Maybe some day.

VPN wise, I do have an outside Wireguard server running on a Hetzner VPS, and setting up Split DNS is something I still have to work on - it should be simple, just returning different IPs based on the interface the query is coming from, so that local stuff still works when I’m on a VPN IP. I do wish a more “mesh” Wireguard thingy existed, so that you don’t have to add every peer to a Wireguard network’s nodes. Some do exist, but none seem to work too well.

Future projects

One annoyance of running something like 20+ services is logging in to all of them. Some like Decker don’t have a login, some others do but don’t support SSO and some other still like Outline (that I’m writing this post on) only support SSO in the form of OIDC. For those, Pocket-ID is great but it doesn’t cover the non-OIDC case. Maybe VoidAuth can cover those too? Would hate to lose passkey support, though.

I’d still like to try and have the whole setup be reproducible - maybe with a set of YAMLs on a k8s cluster, or something like Pulumi (I hate Terraform’s HCL with a passion). Kubernetes does come with a lot of niceties like node transparency for workloads and KubeVIP - but again, too complex for me without some layer on top making sense of it all.

Another annoyance is how my home IP is on the DNS records for my domain - I’d like to avoid that, if possible. Maybe something like ARP proxying with Wireguard. Seems like Hetzner supports it on servers with an added interface.

All of these shortcomings do motivate me to work on side projects related to them - a mesh VPN, a portable standard for selfhosting apps, like OVA for VMs maybe…? I just need to get better at organizing my time for them and actually get something out - if I could just spend less time trying out new self-hosted apps 😅

In reality it’s not as time-consuming of a hobby as most think. And you do learn a lot from it. If you’re wondering what you could be running at home (even on a single old PC you have laying around) there are subreddits for it and even newsletters.

Let me know if you have question with comments on this post’s toot. Until next time 🖖