Starting my Homelab Journey

Finally, after years of hanging out (read: larping) in sysadmin (and adjacent) circles, I got a small machine to play with. And yes, the website is now being served from there!

How?

Now that I got a job I was thinking about hobbies, so naturally (read: being a messed-up person like me) I figured that a homelab would be a nice addition. I went on eBay and started looking at random second-hand mini PCs on eBay, but the specs confuse me, so I sat on the idea for a while.

As you know I'm also a person desperate for third spaces, even more so since I just moved to a new area, so again, naturally (idem... but also having hung out briefly at Foulab), I decided to look for hackerspaces in Seattle. On my first visit to /dev/hack - home of the famed Shibari PC - through some sequence of events I ended up obtaining that ThinkCentre, in a cardboard box with the word "scrap" scribbled on it.

As expected the specs are not great: I don't need a strong CPU, but there's only 4 GB RAM and 32 GB SATA SSD. Suspiciously, the included LetOS (used just for the first boot) somehow has everything removed (the only way to run any terminal commands is through network settings, by using a custom command to test connectivity, and despite the presence of a stub Wi-Fi antenna the OS only accepts wired connections). So the first order of business is to get a new hard drive, then to get a new OS. Actually, before that, I took out the heat sink to inspect the hardware, which ripped out the heat paste, so on a second visit to /dev/hack it was repasted. I certainly owe my thanks to them for jumpstarting my homelab (and I now go there somewhere regularly).

Anyways, back to eBay for a reasonably-sized (and -priced) hard drive. After some advice on looking out for SMART metrics, I decided to acquire a 2%-used 512 GB NVMe from a local seller (apparently selling since 1998). To save even more on top of determining the lowest auto-acceptable bid, I requested local pickup, so on a normal afternoon I met the seller outside a coffee shop near Seattle Center and he handed over the drive in an anti-static bag in a totally not suspicious manner. Actually I was even confused between SATA and NVMe only to realize my rig supports NVMe as well, so there, assembled my machine. After buying another SanDisk USB thumb drive from BestBuy I loaded it with Fedora Server and, after a few tries, installed it on my machine.

Building things out

I know better than exposing my homelab directly to the internet (although I did try that for a few days just to test the ISP), so I need an intermediate VPS. I thought about using Google Compute Engine (I literally work for that), but their free tier is deceptively hidden. Well, I could pay a bit... except if I want IPv6 I'd have to pay for bandwidth at $0.12/GB (only first GB free), not to consider possible future expansion which requires upgrading machine types. At the advice of 4d2.org I ended up settling for a small VPS at Tier.net.

The current layout, simply put, is that layer 4 networking (including some primitive ratelimiting) occurs in the VPS using HAProxy, which proxies through a WireGuard tunnel into my homelab. There's also a WireGuard tunnel from my laptop to the VPS, allowing me to not only access both machines from anywhere, but also seal off port 22 (though in the process my firewall has become quite messy so be mindful of that). Iocaine (with NSE - and admittedly their documentation is quite unreadable) is used to front public-facing services (and apparently Anthropic will crawl your domain right after a TLS certificate is requested, so I guess it is now necessary).

Right now there's a private ntfy instance (for UnifiedPush) and an accompagnied mollysocket instance (for UnifiedPush for Signal). On the VPS there's also Forgejo Actions that allows me to run CI (since Codeberg's resources are quite limited), which is now used to build and publish the website (some rsync wrangling is necessary now that I no longer publish the SSG output on the website). Eventually I will start bringing home other services I currently use, expand the website (so that I can have non-free image assets that don't have to be on Codeberg), properly ban & report abusive IP addresses, maybe finally build the automod for the two large Mastodon instances under my management, and maybe explore around a bit.

It did take me a while to get a nice caddy config working; this is especially a problem when Iocaine-blocked requests started making my log look dirty, so I split it into two locations. In case anyone want to know how I set up the log, it is something like

(subdomain-log) {
        log access_{args[0]} {
                format transform `{common_log}` {
                        time_format "02/Jan/2006:15:04:05 -0700"
                }
                hostnames {args[0]}
                output file /var/log/caddy/{args[0]}.log {
                        mode 666
                        roll_at 00:00
                        roll_local_time
                }
        }
        log blocked_{args[0]} {
                format transform `{common_log}` {
                        time_format "02/Jan/2006:15:04:05 -0700"
                }
                hostnames {args[0]}
                output file /var/log/caddy/blocked/{args[0]}.log {
                        mode 666
                        roll_at 00:00
                        roll_local_time
                }
        }
}
(iocaine) {
        @read method GET HEAD
        reverse_proxy @read 127.0.0.1:42069 {
                @fallback status 421
                handle_response @fallback {
                        log_name access_{args[0]}
                }
                @blocked status 2xx
                handle_response @blocked {
                        log_name blocked_{args[0]}
                        copy_response
                }
        }
}
example.com {
        import subdomain-log example.com
        import iocaine example.com
        # handle...
}

so that the blocked requests reside in the /var/log/caddy/blocked/ directory.

Oh, and I need more RAM! (And hopefully it doesn't grow into a k8s cluster...)

What else is happening?

At-work, projects are moving through, though still confused about what's happening while I navigate through the large corporation technical and procedural mess (it's a lot harder to deal with people than to deal with code), but I've been hanging out with a group of other new grads at the Kirkland campus (where most people are of family-age and thus difficult to socialize with) for lunches and (very occasionally) for weekend getaways, so that's good.

Off-work, since I moved here, I went to a few r/Seattle Discord meetups, hiked (twice, once at Mount Rainier with people from the aforementioned Discord, and once at North Cascades with other Googlers), visited dma.space (it's admittedly a bit more serious than /dev/hack so I hang out at the latter a lot more often), got an e-bike (and rode across Lake Washington), started riding the e-bike to work regularly, fell off from the e-bike (hurt my face badly, and I've only got back to riding recently) and, sprinkled among that, checking out restaurants (read: finally having an excuse to be spoiling myself a bit more).

Sometimes I wonder what people really do off work. It's been a while since I've consumed substantial amount of media, so maybe I need to get back to that. Gotta get something to make me less bored...

Mandatory neofetch output