Server Updates: In Which I Have A Server Now, For Real

This post is older than a year. Information may be outdated.

After years of yearning, longing, and pining (let's not forget pining), I finally got myself an actual server. Or rather, a friend of mine got sick of listening to me whine about my old, terrible server, and gave me a spare that he cannibalized for parts (but was still very good and usable).

I am now the proud owner of an HPE ProLiant DL360 Gen9.

This story starts a few months back, with the Purdue Linux Users Group (of which I am an officer) deciding to put together a mini-datacenter for students to use. It's nothing too fancy, really just a 12U rack sitting in a room with a handful of static IPs from Purdue University. I like the rack, it's a nice source of white noise for when I need a good study space.

I'm the server second-from-the-bottom! My friend has the top rack mount server (and the precarious hard-drives), and my other friend has the bottom-most server.

I got the server, set up the BIOS configurations, and thought to myself, "What the hey, might as well just take the drive out of my old server and put it in the new one!"

I made the mistake of appending that line of thought with the ill-fated words, "What could go wrong?"

It Went Wrong

It is the afternoon of Valentine's Day. I have a meeting in less than an hour, and I have blocked out time later for Valentine's Day Activities™. There are but a few scant hours that day to work on the server. I have less than an hour to work on the server.

Of course, I chose this time to move the drive over.

I skateboard the half-odd mile to my dorm, grab my laptop charger (as I neglected to take it with me in the morning), break it down into the two cables, and stuff it into my jacket. With trepidatious fingers, I shut off my laptop-server, fumbled with the screwdriver for too long, and extracted the expensive SSD I put in there after my last drive failure, proprietary laptop caddy and all. I wrap the drive in a paper bag to hopefully insulate it against my winter jacket's static, lock up my room, and skateboard as fast as I can back to the datacenter.

The adrenaline had taken over my body. I unwrapped the drive, praying that it was intact, and removed the four screws holding it into the Dell laptop caddy. A glance at my phone tell me I have less than thirty minutes before I had to go to my meeting.

I look at the screws I extracted from the drive caddy, and horror of realization dawns on me—these screws looked too short to be used in the server's drive caddies. The panic sets in. My hands are unsteady, I can't align the screws and the drive. I tried to put the screws in anyway, out of desperation, and three screws go in. The fourth stubbornly refused.

Three out of four is good enough for me. Less than twenty minutes left.

I put the caddy into the first drive bay, suffer through the terribly long boot time of enterprise hardware, and open the one-time boot menu.

I had only the option to boot from the network. The drive was not visible.

But Wait, It Gets Worse!

With the fifteen minutes I had left, I booted from an Arch Linux USB drive I had lying around, hoping that it wouldn't cause issues (foreshadowing), and ran fdisk -l, hoping that the drive had survived the journey and was not dead.

Thankfully, my data was intact. My mind is racing—what to do now? Perhaps I have to reinstall Grub? And so I chroot'd into the drive, pulled up the documentation for rebuilding Grub, and was promptly interrupted by my phone telling me that I had to leave now for the meeting otherwise I would be late.

Calming Down, Just a Little

I leave for the meeting, show up too early (I could have worked on the server!), and return to my server an hour and a half later. At this point the downtime had reached about two hours.

I now had just a few more hours before my class. I thought to myself, "I can do this. I have time. I just need to approach this rationally."

In my mind I had narrowed it down to a few possibilities:

1. Grub was broken
2. The server could not find the EFI file
3. Server is broken
4. Some strange hardware incompatibility that would be hard to debug

After an hour of fumbling around with the Arch Linux ISO and running into weird incompatibility issues with the differing installed Kernel versions, I decided to give up and use the Rocky Linux ISO to recover the Rocky Linux install I had on my server. It took me another hour of reading through forums and paywalled documentation to realize that reinstalling Grub was as easy as dnf reinstall grub2-efi grub2-efi-modules.

And yet, the server refused to detect the drive on boot.

The Troubleshooting Rabbit-Hole

I moved onto the next item on my list: the server couldn't find the EFI file. Issue was, efibootmgr showed that the boot menu entry already existed. The server found the boot entry just fine, after it had already booted. Changing the boot order in efibootmgr did nothing—the server refused to detect the SSD on boot.

There's no small amount of disappointment in my mind as I start to fear that my shiny old-new server is broken. I had already updated the iLO¹, but the BIOS updates were hidden behind the HPE support contract. I do not have a contract. That did not stop me from finding the files updating the BIOS, however.

Now that I was no longer running firmware from 2015, I booted the server, checked the boot menu, and the SSD still was not visible.

There was only one thing left on the list: some weird hardware incompatibility. You see, this server has an integrated RAID card for redundant storage at the hardware level. Currently, the RAID card was configured in passthrough mode. From tinkering around the server's guts, I knew that there were additional SAS ports on the motherboard. Currently, the backplane was connected to the RAID card, so I swapped around the connection.

The boot menu showed my SSD. This was it, no more downtime.

I boot into the server, go to log in to set up networking, and get hit with ersei: no shell: permission denied.

Oh great, another problem. I don't really know for sure what caused the problem, but my bet is that the Arch Linux chroot messed with the SELinux labels on the drive. Booting into the recovery Rocky Linux USB and creating a /.autorelabel file fixed the problem.

We were in! I set the public IPs, started the systemD services², connected to the proxy VM, and loaded up my website on my phone.

It's up! My server has successfully moved! After sorting through the hundred or so Zabbix notifications and ensuring that everything was really back up, I was finally done!

Nope, Not Done Yet

See, now I have a server that isn't horribly weak. Naturally, I have to move everything over. I first started with the networking and stopped using the proxy. For the first time in years I finally had a public IP address for my website! I moved my DNS from Cloudflare to Porkbun (slightly more expensive, but only marginally) and started using Hurricane Electric as my nameserver. Why? Because Cloudflare doesn't let you set custom nameservers, and I wanted to run my own nameservers.

I moved the Minecraft servers in Oracle Cloud over to the local machine (the CPU graph doesn't look like it's at zero in Zabbix now, it's now marginally more!) and shut off the Oracle Cloud VMs after the DNS changes had fully propagated. I moved my Minio from the singular SSD to the pool of drives that came with the server by making a new Minio instance and moving the buckets over with mc mirror (I now have erasure coding and some form of redundancy, yay!)

So that brings us to now. Believe it or not, I'm glossing over a bunch of minor issues because they really aren't as funny or interesting (mostly just mucking around in iLO or BIOS) and could just be a list of bullet points:

• Intel TXT was enabled in BIOS which caused the Arch ISO to not boot
• One of the fans had a shot bearing, so I got a replacement from the cannibalized remains of another server
• Moving Minio was unnecessarily painful
• The drives that came with the server did not have SATA secure erase functioning
• Lots more that I do not remember

Wait, There's More?

Oh yeah. You see, I did not get the server on Valentine's Day. I got it a few days beforehand. Of course, I needed to get the environment all set up. Beyond just cleaning out the hardware, updating the iLO firmware, and setting the proper BIOS configuration options, I had to do some networking.

The rack's networking is set up like so: there is one upstream Gigabit network connection to the wall. We are allocated static IPv4 addresses and SLAAC'd IPv6 addresses. Anything that wants a static IP address connects to that network (via a network switch). There is an EdgeRouter X that has NAT enabled for connections that do not require a public address (or IPv6, we'll get to that later).

The main thing connected to the EdgeRouter's NAT'd network are the iLO and iDRAC cards of various servers. Earlier I had set up port forwarding at a high, random port, but this was not ideal as anybody could plausibly access the management console of my server (although it is password-protected). After poking around in the documentation and blog posts of various age and quality, I installed ZeroTier on the EdgeRouter and set up ethernet bridging so that I could access the internal network from anywhere if I was connected to the ZeroTier network. Despite my best efforts to build a newer ZeroTier package for the EdgeRouter, I gave up and decided to live with the older version, which worked fine enough.

However, I cannot, for the life of me, figure out how to pass a public IPv6 address through to the network behind the EdgeRouter. It was tempting to install OpenWRT on the EdgeRouter instead, but that would probably cause more problems than it would solve.

In the end, I gave up and resolved myself to a life of no IPv6 on the internal network.

Speaking of the Internal Network

I broke the network for everybody a few times.

To those whose Mastodon instances went down, I apologize.

Here's what happened: the EdgeRouter only has five total ports, of which one is used for the upstream WAN connection, leaving four connections. We were reaching the point where there were more than four connections to the EdgeRouter, so one had to be disconnected every now and then. We were timesharing the last remaining ethernet port on the EdgeRouter. This was not ideal. The large network switch, however, had 24 ports, of which less than half were used.

I had the genius idea to split the switch into two VLANs: the internal network and the external network. The ports on the right-hand side of the switch were for the internal network (connected to the EdgeRouter), and the left-hand side was connected to the WAN.

Here is a non-exhaustive list of what went wrong:

1. I could not connect to the switch's management interface because of how the network was structured. It took an hour to piece together a suitable DB-9 serial-to-USB cable.
2. Figuring out the serial connection parameters took another hour of trawling shady manuals websites and a lot of guesswork.
3. Going through the basic setup program to reset the password from the defaults caused SPT to activate, which Purdue's network didn't like and killed the connection to our network port for five minutes.
4. In an attempt to avoid any more downtime, I connected two network connections to the switch from the wall, so I could disconnect the other and move the connections around to the right places. This also killed the connection to the port for five minutes.
5. I got the numbering of the switch ports wrong, so the top half and the bottom half of the switch were split into two VLANs, not the left and right side.

I am never touching that switch again. On the plus side, all the potential failure modes are documented now!

Aside: Server Rails

Server rails are a fucking mess. It took weeks to find the compatible rails for my server, and Lily's server is using the wrong rails and is hanging down slightly. I ended up giving up looking for the right rails and decided that two left rails worked just fine (surprisingly well, actually, but the server ears did not lock in on one side, which was fine).

I had to trawl through literal buckets of server rails down at the Purdue surplus store to find the "matching" pairs. I ended up buying four left rails, because I could not find the matching right rails.

I wonder who has them.

So What's Next?

I have plenty of ideas for what to do next. In the time it took me to put this post together, I have already completed the following:

1. A Nix Hydra server to build all of my custom packages for me
2. My own authoritative DNS server (blog post soon!)
3. Minecraft servers

At the time of writing, I still have the following planned:

1. Redundant monitoring so if my server goes down I can get notified
2. Monitoring iLO
3. Monitoring other self-hosted software
4. Actually using the authoritative DNS servers, as soon as Purdue unblocks port 53

I can not wait to write about how strange running an authoritative and redundant DNS server is. See you all in the next one.

Thoughts? Comments? Opinions? Feel free to share (relevant) ones with me! Contact me here if you want.

I've also moved to my own Fediverse instance: @ersei@ersei.net.