Gentoo on OVH

A couple months ago I switched my hosting provider from Hetzner to OVH. And actually truth be told I was quite happy with Hetzner. I cannot think of a single issue I had with them. In 2023 I switched to Hetzner from Linode who I had been with for over a decade.

I was also pretty happy with Linode for a long time, but the last year or so that I was with them the performance of my VM suffered greatly despite them having migrated me to one or two different physical hosts during that time (for unrelated reasons, just maintenance stuff on Linode's end I assume because each of these was initiated by them, not me). After the last switch, the already poor performance of my VM got even worse and I started wondering what the heck I was even paying for especially since Hetzner at that time was offering comparable VMs for cheaper, combined with Hetzner's generally quite good reputation.

I have been running Gentoo Linux both at home and on whatever VMs I've had since the mid-to-late 2000's. Gentoo is a bit of an atypical choice for a server, but Linode continues to support it as an out-of-the-box choice which made Linode an easy choice for me to switch to back when Rackspace acquired and shut down Slicehost in 2011.

Hetzner doesn't provide out-of-the-box support for Gentoo, but you can easily do a manual installation of it on a Hetzner server by just booting to a rescue OS and then follow the Gentoo handbook. I was quite happy with that setup.

However in recent years I've grown very uneasy about having any of my personal data located in the US. Generally speaking nowadays, when I have a choice in the matter, I decline the US-based offering for whatever it is I'm looking at. The other thing I've switched to this year is bunny.net for DNS (as opposed to a more typical choice like Cloudflare, which I have zero interest in using for any of my personal stuff).

Anyway, Hetzner has data-centers in the US and Europe, and to be honest I'd rather not deal with the extra latency to Europe either. Linode does have data-centers in Canada, but given my issues near the end of my time with Linode, I'd rather not switch back to them. Plus, Linode is an American company.

So, I decided to give OVH a look since they also have Canadian data-centers and are not an American company (they do have a US presence of course). OVH doesn't exactly have the most stellar reputation, but I figured I'd give them a shot and see how it went. A big factor in this decision was their Kimsufi line of dedicated servers. This is older hardware that is being offered at quite a discount as compared to equivalent current generation hardware. After Hetzner's recent price increases, it was kind of wild to me looking at this and realizing I could get a perfectly decent dedicated server with twice the specs (in terms of number of CPU cores and memory) of my current Hetzner VM for roughly the same cost. I don't need the latest and greatest hardware. Even 10 year old hardware still performs perfectly fine for many use cases.

But it is very much worth dwelling on some of the negatives that come along with this OVH Kimsufi server choice. This means that I am on old hardware combined with the fact that I suddenly need to care about this versus going with a VM. On a VM the underlying hardware is out of sight. It is not directly visible to you. If there's a hardware problem on the physical host your VM is currently on, your hosting provider takes care of it. Not your problem. With a dedicated server, the hosting provider is still going to take care of it but I do need to worry about it too, most likely. The most important concern is perhaps the disks. I need to have redundancy and a good backup solution in case of a failure. The hosting provider will replace failed hardware, but that's all they do. The software and data that lives on those disks is entirely up to me.

This lesson was made perfectly clear to me when the power supply failed 3 days after getting a server initially set up. OVH's support team caught the problem automatically and addressed it in short order all without my ever having to submit a support ticket. But it was another reminder that this is older hardware. Another reminder of this is that the two SSD's in my OVH Kimsufi server both have over 75,000 hours on them and have had over 800TB of data written to them during their lifetime, as per the SMART data. I've not encountered any issues with them so far, but that could change at any time.

Just like with Hetzner, OVH doesn't offer Gentoo as an out-of-the-box choice when provisioning a server, but that did not deter me as I knew I could just do the same "boot to a rescue OS and do a manual install"-method.

Anyway, that's my typically long preamble about the "what" and "why." I wanted to write this post mainly to share my experiences and tips for doing a Gentoo install on an OVH dedicated server (such as one of their Kimsufi servers) because it wasn't without its hiccups. Some of which were just due to dumb mistakes on my part, as is also typical for me, heh.

Installation

As mentioned, with hosting providers like OVH which don't offer an out-of-the-box OS choice for Gentoo, you can still do a Gentoo install anyway. You just need to switch the boot device in your server on the hosting provider's control panel to use whatever rescue OS they provide. In OVH's case this is a rescue system based on Debian 12. Then you just follow along with the Gentoo handbook.

RAID

From what I can see, all of OVH's Kimsufi servers come with a pair of disks which is good. I don't know if any come with hardware RAID capability. Mine did not.

At any rate, this is not a problem as we can use mdamd to provide software RAID which will work great.

In my case these disks were both Intel SSDPE2MX450G7, and they showed up as /dev/nvme0n1 and /dev/nvme1n1.

Preparing the disks

I am using the following partitioning structure.

Example fdisk -l output for this once all these partitions were created on both disks.

Disk /dev/nvme0n1: 419.19 GiB, 450098159616 bytes, 879097968 sectors
Disk model: INTEL SSDPE2MX450G7
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 74F111A9-E437-3E4A-9265-56D79EF0541E

Device           Start       End   Sectors   Size Type
/dev/nvme0n1p1    2048    264191    262144   128M EFI System
/dev/nvme0n1p2  264192   1312767   1048576   512M Linux RAID
/dev/nvme0n1p3 1312768 879097855 877785088 418.6G Linux RAID


Disk /dev/nvme1n1: 419.19 GiB, 450098159616 bytes, 879097968 sectors
Disk model: INTEL SSDPE2MX450G7
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 7D33174C-BDD5-4242-A78D-BAC4BADF86B3

Device           Start       End   Sectors   Size Type
/dev/nvme1n1p1    2048    264191    262144   128M EFI System
/dev/nvme1n1p2  264192   1312767   1048576   512M Linux RAID
/dev/nvme1n1p3 1312768 879097855 877785088 418.6G Linux RAID

Setup mdadm

First thing's first, we should ensure that any existing software RAID setup is stopped. Probably this is not the case, but it doesn't hurt to be safe, especially if your existing environment differs from mine in some way.

mdadm --stop /dev/md*
mdadm --zero-superblock /dev/nvme0n1
mdadm --zero-superblock /dev/nvme1n1

Next we can create the software RAID on both our / and /boot partitions.

The device partitions specified in the below commands are the same as those listed in the above table for the corresponding file system.

# create software raid for /boot
mdadm --create /dev/md0 --level=10 --layout=f2 --raid-devices=2 --metadata=0.90 /dev/nvme0n1p2 /dev/nvme1n1p2

# create software raid for /
mdadm --create /dev/md1 --level=10 --layout=f2 --raid-devices=2 --metadata=0.90 /dev/nvme0n1p3 /dev/nvme1n1p3

Some notes here:

• Once complete, our RAID file system devices will be /dev/md0 for /boot, and /dev/md1 for the root file system /.
• We're using RAID 10 here with a "far" layout (--level=10 --layout=f2 --raid-devices=2).
• Since these are both for "critical" file systems, we use --metadata=0.90. If we're adding new disks in the future and want to setup RAID for those too, we might not need this option.

These commands will take a while to complete. We can monitor the progress of them with something like:

watch -n 1 cat /proc/mdstat

Once it has finished synchronizing, we should generate /etc/mdadm.conf. Of course keep in mind, that right now we're not even chroot'ed into our new Gentoo install (we're not that far along yet!), so when we generate the mdadm.conf it will be on the rescue OS's temporary file system. We just need to remember to copy this over to our Gentoo file system later on.

mdadm --detail --scan > /etc/mdadm.conf

Now we can create the Gentoo file systems. This is all very simple and right out of the Gentoo handbook for the most part. We're setting up our /efi partitions as FAT32, and we're doing it directly for each since they are both intentionally outside of the RAID setup. However, for both /boot and /, we're specifically creating the file systems on the /dev/md0 and /dev/md1 devices respectively since those are our actual RAID devices.

mkfs.fat -n efi1 -F32 /dev/nvme0n1p1
mkfs.fat -n efi2 -F32 /dev/nvme1n1p1

mkfs.ext4 -T small -L boot /dev/md0
mkfs.ext4 -L root /dev/md1

Then we just need to mount all of this ...

mkdir /mnt/gentoo
mount /dev/md1 /mnt/gentoo
mkdir /mnt/gentoo/boot
mount /dev/md0 /mnt/gentoo/boot

mkdir /mnt/gentoo/{efi,efi2}
mount /dev/nvme0n1p1 /mnt/gentoo/efi
mount /dev/nvme1n1p1 /mnt/gentoo/efi2

Here I've chosen to mount our EFI partitions at /efi and /efi2 within our Gentoo root. Later when we install GRUB and run grub-install, we'll want to run it against both /efi and /efi2 to ensure it installs the same EFI images in both places.

At this point you can proceed along with the rest of the Gentoo handbook and come back to the below sections when you get to those parts of the installation process.

After unpacking the Stage 3 tarball

After you've got your baseline system unpacked via whatever Stage 3 tarball you picked, you should copy over the /etc/mdadm.conf that you previous generated via the mdadm scan mentioned above which was only saved to the rescue OS's temporary file system.

From outside the chroot environment:

cp /etc/mdadm.conf /mnt/gentoo/etc/mdadm.conf

Before installing the kernel

USE flags

I'll be assuming use of both GRUB and Dracut. You can easily ensure both of these are installed at the same time as you install the kernel (at least, when using one of Gentoo's distribution kernel packages) by adding the following to your USE flags:

sys-kernel/installkernel dracut grub -systemd

Note that I add -systemd here even though I am using systemd, I just don't want to use systemd's kernel-install.

Ensure the mdadm package is installed

Within your new Gentoo installation (that is, when you're chroot'ed into it), you should make sure to install sys-fs/mdadm before installing the kernel. This is because Dracut will automatically include the right modules in the initramfs necessary for software RAID as long as you have the sys-fs/mdadm package installed beforehand.

If you miss this, you can fix it by simply re-creating the initramfs after ensuring sys-fs/mdadm is installed. If you're installing the kernel via one of Gentoo's distribution kernel packages, this is as simple as running:

# or perhaps `gentoo-kernel-bin` ...
emerge --config gentoo-kernel

Add kernel cmdline parameter for the root device

We need to ensure the kernel picks the right root device at boot. Run blkid and find the UUID matching your /dev/md1 device, which corresponds to the root file system / mounted from the software RAID.

Then place the following in /etc/dracut.conf.d/00-installkernel.conf, replacing the UUID here with the one you grabbed from blkid above.

kernel_cmdline=" root=UUID=6e5fe2d8-72f3-4053-ba3b-9c83f6be2a70 "

Bootloader

I'm a big fan of rEFInd which I use on all of my personal systems, but I encountered some initial difficulties getting it to boot on my OVH server with the software RAID setup. It's not entirely clear to me what the problem was, but I decided I didn't care enough to troubleshoot it further and ended up going with GRUB since I knew it definitely did have support for mdadm.

It is very important to point out that OVH specifically says not to change the EFI boot order on their dedicated servers. Doing so will likely break things for you. Since we're using GRUB, this means we'll want to pass --no-nvram to grub-install when we get to that point.

I like ensuring that sys-boot/efibootmgr is installed on all my systems so that I can verify the EFI boot order is correct (by running efibootmgr -v before and after GRUB installation) and, if needed, fix things up manually if I accidentally muck something up.

Configure GRUB

At the very least, we need to update GRUB to ensure it passes the domdadm command line argument to the kernel.

Additionally, on OVH there are a few other things you'll definitely want to change to have everything work better with OVH's serial console interface to your server during bootup (relevant for use of either serial console's web interface or SSH interface).

Ensure the following values are set in /etc/default/grub.

# ensure mdadm support and ensure the kernel boot output, etc is all visible
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200 domdadm"

# ensure that the grub menu is visible
GRUB_TERMINAL=console
GRUB_GFXPAYLOAD_LINUX=text

Most importantly, we add domdadm to the cmdline string in GRUB_CMDLINE_LINUX_DEFAULT that GRUB will append when booting the kernel.

But we also add console=tty0 console=ttyS0,115200 to GRUB_CMDLINE_LINUX_DEFAULT as well. This prevents the kernel from switching into some graphical text mode. Then the subsequent values used with GRUB_TERMINAL and GRUB_GFXPAYLOAD_LINUX ensure that GRUB itself will also not start up in any kind of graphical mode either.

If you do not include these options, what I noticed on my OVH server is that GRUB would boot up in a graphical mode of some sort and it would not be visible at all when using the OVH serial console interface. Subsequently, when the kernel boots it would also use some graphical-ish mode and its output would also still not be visible. You'd just end up seeing this, and never anything else beyond this point.

It is still actually booting up, just you cannot see anything from the serial console interface. This would obviously be problematic when troubleshooting boot failures.

Install GRUB EFI image

Depending on the order in which you did things, it might be a good idea to force a rebuild of the kernel initramfs just in case.

emerge --config gentoo-kernel

And then we can generate the final grub.cfg and install the EFI images to our two EFI partitions, /efi and /efi2.

grub-mkconfig -o /boot/grub/grub.cfg
grub-install --no-nvram --efi-directory=/efi /dev/nvme0n1
grub-install --no-nvram --efi-directory=/efi2 /dev/nvme1n1

Again as mentioned previously, on OVH we should always remember to pass --no-nvram to grub-install to prevent modifying the EFI boot order from how OVH pre-configures it, else we will probably end up with an un-bootable system.

Conclusion

One of the main stumbling blocks for me during this whole process was actually the whole "force GRUB and the kernel to not switch to a graphical text mode" thing as mentioned above in the GRUB configuration step. That left me wondering if I'd actually broken my boot configuration because I couldn't see anything at all. Combined with the fact that when I first encountered this problem, I actually did have a problem with my kernel cmdline, so even just waiting for a while until I could remotely SSH in didn't work and I was thoroughly confused.

I also must confess that before this whole procedure I'd never installed an OS with a RAID setup of any kind before. It ended up being a lot easier than I thought, which was nice. However, I will admit that some of the documentation surrounding it felt a bit scattered, so I hope that by writing this post I can help someone else out there.