5 Steps to Safely Replace a Drive in a Linux ZFS Array

Let’s lay out an example scenario - say we have a mirrored (RAID1) array, and I just got an Email alert from smartmontools telling me that a drive /dev/sdf is failing in my ZFS RAID 10 array.

Before I configure an array, I like to make sure all drive bays are labelled with the corresponding drive"s serial number, that makes this process much easier! I have a DYMO LetraTag LT-100H label maker that does the job just fine. - [Affiliate Link]

Notes

1. If you have an extra drive bay available, refrain from removing the old drive until after the resilver is complete.
2. ZFS is completely capable of replacing a disk with an unformatted one, there are some scenarios that require manual formatting (e.g. an array who’s disks contain multiple partitions), but you should otherwise be fine skipping steps 3 and 4. If your system does require manual formatting, there may be other steps needed after the resilver is complete (e.g. re-installing the bootloader with grub-install /dev/sdX for example), you should also avoid rebooting the system while ZFS is resilvering in this case.

1) Gather Information

The first thing we need to do is collect some information we will want handy during the process. I highly recommend opening up a text editor on your workstation and dropping the information there while you work.

GUID, Pool Name, and a Similarly Partitioned Disk

Use the command zdb to list out some data for all pools, you can see from the output below the my failing drive has a GUID of 4024410420552873090, lives in the pool raid10, and has an adjacent drive /dev/sde which will have the same partition table.

Serial Number

If you don’t already have it, the serial number of the failed drive is easily attained by running the following command. The smartctl command is supplied with the package smartmontools.

Installed Disks

Just to make sure we don’t wipe out the wrong disk, let’s get a list of what we have installed. You can see from the output below that I have disks /dev/sda through /dev/sdh installed.

2) Remove the Failing Disk

Now that we have all the information we need let’s get rid of the failing disk, first we’ll remove it from the ZFS pool.

Note: If this command fails, which may happen if the drive has completely died, use the disks GUID instead: zpool offline raid10 4024410420552873090

We should check that it’s been removed before moving on.

Now let’s remove the disk from the SCSI subsystem to ensure its disconnected cleanly.

3) Format the New Disk

After physically replacing the disk, we’ll need to copy the partition table from the similar disk we found earlier, to find the new disk we’ll use the command lsblk | grep sd again, it’s usually the same as before, in my case /dev/sdf.

IMPORTANT: The syntax of this command is counter-intuitive in my opinion, read these steps carefully, getting the source and target backwards here may hose your data!

The command we are going to use is sgdisk --replicate=/dev/TARGET /dev/SOURCE, where TARGET is the new blank disk, and SOURCE is the live disk with a similar partition table.

4) Randomize GUID

To prevent some really bad potential mix-ups by ZFS, each disk should have a unique GUID, we’ll need to address that since we cloned the partition table from another disk.

5) Add new Disk to ZFS Pool

Use the zpool replace command to add the new drive into the pool.

Check to make sure it has been added successfully.

Now we wait! You can keep an eye on the status with the command watch zpool status raid10 -v, my system took about three hours to finish resilvering 186 GiB of data.