I’ve been doing some more NixOS pottering over this weekend, working out the details of a thorough backup strategy. Before I moved to Linux, I used (and still use) iDrive for my backups. Its reasonably priced, and has servers based in Europe, and seems to work well enough. You can also run it on Linux, but I had some trouble getting it working with NixOS. I have also used the Gnome Circle app Pika Backup, which is delightful, but that is obviously not cross-platform. It is based on Borg Backup which is cross-platform (across Linux and macOS anyway), but I struggled a bit to think how I would link everything up.

After quite a bit of reading on what might work well, I settled on Restic. This is cross-platform, has a long development history, is now quite well-established and stable, and has incremental snapshots and encryption built in. It enables backing up to a variety of remote locations natively, but can also be paired with rclone backends to enable an even wider range of possibilities. It is available as a brew on macOS, but on NixOS, you can set it up as a service for automatic backups using a systemd timer.

Setup of backups from the local host

So far I have set up the Linux side of it for my NixOS desktop, ‘morse’. The macOS part is yet to come, but as they are still backing up to iDrive (the subscription runs for quite a few more months), I will leave them as they are until I have time to set up those machines with Nix and run home-manager or nix-darwin on them. However, I have — as far as possible — tried to make the setup portable to other machines, and the plan is that all my computers would share the same backup structure.

You can see the rough structure in the sketch above. The idea was to have a proper ‘3-2-1’ backup. Each host would back up to a locally-attached drive (for the quickest recovery), to my NAS (to store the backup on a machine that was not attached to my desktop and uses spinning hard drives rather than SSDs), and also an off-site location in the case of my house burning down or flooding. Each host will create two ‘repositories’ (the term restic uses for backup targets). One, named ‘local’, backs up to an external USB-C SSD drive (‘Locutus’ in the sketch above^[1]) mounted locally. The other backs up to the NAS (‘Thursday’) via a Samba mount. This is also set up in the nix restic services module:

# in services/restic.nix
fileSystems."/mnt/thursday/Backups" = {
	device = "//192.168.178.67/external_backups";
	fsType = "cifs";
	options = [
	  "credentials=${config.sops.templates."sambaNas".path}"
	  "x-systemd.automount"
	  "noauto"
	  "x-systemd.idle-timeout=60"
	  "x-systemd.device-timeout=5s"
	  "x-systemd.mount-timeout=5s"
	  "rw"
	  "uid=1000"
	  "gid=100"
	];
};

Both are set up on systemd timers and run at set times: 17:00 every day for the local backup and 18:00 for the NAS backup. These times are ones that I am most likely to have my desktop awake, so it works well. I can always run either backup manually if needed.

Purge is set up for both to keep about a week of daily snapshots, 5 weekly and 12 monthly, though I will monitor the degree to which that increases the size of the repo. The paths backed up are /home and /var/lib/nixos as those seem to be the main paths that I need to restore which are not generated by NixOS itself as part of a build. One of the lovely things about having a declarative operating system is that almost all of the system part does not need to be backed up because you can just regenerate it from the version controlled configuration files.

NixOS automatically creates a convenient wrapper script from which you can manually run restic. This is named as restic-<name> where name is the name of the backup set in the configuration, so restic-local and restic-nas for me. This also automatically supplies the password for you (stored securely with SOPS), so you don’t need to enter it.

Using the local backup as an example:

sudo restic-local snapshots # to view snapshots using the wrapper
sudo restic-local check # check integrity of repo

I’ve tested restoration of files from the repository. You can specify a whole path, or else only a sub-path/single file to restore. You need to specify the snapshot hash to use or latest and also where the restored files should be placed. It’s easy to use once you get used to the conventions, and quite flexible.

Management of services

If you want to look at the service or timer, both are named restic-backups-<name>.{service,timer}. So, again for the local backup,

sudo systemctl status restic-backups-local.service
sudo systemctl status restic-backups-local.timer
# run the backup manually
sudo systemctl start restic-backups-local.service
# or to see more detail
journalctl -xeu restic-backups-local.service

Off-site backups

The last part of the backup is to ensure that we have backups stored ‘off-site’. I have plenty of space on pCloud (2TB), and you can set up pCloud as a remote via rclone. I made one full backup this way, which worked, but was rather slow, so I decided to buy a 1TB Hetzner box and use that as a remote. I might still use pCloud to back up my Unraid settings and Docker shares, and pCloud could definitely be used to back up my Photos and Music shares, as these change much less frequently.

I could have used restic on Unraid to do a restic copy of the latest snapshot to Hetzner. However, I decided that I would like to preserve all the snapshots in the repo in case of disaster. Since all the work of snapshotting/encryption and so on is already done, the consensus online seemed to be that using rclone to make a mirror of the whole repo would a bit faster, so that’s what I did. By running the clone in the middle of the night, it ensures that the restic snapshots are up to date for the day. I should probably do a restic check first in the script that runs rclone and then exit if any errors are found, but I will leave that until I have setup restic on my NAS.

I set up the relevant rclone config for the Hetzner Box on morse as well. This means that I can directly restore from the Hetzner Box, which is handy if the local backup died or something happened to the NAS so that I couldn’t restore from there. This is actually vital since the off-site backup is the disaster recovery in the case that all local machines and drives are destroyed in some catastrophic event. I’ve checked that this works with randomly chosen files, and it all went as planned.

Monitoring

I tried setting up an *.OnFailure service option to send a notification, but couldn’t seem to get that to work. I decided in the end to get a free healthchecks.io account. This enables me to set up a ping when it backs up so I end up with a dashboard indicating current and historical issues shown by red, amber or green symbols. I have set it up to email me in the case of failure. You can self-host Healthchecks, so I might consider that in the future, but it is free for 20 checks/pings so I don’t see any immediate need to do that.

The pings are set up like this for the NAS backup as an example

systemd.services = {
    restic-backups-nas = {
       preStart = "${pkgs.curl}/bin/curl -sS -m 10 --retry 5 https://hc-ping.com/${lies.hcheckPing}/morse-restic-backups-nas/start";
       postStop = "${pkgs.curl}/bin/curl -sS -m 10 --retry 5 https://hc-ping.com/${lies.hcheckPing}/morse-restic-backups-nas/\${EXIT_STATUS}";
    };
};

Future tweaks

As I mentioned earlier, the plan is to roll the same setup out for my macOS mini and MacBook. I would also like to convert the backups I currently run with Duplicati on the NAS to back up the Unraid configuration and shares. Duplicati seems fine, but I would rather standardise on one system to keep things manageable. I need a bit of a break before tackling that though, and I want to see how things go with restic. However, I’m really happy with it overall. It is simple to use once you understand the principles and terminology, secure, robust, and pretty fast. Even my off-site backup to Hetzner only took about 24 hours for nearly 200GB of data, which I think is pretty good. Subsequent incremental backups take only seconds to complete.