Disaster Recovery

Restoring a Failed Build

If shit hits the fan and the server is not responding to pings, there is little left to do. As you won't be able to rebuild the messed up configuration without ssh, so you will have to fire up the rescue console.

Once booted into the live system, it is best to enter a NixOS via the kexec method described in the ReadMe.

To restore your system you have two options:

1. Boot a previous version of the system with grub-reboot
2. Mount the system and use nixos-enter to chroot into the system and rebuild the config to your liking.

Booting into a previous version of the system

1. mkdir /boot
2. zpool import bpool
3. mount -t zfs bpool/nixos/root /boot
4. Find a Grub Entry by checking the /boot/grub/grub.cfg file for menuentry
5. grub-reboot "<menuentry>"
6. zpool export -a
7. reboot

Mounting the existing system

1. If you are using an encrypted root, decrypt every drive in the rpool:

cryptsetup luksOpen /dev/disk/by-id/<name>-part3 luks-rpool-<name>-part3

2. Import all pools: zpool import -a
3. Check that all pool have been successfully imported by executing zpool list
4. Mount all volumes according to NixDotfiles/modules/boot/default.nix (zfs-root.fileSystems.datasets):

mount -t zfs rpool/nixos/root /mnt
mount -t zfs bpool/nixos/root /mnt/boot
mount -t zfs rpool/nixos/home /mnt/home
mount -t zfs rpool/nixos/var/lib /mnt/var/lib
mount -t zfs rpool/nixos/var/log /mnt/var/log

5. nixos-enter
6. Edit the configuration to your liking
7. Rebuild the system by executing nixos-rebuild switch --flake .#<hostname> while in the NixDotfiles

---

The rebuilding might fail with an error such as

'/nix/store/gz9kldhk94fkzrirjxr0rq0vafxh37nd-system-path/bin/busctl --json=short call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager ListUnitsByPatterns asas 0 0' exited with value 1

In that case exit the chroot with Ctrl+D and do the following:

1. Backup the /run dir: cp -r /run /run.bak
2. mount -o bind /run /mnt/run
3. nixos-enter
4. nixos-rebuild switch --flake .#<hostname> while in the NixDotfiles
5. A lot of errors will arise. The rebuilding will still have happened, but its not "clean".
6. Exit the chroot with Ctrl+D
7. /run.bak/current-system/sw/bin/zpool export -a
8. /run.bak/current-system/sw/bin/reboot