ZFS

Decisions

While encrypting the pool volume is possible, I decided to go with a sub-volume, just in case I need to add un-encrypted data later
The options for encryption keys passphrase, raw key, and hex key. I'll be using a hex key for ease of backup in my password database.
When creating the pool use the entire disk, not partitions where possible
When creating the pool, use /dev/disk/by-id, not /dev/sda, as these can change on boot

Create the pool

Grab the disk IDs as shown below, and use those IDs to create the pool

ls -lah /dev/disk/by-id
zpool create datapool -o ashift=12 -o autoexpand=on -O compression=on -O atime=off -O mountpoint=none mirror diskID1 diskID1

Creating the Volume

Normal Dataset

zfs create poolname/datasetname

Encrypted Dataset

mkdir /etc/zfs/keys
chmod 700 /etc/zfs/keys
openssl rand -hex -out /etc/zfs/keys/datasetname 32
chmod 600 /etc/zfs/keys/datasetname
zfs create -o encryption=aes-256-gcm -o keyformat=hex -o keylocation=file:///etc/zfs/keys/datasetname poolname/datasetname

Backing up data normally

Create an initial snapshot and backup

zfs snapshot -r poolname@snap1
zfs send -Rw poolname@snap1 | zfs recv -Fdu backuppool

After some changes, send an incremental

zfs snapshot -r poolname@snap2
zfs send -Rw -I poolname@snap1 poolname@snap2 | zfs recv -Fdu backuppool
zfs destroy -r poolname@snap1

Importing Backup Disk without Mounts

To safely import the backup pool disk without fudging existing mounts, use:

zpool import -N backuppool

Setting Max ARC Size

To check your current arc max (in bytes):

# grep c_max /proc/spl/kstat/zfs/arcstats
c_max                           4    10485760000

To set your new arc max, update modprobe and reboot:

# cat /etc/modprobe.d/zfs.conf 
options zfs zfs_arc_max=10485760000

Programmatically Deleting Snapshots

Let's say you have a few snapshots you've grep'd for, how would you delete them?

Use zfs list with the following flags:

-H for scripting mode, omitting headers
-o name to only grab the name field

Then use xargs to pipe to zfs destroy. Let's look at an example. I want to delete old bookstack snapshots with the offline-backups tag. I can find them easily with grep, let's apply the zfs list flags from above:

~# zfs list -H -o name -rt snapshot datapool/backup/appdata/bookstack | grep offline-backup
datapool/backup/appdata/bookstack@syncoid_offline-backup_hv-02_2020-11-12:19:09:58
datapool/backup/appdata/bookstack/mariadb@syncoid_offline-backup_hv-02_2020-11-12:19:10:08
datapool/backup/appdata/bookstack/webroot@syncoid_offline-backup_hv-02_2020-11-12:19:11:29

I've verified these are indeed the snapshots I'd like to delete. Let's include xargs -n1 zfs destroy to perform the deletion:

zfs list -H -o name -rt snapshot datapool/backup/appdata/bookstack | grep offline-backup | xargs -n1 zfs destroy

Mount Encrypted ZFS Datasets on Boot

By default in Ubuntu 20.04 Encrypted datasets are not mounted on boot.

A quick fix for this is done by modifying the zfs-mount.service:

sudo systemctl edit zfs-mount.service
## Paste in the following content:
[Service]
ExecStart=/sbin/zfs mount -l -a

Do it all... with Sanoid / Syncoid!

Learning ZFS has been fun, so I've wanted to try to do everything manually, but it turns out Sanoid is the end-all-be-all of everything ZFS I need. It'll handle your backups and replication automatically.

Syncoid commands

I have some encrypted datasets, which is a relatively new feature of ZFS and requires the raw send option. To do this with syncoid:

syncoid pool/to/backup bkppool/destination --sendoptions=w

Increasing the Pool After

I'm going to be creating a 4 disk pair of mirrors, but starting with 3 disks. I'm writing it down here so I won't forget!

Initial creation:

zpool create datapool /dev/disk3 mirror /dev/disk1 /dev/disk2

And adding later:

zpool attach datapool /dev/disk3 /dev/disk4

ZFS