How to partition your drive using OpenZFS w/ GELI Encryption and continue using the FreeBSD Installer

Author: Jonathan Vasquez <jon@xyinn.org>
Last Updated: 2022-07-27-1230
Tested On: FreeBSD 14.0-CURRENT #0 main-n256882-8f733dabcc3: Fri Jul 22 08:31:37 UTC 2022     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

Preface

This small guide will show you how I install FreeBSD, still using the FreeBSD Installer, but manually partitioning my system with OpenZFS and GELI Encryption. This guide will use a single drive system and GPT + UEFI. This will be a 14.0-CURRENT install, so I will also be having a physical swap partition that is as big as my RAM in order to support core dumps. A swap file can't be used for this. I also won't be putting swap inside of ZFS since coming from ZFS on Linux, there's been a lot of hard crashes I've experienced in the past (even with optimizations), and lastly I'm not using any disk labels.

Adjust any of the information below according to your setup.

NOTE: If you are testing this in VirtualBox, be aware that your system might not boot due to VirtualBox's weirdness with BIOS/UEFI/GPT setups. In the past (13.0-RELEASE), I noticed that I was only able to install FreeBSD in BIOS mode, but I wasn't able to boot the system successfully afterwards since I received a zio_read: error 5. This error occurred even if I did a regular vanilla installation as well. If I tried to boot and install FreeBSD in EFI mode, it wouldn't boot the install iso at all. I had to install the system in BIOS Mode, and then once setup was complete, I switched back to EFI mode to actually boot the system. You can enable EFI mode by going into the VM settings and checking the Enable EFI (special OSEs Only) box. I'm not surprised given ZFS was primarily made for GPT layout, and MBR (in conjunction with UEFI) may have some issues depending on the hardware, and other variables. There may also be some weirdness with FreeBSD as well though. VirtualBox will most likely throw you into the its UEFI Interactive Shell (and not automatically boot into your UEFI environment). All you need to do is point the shell to your FreeBSD UEFI loader. You'll need to adjust this according to where your FAT32 partition is located:

Shell> FS0:\efi\boot\bootx64.efi

Based On: Previous Instructions for Unencrypted Mirrored Setup

Let the games begin <3

Download and Boot Installation Media

  1. Download FreeBSD, put it on a some device, and boot off of it. I'll be using a 14.0-CURRENT snapshot.
  2. Start up the installer and follow the steps as normal. Once you get to the Partitioning step, select Shell.

Partitioning

Find Drives

# sysctl kern.disks

or

# camcontrol devlist

I'll be using /dev/nvd0 as my main drive.

Wipe Partition Layout

# gpart destroy -F nvd0

Create Partition Layout

# gpart create -s gpt nvd0

# gpart add -t efi -s 512M nvd0
# gpart add -t freebsd-swap -s 32G nvd0
# gpart add -t freebsd-zfs nvd0

Create GELI encrypted device for our zpool

# geli init -g -s 4k nvd0p3
# geli attach nvd0p3

NOTE: The -g for init is critical since that's what will allow the bootloader to ask for the passphrase and allows booting from this encrypted device.

Create our zpool and corresponding datasets

In the following layout, we are keeping it simple and just having one dataset for our /, and one for our /home. The layout is compatible with Boot Environments. You can use either bectl for boot environment management, or you can manage it yourself by snapshotting and cloning. The / dataset will be found by the bootloader by reading the pool's bootfs property (which we will be setting soon). If you are managing it yourself, you can update the bootfs property in the future whenever you need to change it. You can also test out a boot environment temporarily by selecting the Boot Environments option in the bootloader and selecting your desired dataset under the tank/os hierarchy. You'll need at least 2 datasets under tank/os in order for the Boot Environments entry to be shown in the bootloader.

Lastly, your / datasets should not have a mountpoint set, since this can cause issues if multiple / datasets are attempted to be mounted at the same location. For the purposes of the installation, we'll specify a / mountpoint on our tank/os/main dataset so it can automatically mount it in the correct location and allow the installer to complete. Once the installation is complete, we'll unset it.

# zpool create \
-o ashift=12 \
-O compression=lz4 \
-O atime=off \
-m none \
-R /mnt \
tank \
nvd0p3.eli

# zfs create -o canmount=off tank/os
# zfs create -o canmount=noauto -o mountpoint=/ tank/os/main
# zfs mount tank/os/main
# zfs create -o mountpoint=/usr/home tank/home

NOTE: On FreeBSD, /home is a symlink to /usr/home.

Set the ZFS root dataset for booting

# zpool set bootfs=tank/os/main tank

Create and Mount the EFI directory

We'll create and mount our EFI directory so that when the FreeBSD installer extracts the base system, the EFI files will land at this location appropriately.

# mkdir -p /mnt/boot/efi
# newfs_msdos -F 32 -S 4096 -c 1 /dev/nvd0p1
# mount -t msdosfs /dev/nvd0p1 /mnt/boot/efi

Add fstab entries

We'll add our swap and efi partitions to fstab. The swap will be automatically encrypted per boot since we are specifying the .eli extension on it.

# vi /tmp/bsdinstall_etc/fstab

/dev/nvd0p1        /boot/efi    msdosfs    rw    2    2
/dev/nvd0p2.eli    none         swap       sw    0    0

Return and Continue Installation

Return back to the installer and continue the setup until you reach the Final Configuration step. Select Exit. The next step should be: Manual Configuration. Select Yes so that we can drop back into a chrooted environment of our soon-to-be system. We'll do some last minute configuration needed to ensure a successful boot.

Final Configuration

Enable GELI module loading

# vi /boot/loader.conf

# The following two should already be in here.
cryptodev_load="YES"
zfs_load="YES"

# Add GELI module loading.
geom_eli_load="YES"

# If you are using something lower than FreeBSD 13, you'll also need:
aesni_load="YES"

Enable ZFS script to load on start up

Enable ZFS in rc.conf so your datasets load correctly.

# vi /etc/rc.conf

zfs_enable="YES"

Unsetting the mountpoint on our tank/os/main dataset

You should now have a bootable system with your desired layout! We'll wrap up by unsetting the mountpoint property on our tank/os/main dataset. Since we are still inside the chroot environment, the dataset is being used and cannot be unset yet. Let's resolve this.

Go ahead and type exit but do not reboot. You should now be at the Complete step. Select Live CD and log in as root.

Now that we are in the Live CD, we can unset the mountpoint property we initially set during installation:

# zfs unmount tank/home
# zfs unmount tank/os/main
# zfs inherit mountpoint tank/os/main

If you now check your mountpoint properties, they should look like this:

# zfs get -r mountpoint tank

NAME          PROPERTY    VALUE           SOURCE
tank          mountpoint  none            local
tank/home     mountpoint  /mnt/usr/home   local
tank/os       mountpoint  none            inherited from tank
tank/os/main  mountpoint  none            inherited from tank

and the canmount properties should look like this:

# zfs get -r canmount tank

NAME          PROPERTY  VALUE     SOURCE
tank          canmount  on        default
tank/home     canmount  on        default
tank/os       canmount  off       local
tank/os/main  canmount  noauto    local

That's it! Type reboot, and have fun :).

Once you are logged into your system, you'll see everything worked and it will look something like this:

# zfs list

NAME           USED  AVAIL  REFER  MOUNTPOINT
tank           619M  43.0G    96K  none
tank/home       96K  43.0G    96K  /usr/home
tank/os        618M  43.0G    96K  none
tank/os/main   618M  43.0G   618M  none

References