How to partition your drive with OpenZFS and continue using the FreeBSD Installer

Author: Jonathan Vasquez <jon@xyinn.org>
Last Updated: 2022-04-29-1600
Tested On: FreeBSD 13-RELEASE (VirtualBox 6.1.22 r144080)

Deprecated

This guide is deprecated. You should use the newer version. I'm still keeping this one available to publicly document some of the mirroring layout steps. You can pick and choose steps from both guides according to your needs.

Preface

This small guide will show you how I use the FreeBSD Installer as usual, but set up my drives with ZFS manually for the installation. This will use UEFI as well w/ GPT. I will be setting up a RAID10 configuration, where the swap partition is encrypted and outside of ZFS. After coming from ZFS on Linux, and knowing the problems (crashes :() of using a zvol to be used as swap (Even with optimizations), I've read FreeBSD has similar problems so I'm avoiding putting swap on there still. I will use gmirror to mirror the swap so we can utilize all of the drives. You could technically also just add each swap partition to fstab as well to have a bigger swap (from what I've read). I will also mirror the FAT32 based EFI partition as well.

NOTE: When I was testing all of this in VirtualBox, I noticed that I was only able to install FreeBSD in BIOS mode, but I wasn't able to boot the system successfully afterwards since I received a zio_read: error 5. This error occurred even if I did a regular vanilla installation of FreeBSD 13 as well. If I tried to boot and install FreeBSD in EFI mode, it wouldn't boot the install iso at all. I had to install the system in BIOS Mode, and then once setup was complete, I switched back to EFI mode to actually boot the system. You can enable EFI mode by going into the VM settings and checking the Enable EFI (special OSEs Only) box. I'm not surprised given ZFS was primarily made for GPT layout, and MBR seems to usually have some issues depending on the hardware, and other variables. This may also be some weirdness with FreeBSD as well though.

Jason Tubnor also seems to have gotten similar results regarding BIOS vs UEFI systems.

Let the games begin ;D

Download and Boot

Download FreeBSD and put it on a some device, and boot off of it.
Start up the installer and follow the steps as normal. Once you get to the partitioning step, select Shell.

Find the drives you need

You can use any of the following:

# sysctl kern.disks

or

# camcontrol devlist

Partitioning

Adjust the drives to your particular setup

Setup Partitions

Wipe information on drives if needed.

# gpart destroy -F ada0      (repeat for each drive)

Create layout

I'm not using labels in my current set up so I will create the layout once and replicate it with gpart backup | gpart restore. I will then add the efi partition, swap, and zfs. Afterwards we will format the efi partition and copy the loader.

(See the Extra section at the end for some steps on Non-UEFI setups)

# gpart create -s gpt ada0

# gpart add -t efi -s 512M ada0
# gpart add -t freebsd-swap -s 4G ada0
# gpart add -t freebsd-zfs ada0

Replicate layout

# gpart backup ada0 | gpart restore -F ada1 ada2 ada3 ada4 ada5

Create mirrors for the EFI and swap partitions

# gmirror load
# gmirror label -v -b round-robin efi \
/dev/ada0p1 /dev/ada1p1 \
/dev/ada2p1 /dev/ada3p1 \
/dev/ada4p1 /dev/ada5p1

# newfs_msdos -F 32 -c 1 /dev/mirror/efi
# mkdir /tmp/efi
# mount -t msdosfs /dev/mirror/efi /tmp/efi
# mkdir -p /tmp/efi/efi/boot
# cp /boot/loader.efi /tmp/efi/efi/boot/bootx64.efi

Create the mirror for the swap as well.

# gmirror label -v -b round-robin swap \
/dev/ada0p2 /dev/ada1p2 \
/dev/ada2p2 /dev/ada3p2 \
/dev/ada4p2 /dev/ada5p2

Add swap and efi mirror info to fstab

Add the swap/efi to fstab. We will use the .eli extension on the swap so FreeBSD will automatically encrypt the swap on each boot.

# vi /tmp/bsdinstall_etc/fstab

/dev/mirror/swap.eli none swap sw 0 0
/dev/mirror/efi /boot/efi msdosfs rw 0 0

Creating your zpool and corresponding datasets

# zpool create \
-o ashift=12 \
-O compression=on \
-O atime=off \
-m none \
-R /mnt \
tank \
mirror ada0p3 ada1p3 \
mirror ada2p3 ada3p3 \
mirror ada4p3 ada5p3 

# zfs create -o mountpoint=/ tank/os
# zfs create -o mountpoint=/usr/home tank/home
# zfs create -o mountpoint=/usr tank/usr
# zfs create -o mountpoint=/usr/ports tank/ports
# zfs create -o mountpoint=/var tank/var

Set ZFS root dataset for booting

# zpool set bootfs=tank/os tank

Return back to the installer

Return back to the installer and continue the setup until the end.

# exit

Before you exit the installer, at the last window labeled Complete, select the Live CD option so we can drop back into the shell and finish adding some last minute configuration.

Last minute configuration

Your actual system will still be mounted at /mnt.

Enable the zfs script to load on start up

Enable ZFS in rc.conf so your datasets load correctly.

# vi /mnt/etc/rc.conf

zfs_enable="YES"

Add required modules to be loaded on boot

We will add geom_mirror module to the file so that the module is loaded on boot.

# vi /mnt/boot/loader.conf

geom_mirror_load="YES"

You can also check the file and make sure that the installer added zfs_load="YES", if it didn't, add that line as well.

That's it! You should now have a bootable system with your desired layout.

Go ahead and type reboot. Enjoy ;).

Extra

If you are not doing a UEFI install, then you'll need to substitute the efi related steps for the following:

# gpart add -t freebsd-boot -s 512k ada0

Install bootcode to each of the drives that can be used for boot:

# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0