• MasonLoringBliss
• ZFSandGELIbyHAND

This example assumes FreeBSD 13 or thereabouts, a pair of disks, and a desire to mirror everything. It accomodates both UEFI and legacy installs, and defaults to a partitioning scheme that can have both installs at once.

Unlike bsdinstall's default for ZFS, it doesn't set explicit mountpoints for datasets, but instead relies on inheritance.

While it's possible to use gmirror for the EFI boot partition, a notion lifted from CentOS, some UEFI firmware has been known to write files to the ESP, without awareness that it's writing to half of a mirror. While uncommon, this will result in your ESP mirror being inconsistent. Setting the mirror balance mode to "prefer" and causing a resync at boot might help work around this, but better still is the method adopted by Ubuntu 20.04, wherein the install maintains and updates multiple ESPs. As FreeBSD doesn't automate bootcode update, there's no scaffolding to be rewritten to support this, and this is the approach taken by the current version of this guide.

This set of instructions set up a pair of freebsd-boot partitions, a pair of ESP partitions, GELI-encrypted swap on a gmirror, and a GELI-encrypted root pool. (It costs almost nothing in disk space and is useful to accomodate both legacy and UEFI installs. The instructions are easy to adapt if you don't want legacy. An example of why you might want this: I have a SuperMicro system that corrupts the back-up GPT partition reliably in UEFI mode, but not in legacy mode.)

You might be interested in the InstallEnvironment I use for installs.

# We're clearing off the disks to start. This is a destructive operation.
gpart destroy -F ada0
gpart destroy -F ada1

# Set up new GPT partition tables.
gpart create -s gpt ada0
gpart create -s gpt ada1

# Add partitions. We'll access them by name everywhere.

# Note for futureproofing: The freebsd-boot partition should be smaller
# than 545 KB, but large enough to accomodate the second-stage bootstrap 
# code. /boot/gptzfsboot on my test FreeBSD 13.0 AMD64 system is 158,858 
# bytes, so we'll need something larger than that. 512k seems like a nicely
# natural number and it's what the FreeBSD installer uses.

# Boot partitions, first legacy and then UEFI/ESP:
gpart add -t freebsd-boot -s 512k -l boot0 ada0
gpart add -t freebsd-boot -s 512k -l boot1 ada1
gpart add -t efi -s 1m -l efi0 ada0
gpart add -t efi -s 1m -l efi1 ada1

# ESP must be vfat/fat32:
newfs_msdos /dev/gpt/efi0
newfs_msdos /dev/gpt/efi1

# Swap - these will go into a geom_mirror:
gpart add -t freebsd-swap -l swap0 -a 1m -s 4096m ada0
gpart add -t freebsd-swap -l swap1 -a 1m -s 4096m ada1

# CAVEAT: If you have mismatched disks, you'll want to make these have a
# matching size:
gpart add -t freebsd-zfs -l tank0 -a 1m ada0
gpart add -t freebsd-zfs -l tank1 -a 1m ada1

# Set up swap gmirror
# XXX Note that you don't need geom_mirror loaded to run gmirror, at least
# XXX as of 13.0-RELEASE, and we avoid BZ#230246 if we don't load it.
# kldload geom_mirror
gmirror label -v swap gpt/swap0 gpt/swap1

# Set up GELI
geli init -s 4096 -b -g gpt/tank0
geli init -s 4096 -b -g gpt/tank1
geli attach gpt/tank0
geli attach gpt/tank1

# Create the root pool.
zpool create -R /mnt -o cachefile=/tmp/zpool.cache -O mountpoint=/ \
    -O atime=off -O canmount=off -O compression=on \
    tank mirror gpt/tank0.eli gpt/tank1.eli
zfs create -o canmount=off -o mountpoint=none tank/ROOT
zfs create -o mountpoint=/ tank/ROOT/default
zpool set bootfs=tank/ROOT/default tank
zfs create tank/home
zfs create -o canmount=off tank/usr
zfs create tank/usr/local
zfs create tank/usr/obj
zfs create tank/usr/src
zfs create tank/usr/ports
zfs create tank/usr/ports/distfiles
zfs create -o canmount=off tank/var
zfs create tank/var/jail
zfs create tank/var/log
zfs create tank/var/tmp
zfs create tank/tmp

# Copy over the zpool cache so we'll have it on the running system.
mkdir -p /mnt/boot/zfs
cp /tmp/zpool.cache /mnt/boot/zfs

# Personal preference - I don't like /usr/home.
ln -s /home /mnt/usr/home

# Don't forget to take the option of a shell to finish configuration at the
# end of the install.
exit

The install will proceed.

When offered a post-install shell, take it, and:

cat >> /boot/loader.conf <<END

geom_eli_load="YES"
geom_mirror_load="YES"

zpool_cache_load="YES"
zpool_cache_name="/boot/zfs/zpool.cache"
zpool_cache_type="/boot/zfs/zpool.cache"

vfs.root.mountfrom="zfs:tank/ROOT/default"
END

cat >> /etc/rc.conf <<END
zfs_enable="YES"
zfsd_enable="YES"
END

cat >> /etc/fstab <<END
/dev/gpt/efi0 /boot/efi0 msdosfs rw,late 0 0
/dev/gpt/efi1 /boot/efi1 msdosfs rw,late 0 0
/dev/mirror/swap.eli none swap sw 0 0
END

# If /dev/ has been depopulated - I see this under 11+
rm /dev/null
mount -t devfs devfs /dev

# Make mountpoints and mount ESP:
mkdir /boot/efi0
mkdir /boot/efi1
mount -t msdosfs /dev/gpt/efi0 /boot/efi0
mount -t msdosfs /dev/gpt/efi1 /boot/efi1

# UEFI with fallback naming:
# If we have a box that's happy with the fallback naming, we can just do
# that. You won't be able to do this ''and'' have room for a named copy
# of the bootloader using efibootmgr, so please choose only one.
mkdir -p /boot/efi0/efi/boot
mkdir -p /boot/efi1/efi/boot
cp /boot/loader.efi /boot/efi0/efi/boot/bootx64.efi
cp /boot/loader.efi /boot/efi1/efi/boot/bootx64.efi

# UEFI with boot variables:
kldload efirt
mkdir -p /boot/efi0/efi/freebsd
mkdir -p /boot/efi1/efi/freebsd
cp /boot/loader.efi /boot/efi0/efi/freebsd/
cp /boot/loader.efi /boot/efi1/efi/freebsd/

# NOTE: efibootmgr syntax has changed. If this errors, you can
# use a GPT label to specify the specific device. For example:
#   -l gpt/efiboot0:/efi/freebsd/loader.efi
efibootmgr -c -a -L freebsd0 -l ada0p2:/efi/freebsd/loader.efi
efibootmgr -c -a -L freebsd1 -l ada1p2:/efi/freebsd/loader.efi

# Check to see what numbers were actually assigned first.
# efibootmgr -o 1,2

# Install legacy bootcode as well:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

# Note that the ordering command seems not to work, at least on some
# hardware. You might be forced to set boot order manually on first
# boot. Also note that the documentation is (was?) incorrect and shows
# the Linux tool's syntax. BZ#230871

# You should be good to go at this point. Variations on this can include
# setting up keys rather than passphrases for GELI. Also, note that you
# can create "degraded arrays" in gmirror by only specifying one device.
# This can be useful for initial set-up work where you intend to later
# add the other half of a mirror.

# At this point, you can also set up things like pkg or custom networking 
# and other bits of config you'd like to be there on first boot, and you 
# can do things like run "freebsd-update fetch install" and so forth. Don't 
# forget to install and set up cpu-microcode (formerly devcpu-data) for
# those microcode fixes!

exit

Special cases, older instructions, and things with a limited audience but worth preserving:

# if needed (legacy BIOS with GPT)
#gpart set -a lenovofix ada0
#gpart set -a lenovofix ada1

FreeBSD < 13.1 or thereabouts seems not to include this by default:

cat >> /boot/loader.conf <<END
zfs_load="YES"
END

Post-install shell, load aesni if we're on FreeBSD < 13:

cat >> /boot/loader.conf <<END
aesni_load="YES"
END

Not needed for some time, but in the past it was necessary to specify this to get prompted during the loader's run, as opposed to after the kernel has launched. This was problematic given some USB keyboard that weren't set up properly in time to enter the passphrase(s). It's now evidently the default no no longer needs to be specified. (Version?)

cat >> /boot/loader.conf <<END
geom_eli_passphrase_prompt="YES"
END

FreeBSD efibootmgr(8) syntax has changed:

# With FreeBSD < 13.0, instead of the previously noted syntax:
efibootmgr -a 1
efibootmgr -a 2

CategoryHowTo CategoryZfs