Warning - the following instructions will ravage your machine and then eat all the food in your refrigerator, and unlike the dwarves, they won't be kidding about breaking your dishes.


This example assumes FreeBSD 12 or thereabouts, a pair of disks, and a desire to mirror everything. It accomodates both UEFI and legacy installs.

Unlike bsdinstall's default for ZFS, it doesn't set explicit mountpoints for datasets, but instead relies on inheritance.

An available trick is the use of gmirror for the EFI boot partition, a notion lifted from CentOS. That said, a caveat: Some UEFI firmwares have been known to write files to the ESP. While uncommon, this would result in your ESP mirror being inconsistent. Setting the mirror balance mode to "prefer" and causing a resync at boot might help work around this. Better still, and the method adopted by Ubuntu 20.04, is to just maintain multiple ESPs.

This set of instructions leaves nothing on-disk but a mirrored ESP (if you're on UEFI), a GELI-encrypted swap, and a GELI-encrypted root pool. Recently I've decided that it costs almost nothing in disk space and is an amazing boon to accomodate both legacy and UEFI installs, so by default, I'd recommend partitioning both both in mind, and installing whichever loader is immediately appropriate at the end. (An example that might make you want this: I have a SuperMicro system that corrupts the back-up GPT partition reliably in UEFI mode, but not in legacy mode.)

You might be interested in the InstallEnvironment I use for installs.


# We're clearing off the disks to start. This is a destructive operation.
gpart destroy -F ada0
gpart destroy -F ada1

gpart create -s gpt ada0
gpart create -s gpt ada1

# if needed (legacy BIOS with GPT)
#gpart set -a lenovofix ada0
#gpart set -a lenovofix ada1

# Add partitions. We'll access them by name everywhere.

# Let's add partitions for both UEFI and legacy, for flexibility. We don't
# need to populate both, but having the partitions there will make
# migrations easier.

# Legacy:
gpart add -t freebsd-boot -s 128k -l boot0 ada0
gpart add -t freebsd-boot -s 128k -l boot1 ada1
# UEFI:
gpart add -t efi -s 1m -l efi0 ada0
gpart add -t efi -s 1m -l efi1 ada1
newfs_msdos /dev/gpt/efi0
newfs_msdos /dev/gpt/efi1

gpart add -t freebsd-swap -l swap0 -a 1m -s 8192m ada0
gpart add -t freebsd-swap -l swap1 -a 1m -s 8192m ada1

gpart add -t freebsd-zfs -l tank0 -a 1m ada0
gpart add -t freebsd-zfs -l tank1 -a 1m ada1

# Set up swap gmirror
kldload geom_mirror
gmirror label -v swap gpt/swap0 gpt/swap1

# There's a bug in GPT label handling. If you run into it, unrelated labels
# will disappear as others are consumed by gmirror. For instance, in this set,
# after consuming gpt/efi0 and gpt/efi1, I might find that gpt/swap1 has gone
# missing. If you hit this, you can set up your mirrors as /dev/ada0p1 rather
# than gpt/efi0, using this guide as an example. BZ#230246 To save a restart,
# construct as much of the swap mirror as you can at the end of install, and
# add the rest on first boot.

# Set up GELI
geli init -s 4096 -b -g gpt/tank0
geli init -s 4096 -b -g gpt/tank1
geli attach gpt/tank0
geli attach gpt/tank1

# Create the root pool.
zpool create -R /mnt -o cachefile=/tmp/zpool.cache -O mountpoint=/ \
    -O atime=off -O canmount=off -O compression=on \
    tank mirror gpt/tank0.eli gpt/tank1.eli
zfs create -o canmount=off -o mountpoint=none tank/ROOT
zfs create -o mountpoint=/ tank/ROOT/default
zpool set bootfs=tank/ROOT/default tank
zfs create tank/home
zfs create -o canmount=off tank/usr
zfs create tank/usr/local
zfs create tank/usr/obj
zfs create tank/usr/src
zfs create tank/usr/ports
zfs create tank/usr/ports/distfiles
zfs create -o canmount=off tank/var
zfs create tank/var/jail
zfs create tank/var/log
zfs create tank/var/tmp
zfs create tank/tmp

# Copy over the zpool cache so we'll have it on the running system.
mkdir -p /mnt/boot/zfs
cp /tmp/zpool.cache /mnt/boot/zfs

# Personal preference - I don't like /usr/home.
ln -s /home /mnt/usr/home

# Don't forget to take the option of a shell to finish configuration at the
# end of the install.
exit


The install will proceed.

When offered a post-install shell, take it, and:


cat > /boot/loader.conf <<END
aesni_load="YES"
zfs_load="YES"

geom_eli_load="YES"
#geom_eli_passphrase_prompt="YES"
geom_mirror_load="YES"

zpool_cache_load="YES"
zpool_cache_name="/boot/zfs/zpool.cache"
zpool_cache_type="/boot/zfs/zpool.cache"

vfs.root.mountfrom="zfs:tank/ROOT/default"
END

cat >> /etc/rc.conf <<END
zfs_enable="YES"
zfsd_enable="YES"
END

cat >> /etc/fstab <<END
/dev/gpt/efi0 /boot/efi0 msdosfs rw,late 0 0
/dev/gpt/efi1 /boot/efi1 msdosfs rw,late 0 0
END

cat >> /etc/fstab <<END
/dev/mirror/swap.eli none swap sw 0 0
END

# If /dev/ has been depopulated - I see this under 11+
rm /dev/null
mount -t devfs devfs /dev

# UEFI:
mkdir /boot/efi0
mkdir /boot/efi1
mount -t msdosfs /dev/gpt/efi0 /boot/efi0
mount -t msdosfs /dev/gpt/efi1 /boot/efi1

# UEFI with fallback naming:
# If we have a box that's happy with the fallback naming, we can just do that.
mkdir -p /boot/efi0/efi/boot
mkdir -p /boot/efi1/efi/boot
cp /boot/loader.efi /boot/efi0/efi/boot/bootx64.efi
cp /boot/loader.efi /boot/efi1/efi/boot/bootx64.efi

# UEFI:
kldload efirt
mkdir -p /boot/efi0/efi/freebsd
mkdir -p /boot/efi1/efi/freebsd
cp /boot/loader.efi /boot/efi0/efi/freebsd/
cp /boot/loader.efi /boot/efi1/efi/freebsd/
# ada0p2 if we include both partition schemes - ada0p1 if we didn't include legacy
efibootmgr -c -L freebsd0 -l ada0p2:/efi/freebsd/loader.efi
efibootmgr -c -L freebsd1 -l ada1p2:/efi/freebsd/loader.efi

# Check to see what numbers were actually assigned first.
# Under FreeBSD < 13:
efibootmgr -a 1
efibootmgr -a 2
# Under FreeBSD 13, the syntax has changed:
efibootmgr -a -b 1
efibootmgr -a -b 2
# Under 13 this might be unnecessary. Check.
efibootmgr -o 1,2

# Legacy:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

# Note that the ordering command seems not to work, at least on some hardware.
# You might be forced to set boot order manually on first boot. Also note that
# the documentation is incorrect and shows the Linux tool's syntax. BZ#230871

# You should be good to go at this point. Variations on this can include
# setting up keys rather than passphrases for GELI. Also, note that you
# can create "degraded arrays" in gmirror by only specifying one device.
# This can be useful for initial set-up work where you intend to later
# add the other half of a mirror.
exit


CategoryHowTo CategoryZfs

MasonLoringBliss/ZFSandGELIbyHAND (last edited 2021-07-28T17:10:04+0000 by MasonLoringBliss)