Installing FreeBSD Root on ZFS (Mirror) using GPT

1. Creating a bootable ZFS Filesystem

  1. Boot FreeBSD install DVD or USB Memstick
  2. Choose Fixit option in sysinstall
  3. Create GPT Disks
     Fixit# gpart create -s gpt ad0
     Fixit# gpart create -s gpt ad1
  4. Create the boot, swap and zfs partitions

    Create 3 partitions on both drives ad0 and ad1. The first partition contains the gptzfsboot loader which is able to recognize and load the loader from a ZFS partition. The second partition is a 4 GB swap partition. The third partition is the partition containing the zpool (60GB).

     Fixit# gpart add -b 34 -s 512k -t freebsd-boot ad0
     Fixit# gpart add -s 4G -t freebsd-swap -l swap0 ad0
     Fixit# gpart add -s 60G -t freebsd-zfs -l disk0 ad0
     Fixit# gpart add -b 34 -s 512k -t freebsd-boot ad1
     Fixit# gpart add -s 4G -t freebsd-swap -l swap1 ad1
     Fixit# gpart add -s 60G -t freebsd-zfs -l disk1 ad1

    Note:

    1. While a ZFS Swap Volume can be used instead of the freebsd-swap partition, crash dumps can't be created on the ZFS Swap Volume.

    2. Sizes and offsets are specified in sectors by default (1 sector is typically 512 bytes) but can have 'M' or 'G' suffixes for MB and GB.
  5. Install the Protective MBR (pmbr) and gptzfsboot loader to both drives
     Fixit# gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ad0
     Fixit# gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ad1
    This may fail with an "operation not permitted" error message, since the kernel likes to protect critical parts of the disk. If this happens for you, run:
     Fixit# sysctl kern.geom.debugflags=0x10
  6. Load ZFS kernel module
     Fixit# kldload /mnt2/boot/kernel/opensolaris.ko
     Fixit# kldload /mnt2/boot/kernel/zfs.ko
  7. Create ZFS Pool zroot
     Fixit# mkdir /boot/zfs
     Fixit# zpool create zroot mirror /dev/gpt/disk0 /dev/gpt/disk1
     Fixit# zpool set bootfs=zroot zroot

    Note:

    • zroot is the name of the ZFS Pool, it could be anything (i.e. tank, data,...)

2. Installing FreeBSD to the ZFS filesystem

  1. Create ZFS filesystem hierarchy

    The fletcher4 algorithm should be more robust than the fletcher2 algorithm.

     Fixit# zfs set checksum=fletcher4                                      zroot
     Fixit# zfs create -o compression=on    -o exec=on      -o setuid=off   zroot/tmp
     Fixit# chmod 1777 /zroot/tmp
     Fixit# zfs create                                                      zroot/usr
     Fixit# zfs create                                                      zroot/usr/home
     Fixit# cd /zroot ; ln -s /usr/home home
     Fixit# zfs create -o compression=lzjb                  -o setuid=off   zroot/usr/ports
     Fixit# zfs create -o compression=off   -o exec=off     -o setuid=off   zroot/usr/ports/distfiles
     Fixit# zfs create -o compression=off   -o exec=off     -o setuid=off   zroot/usr/ports/packages

    Note:

    If you use nullfs or nfs to mount /usr/ports to different locations/servers, you will also need to nullfs/nfs mount /usr/ports/distfiles and/or /usr/ports/packages.

     Fixit# zfs create -o compression=lzjb  -o exec=off     -o setuid=off   zroot/usr/src
     Fixit# zfs create                                                      zroot/var
     Fixit# zfs create -o compression=lzjb  -o exec=off     -o setuid=off   zroot/var/crash
     Fixit# zfs create                      -o exec=off     -o setuid=off   zroot/var/db
     Fixit# zfs create -o compression=lzjb  -o exec=on      -o setuid=off   zroot/var/db/pkg
     Fixit# zfs create                      -o exec=off     -o setuid=off   zroot/var/empty
     Fixit# zfs create -o compression=lzjb  -o exec=off     -o setuid=off   zroot/var/log
     Fixit# zfs create -o compression=gzip  -o exec=off     -o setuid=off   zroot/var/mail
     Fixit# zfs create                      -o exec=off     -o setuid=off   zroot/var/run
     Fixit# zfs create -o compression=lzjb  -o exec=on      -o setuid=off   zroot/var/tmp
     Fixit# chmod 1777 /zroot/var/tmp

    Note:

    1. Compression may be set to on, off, lzjb, gzip, gzip-N (where N is an integer from 1 (fastest) to 9 (best compression ratio. gzip is equivalent to gzip-6).
    2. Compression will cause some latency when accessing files on the ZFS filesystems. Use compression on ZFS filesystems which will not be accessed that often.

  2. Install FreeBSD to zroot
     Fixit# cd /dist/8.0-*
     Fixit# export DESTDIR=/zroot
     Fixit# for dir in base catpages dict doc games info lib32 manpages ports; \
              do (cd $dir ; ./install.sh) ; done
     Fixit# cd src ; ./install.sh all
     Fixit# cd ../kernels ; ./install.sh generic
     Fixit# cd /zroot/boot ; cp -Rlp GENERIC/* /zroot/boot/kernel/
  3. Make /var/empty readonly
     Fixit# zfs set readonly=on zroot/var/empty
  4. chroot into /zroot
     Fixit# chroot /zroot
  5. Create /etc/rc.conf
     Fixit# echo 'zfs_enable="YES"' > /etc/rc.conf
     Fixit# echo 'hostname="beastie.mydomain.local"' >> /etc/rc.conf
     Fixit# echo 'ifconfig_re0="DHCP"' >> /etc/rc.conf

    Note:

    • Replace re0 with the name of the Network interface for the new system
  6. Create /boot/loader.conf
     Fixit# echo 'zfs_load="YES"' > /boot/loader.conf
     Fixit# echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf
  7. Change root's password
     Fixit# passwd
  8. Set the local time zone
     Fixit# tzsetup
  9. Create /etc/mail/aliases.db
     Fixit# cd /etc/mail
     Fixit# make aliases
  10. Exit from the /zroot
     Fixit# umount /dev
     Fixit# exit
  11. Install zpool.cache to the ZFS filesystem
     Fixit# cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache

3. Finish install

  1. Using swap

    There are 2 ways to use the gpt/swap0 and gpt/swap1 partitions

    1. Create /etc/fstab to use both swap partitions
       Fixit# cat << EOF > /zroot/etc/fstab
       # Device                       Mountpoint              FStype  Options         Dump    Pass#
       /dev/gpt/swap0                 none                    swap    sw              0       0
       /dev/gpt/swap1                 none                    swap    sw              0       0
       EOF
    2. Use gmirror to mirror the swap partitions

      1. Load gmirror kernel module
         Fixit# kldload /mnt2/boot/kernel/geom_mirror.ko
      2. Create the gmirror swap
         Fixit# gmirror label -b prefer swap gpt/swap0 gpt/swap1

        Note:

        The 'prefer' balance algorithm can be replaced by 'round-robin'. See the gmirror(8) man page about problem using the 'round-robin' balance algorithm and kernel dumps

      3. Create /etc/fstab to use the gmirror swap partition
         Fixit# cat << EOF > /zroot/etc/fstab
         # Device                       Mountpoint              FStype  Options         Dump    Pass#
         /dev/mirror/swap                       none                    swap    sw              0       0
         EOF
      4. Add gmirror to /boot/loader.conf
         Fixit# echo 'geom_mirror_load="YES"' >> /zroot/boot/loader.conf
  2. export LD_LIBRARY_PATH
     Fixit# export LD_LIBRARY_PATH=/mnt2/lib 
  3. Unmount all zfs filesystems
     Fixit# zfs unmount -a
  4. Change mount points for zroot pool
     Fixit# zfs set mountpoint=legacy zroot
     Fixit# zfs set mountpoint=/tmp zroot/tmp
     Fixit# zfs set mountpoint=/usr zroot/usr
     Fixit# zfs set mountpoint=/var zroot/var
  5. Exit Fixit mode and sysinstall. Remove the FreeBSD install DVD/Memstick and the system will boot using the ZFS root.

RootOnZFS/GPTZFSBoot/Mirror (last edited 2017-02-09T12:21:26+0000 by MiroslavLachman)