Installing FreeBSD Root on ZFS (RAIDZ1) using GPT

Contents

  1. Creating a bootable ZFS Filesystem
  2. Installing FreeBSD to the ZFS filesystem
  3. Finish install

1. Creating a bootable ZFS Filesystem

  1. Boot FreeBSD install CD/DVD
  2. Choose Fixit option in sysinstall
  3. Create GPT Disks 
     Fixit# gpart create -s gpt ad0
 Fixit# gpart create -s gpt ad1
 Fixit# gpart create -s gpt ad2
  4. Create the boot, swap and zfs partitions

    Create 3 partitions on drives ad0, ad1 and ad2. The first partition contains the gptzfsboot loader which is able to recognize and load the loader from a ZFS partition. The second partition is a 4 GB swap partition. The third partition is the partition containing the zpool (60GB). 

     Fixit# gpart add -b 34 -s 128 -t freebsd-boot ad0
 Fixit# gpart add -b 162 -s 8388608 -t freebsd-swap -l swap0 ad0
 Fixit# gpart add -b 8388770 -s 125829120 -t freebsd-zfs -l disk0 ad0
     Fixit# gpart add -b 34 -s 128 -t freebsd-boot ad1
 Fixit# gpart add -b 162 -s 8388608 -t freebsd-swap -l swap1 ad1
 Fixit# gpart add -b 8388770 -s 125829120 -t freebsd-zfs -l disk1 ad1
     Fixit# gpart add -b 34 -s 128 -t freebsd-boot ad2
 Fixit# gpart add -b 162 -s 8388608 -t freebsd-swap -l swap2 ad2
 Fixit# gpart add -b 8388770 -s 125829120 -t freebsd-zfs -l disk2 ad2

    Note:

    1. While a ZFS Swap Volume can be used instead of the freebsd-swap partition, crash dumps can't be created on the ZFS Swap Volume.

    2. Sizes and offsets are specified in sectors (1 sector is typically 512 bytes).

  5. Set partition active (Optional)

    Some BIOSes might require setting a Slice active. Here we set partition 1 as the active partition. 

     Fixit# gpart set -a active -i 1 ad0

    Note:

    • This is no longer required on 9-CURRENT, as installing the bootcode will mark the partition active.
  6. Install the Protected MBR (pmbr) and gptzfsboot loader to both drives 
     Fixit# gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ad0
 Fixit# gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ad1
 Fixit# gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ad2
    This may fail with an "operation not permitted" error message, since the kernel likes to protect critical parts of the disk. If this happens for you, run: 
     Fixit# sysctl kern.geom.debugflags=0x10
  7. Load ZFS kernel module 
     Fixit# kldload /mnt2/boot/kernel/opensolaris.ko
 Fixit# kldload /mnt2/boot/kernel/zfs.ko
  8. Create ZFS Pool zroot 
     Fixit# mkdir /boot/zfs
 Fixit# zpool create zroot raidz1 /dev/gpt/disk0 /dev/gpt/disk1 /dev/gpt/disk2
 Fixit# zpool set bootfs=zroot zroot

    Note:

    1. zroot is the name of the ZFS Pool, it could be anything (i.e. tank, data, ...)
    2. A raidz1 pool requires a minimum of 3 disks and a maximum of 9 disks

2. Installing FreeBSD to the ZFS filesystem

  1. Create ZFS filesystem hierarchy

    The fletcher4 algorithm should be more robust than the fletcher2 algorithm. 

     Fixit# zfs set checksum=fletcher4                                      zroot
     Fixit# zfs create -o compression=on    -o exec=on      -o setuid=off   zroot/tmp
 Fixit# chmod 1777 /zroot/tmp
     Fixit# zfs create                                                      zroot/usr
 Fixit# zfs create                                                      zroot/usr/home
 Fixit# cd zroot ; ln -s /usr/home home
 Fixit# zfs create -o compression=lzjb                  -o setuid=off   zroot/usr/ports
 Fixit# zfs create -o compression=off   -o exec=off     -o setuid=off   zroot/usr/ports/distfiles
 Fixit# zfs create -o compression=off   -o exec=off     -o setuid=off   zroot/usr/ports/packages
 Fixit# zfs create -o compression=lzjb  -o exec=off     -o setuid=off   zroot/usr/src
     Fixit# zfs create                                                      zroot/var
 Fixit# zfs create -o compression=lzjb  -o exec=off     -o setuid=off   zroot/var/crash
 Fixit# zfs create                      -o exec=off     -o setuid=off   zroot/var/db
 Fixit# zfs create -o compression=lzjb  -o exec=on      -o setuid=off   zroot/var/db/pkg
 Fixit# zfs create                      -o exec=off     -o setuid=off   zroot/var/empty
 Fixit# zfs create -o compression=lzjb  -o exec=off     -o setuid=off   zroot/var/log
 Fixit# zfs create -o compression=gzip  -o exec=off     -o setuid=off   zroot/var/mail
 Fixit# zfs create                      -o exec=off     -o setuid=off   zroot/var/run
 Fixit# zfs create -o compression=lzjb  -o exec=on      -o setuid=off   zroot/var/tmp
 Fixit# chmod 1777 /zroot/var/tmp

    Note:

    1. Compression may be set to on, off, lzjb, gzip, gzip-N (where N is an integer from 1 (fastest) to 9 (best compresion ratio. gzip is equivalent to gzip-6).

    2. Compression will cause some latency when accessing files on the ZFS filesystems. Use compression on ZFS filesystems which will not be accessed that often.

  2. Install FreeBSD to zroot 
     Fixit# cd /dist/8.0-*
 Fixit# export DESTDIR=/zroot
 Fixit# for dir in base catpages dict doc games info lib32 manpages ports; \
          do (cd $dir ; ./install.sh) ; done
 Fixit# cd src ; ./install.sh all
 Fixit# cd ../kernels ; ./install.sh generic
 Fixit# cd /zroot/boot ; cp -Rlp GENERIC/* /zroot/boot/kernel/
  3. Make /var/empty readonly 
     Fixit# zfs set readonly=on zroot/var/empty
  4. chroot into /zroot 
     Fixit# chroot /zroot
  5. Create /etc/rc.conf 
     Fixit# echo 'zfs_enable="YES"' > /etc/rc.conf
 Fixit# echo 'hostname="beastie.mydomain.local"' >> /etc/rc.conf
 Fixit# echo 'ifconfig_re0="DHCP"' >> /etc/rc.conf

    Note:

    • Replace re0 with the name of the Network interface for the new system
  6. Create /boot/loader.conf 
     Fixit# echo 'zfs_load="YES"' > /boot/loader.conf
 Fixit# echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf
  7. Install ZFS aware /boot/loader 
     Fixit# echo 'LOADER_ZFS_SUPPORT=YES' > /etc/src.conf
 Fixit# mount -t devfs devfs /dev
 Fixit# export DESTDIR=""
 Fixit# cd /usr/src/sys/boot/
 Fixit# make obj
 Fixit# make depend
 Fixit# make
 Fixit# cd i386/loader
 Fixit# make install

    Note:

    This step can be skipped on FreeBSD 9.0-CURRENT as of SVN revision r199714. A change was made to zfsboot and zfsgptboot so that they will now load a zfs aware boot loader (/boot/zfsloader) instead of the traditional /boot/loader.

  8. Change root's password 
     Fixit# passwd
  9. Set the local time zone 
     Fixit# tzsetup
  10. Create /etc/mail/aliases.db 
     Fixit# cd /etc/mail
 Fixit# make aliases
  11. Exit from the /zroot 
     Fixit# umount /dev
 Fixit# exit
  12. Install zpool.cache to the ZFS filesystem 
     Fixit# cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache

3. Finish install

  1. Create /etc/fstab 
     Fixit# cat << EOF > /zroot/etc/fstab
 # Device                       Mountpoint              FStype  Options         Dump    Pass#
 /dev/gpt/swap0                 none                    swap    sw              0       0
 /dev/gpt/swap1                 none                    swap    sw              0       0
 /dev/gpt/swap2                 none                    swap    sw              0       0
 EOF
  2. export LD_LIBRARY_PATH 
     Fixit# export LD_LIBRARY_PATH=/mnt2/lib
  3. Unmount all zfs filesystems 
     Fixit# zfs unmount -a
  4. Change mount points for zroot pool 
     Fixit# zfs set mountpoint=legacy zroot
 Fixit# zfs set mountpoint=/tmp zroot/tmp
 Fixit# zfs set mountpoint=/usr zroot/usr
 Fixit# zfs set mountpoint=/var zroot/var
  5. Exit Fixit mode, and then sysinstall. Remove the FreeBSD install CD and the system will boot using the ZFS root.

RootOnZFS/GPTZFSBoot/RAIDZ1 (last edited 2009-11-04 03:07:57 by ScotHetzel)