Switching a remote server from Linux to FreeBSD
How to switch a remote server from Linux to FreeBSD through ssh in a few easy steps. In this particular case, we want to retain the data, stored on ZFS in a two-way mirror pool zdata. The pool's providers are LUKS encrypted partitions, sda3 and sdb3 respectively, so our current set up is like this:
- sda1 - the bios boot partition, 1M
- sda2 - system root, mdadm md0 device
- sda3 - LUKS provider for the zdata pool
sdb mirrors sda. This process assumes that a single disc can hold our zdata data twice and still have room for zroot and the bootloader partition. So for example, this was done with 2x3TB HDDs and zdata pool holding some 400GB so there was plenty of room for transfer.
1. Disable any usage of the '''sdb''' disk
First let's drop sdb from the zdata pool and from md0 device, because we need to completely rework the sdb disk.
# zpool offline zdata /dev/mapper/sdb3_crypto # zpool detach zdata /dev/mapper/sdb3_crypto # zpool labelclear /dev/mapper/sdb3_crypto # mdadm /dev/md0 --fail /dev/sdb2 --remove /dev/sdb2
2. Repartition '''sdb''' for the new, temporary layout
With sdb unused now, we repartition it for the new, temporary, layout using parted, or any of the partitioning tools of linux. With parted:
parted -s /dev/sdb mklabel gpt parted -s /dev/sdb mkpart primary 40 512k parted -s /dev/sdb mkpart primary 1Mib 10GiB parted -s /dev/sdb mkpart primary 10GiB 1010GiB parted -s /dev/sdb mkpart primary 1010GiB 100%
So we created the bootloader partition, sdb1 (future ada1p1); the zroot partition sdb2 (future ada1p2); the zdata partition sdb3 (future ada1p3) and the ztransfer partition sdb4. The idea is to move data from current zdata pool into ztransfer on sdb (ada1). Reboot and reinstall FreeBSD using ada0p2 + ada1p2 as providers for zroot, ada0p3 and ada1p3 as providers for GELI for zdata, move the data from ztransfer to new zdata, kill ztransfer, expand ada0p3 and ada1p3, expand the GELI providers, expand the zdata pool.
So far the server has been online and zdata serving without interruption even in degraded state. Before we continue with FreeBSD installation, we need to actually save the data from the old zdata mirrored pool into temporary ztransfer. Doing this after we've removed sdb from the pool allows disk-to-disk transfer which is fastest, otherwise ZFS would load balance sda and sdb at reading from it, while writing to ztransfer on sdb basically halving the available bandwidth, as sdb will be both read from and written to.
zfs snapshot -r zdata@transfer zfs send -Rv zdata@transfer | zfs receive ztransfer
After this is done, the data has been copied over to the temporary ztransfer pool and we can proceed with reinstalling FreeBSD which will result in removal our existing root and zdata.
3. Reboot and install FreeBSD
Reboot into FreeBSD installation / rescue environment and perform a regular install through ssh with bsdinstall, or by unpacking the tarballs. We partition the ex sda, now ada0:
gpart destroy -F ada0 gpart create -s gpt ada0 gpart add -t freebsd-boot -s 512k -b 40 ada0 gpart add -t freebsd-zfs -s 10G -a 1M ada0 gpart add -t freebsd-zfs -s 1000G ada0
Note that we only create three partitions on ada0. p1 will be for the gpt bootloader, p2 will hold our zroot pool and p3 will hold our zdata pool. We currently create it to match the size of sdb3, which is temporarily 1T, but that is not strictly required as ZFS allows providers of different sizes in the mirror vdev, so technically ada0p3 can be full remaining disk size.
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 zpool create zroot mirror ada0p2 ada1p2 geli init -l 256 -s 4096 ada0p3 geli init -l 256 -s 4096 ada1p3 geli attach ada0p3 geli attach ada1p3 zpool create zdata mirror ada0p3.eli ada1p3.eli
So now we have the new zdata pool, already in a mirror vdev that can receive data back from the temporary ztransfer. At this point we need to run bsdinstall or use whatever method to install FreeBSD onto the prepared zroot and zdata.
Note: With FreeBSD 11 it is possible to boot from encrypted zroot, so having two pools may not be needed.
4. Reboot into installed FreeBSD and continue operations
After installing FreeBSD we can reboot into it and configure the server for its purpose. At this point the zdata pool only needs to receive data back from the temporary ztransfer pool, so let's get it back:
zfs send -Rv ztransfer@transfer | zfs receive zdata
When that is complete the server can resume operation. All that remains to be done is remove the temporary ztransfer pool and expand zdata to use full remaining disk space:
# ...
Total server downtime is time required to copy zdata over to ztransfer and back, plus time needed to partition and install FreeBSD and configure it for the wanted operation. One day, when ZFS allows creating degraded mirror pools, this operation can be reduced down to time required to reboot and reinstall FreeBSD, since in that case sdb can be partitioned and prepared with new (degraded) zdata, data copied over while the server is operational, without the requirement to use temporary ztransfer pool.