FreeBSD Containers and Orchestration
FreeBSD introduced its container, OS-level virtualization primitive in 1999 in the form of a security-oriented isolation framework and subsystem called Jails. Similar to OpenVZ Containers in 2005, Solaris Zones, LXC, Docker and other implementations, FreeBSD Jails allow isolation of applications or entire stacks with their own processes, filesystems and users, whilst using the same host operating system kernel.
Contents
Container Tools: Base System
Out of the box, FreeBSD provides:
Container Tools: Third-Party
These third-party tools aim to simplify and speed up the process of creating and managing FreeBSD jail-based containers on individual hosts, and many include additional support and integration for FreeBSD features such as bhyve virtualization, ZFS, Virtual Networks (VNET), Templating, Import/Export among others.
AppJail
Jail framework written in C and posix shell to create, deploy and maintain jail-based containers.
- First Release: 2022
- Latest Release: 2024 (3.4.0)
Tools:
Supports:
- Supervisor (healthcheckers)
Parallel startup (Healthcheckers, jails & NAT).
- ZFS support.
- RACCT/RCTL support.
- NAT support.
- Port forwarding.
- IPv4 and IPv6 support.
- DHCP and SLAAC support.
- Virtual networks.
- Bridge support.
- VNET support.
- Text file to make jails: Makejail.
- Netgraph support.
LinuxJails support.
- Supports thin and thick jails.
TinyJails - Experimental feature to create a very stripped down jail that is very useful to distribute.
- Startup order control.
- Jail dependency support.
InitScript for interactive use of jails.
Commands to import/export jails (ZFS & Tarballs).
Table interface to easily integrate AppJail with scripts.
- Images.
- Dynamic DEVFS ruleset management.
Orchestration with AppJail Director & LittleJet.
podman, buildah
A FreeBSD port of the https://github.com/containers stack. Install sysutils/podman-suite from FreeBSD Ports/Packages.
- Suitable for testing and evaluation.
- Container storage using the zfs and vfs storage drivers. ZFS is strongly preferred since its use of snapshots and clones makes it more efficient than vfs.
- Podman provides a CLI which is a drop-in replacement for docker.
Supports docker-style networking using a port of https://github.com/containernetworking/plugins.
- Container images use the same formats and infrastructure as containerd and can be shared between the two implementations.
runj / containerd / nerdctl
Experimental FreeBSD Jail execution runtime implementation and FreeBSD OCI specification development.
First Release: 20211
- Latest Release:
- runj: 2024 (0.1.0)
- containerd: 2024 (1.7.22)
- nerdctl: 2024 (1.7.7)
- nerdctl provides a Docker-compatible CLI for containerd
Supports:
containerd support2
- Linux Jail support
Network Support (Experimental)
Bastille
Command-line (shell) tool and automation framework for jail-based containers.
First Release: 2018 3
- Latest Release: 2023 (0.10.2023*)
Supports:
- FreeBSD Features: ZFS, VNET
- Other Features: Template Creation, Import, Export
- Has a container / template registry
pot
First Release: 2018 4
- Latest Release: 2024 (0.16.0)
Command-line (shell) tool for jail-based containers.
Supports:
cbsd
Command-line and TUI (shell) tool for jail-based containers.
First Release: ~2013 6
- Last Release: 2024 (14.1.1)
Supports:
FreeBSD Features: ZFS, VNET, bhyve, Xen
- Other Features: Templates and Profiles
None of the above orchestration tools have a "run a plain vanilla linux inside the container" feature, but the start/stop/templating feature of them does not prevent a manually created plain-vanilla-linux-jail.
iocage
Command-line (Python, originally shell) tool for jail-based containers.
First Release: ~2016 7
- Latest Release: 2019 (1.2)
Supports:
- FreeBSD Features: ZFS (required), VNET
- Other Features: Template Creation, Import, Export
ioc
Re-implementation (Python) of iocage.
First Release: Unknown, but ~2017 7
- Latest Release: 2019 (0.8.2)
iocell
Fork of original (shell) iocage
First Release: ~2016 ("v2.0.0") 7
- Latest Release: 2017 (2.1.2)
ezjail
Command-line (shell) tool for jail-based containers.
First Release: 2005 8
- Latest Release: 2015 (3.4.2)
Supports:
- FreeBSD Features: ZFS, VNET (via manual scripting)
- Other Features: Template Creation, Import, Export
Container Orchestration
The container tools above can be used as a base for multi-host orchestration, by exporting containers from one host, and either moving the container to another host, or by utilising shared storage (eg: SAN) or ZFS to switch container datasets from one host to another and importing it on the target host.
Additionally the following FreeBSD container orchestration tools are available:
HashiCorp Nomad (jail-task driver)
Container Orchestration Distributions
XXX: add some FreeBSD derived distributions which have their focus (not only) on container virtualisation.
CloneOS - WebUI for bhyve, jails, xen, ...
https://samuel.karp.dev/blog/2021/03/runj-a-new-oci-runtime-for-freebsd-jails/ (1)
https://samuel.karp.dev/blog/2021/05/running-freebsd-jails-with-containerd-1-5/ (2)
See pot homepage, sysutils/nomad and sysutils/nomad-pot-driver (5)
If someone can identify an authoritative initial history/announcement, please update (7 8 9)