/ToDo /VNET |
Jails
Introduction
Jails were introduced in FreeBSD 4.x by Poul-Henning Kamp
More information:
Contents
Patches
Feature |
Status |
Author |
Description |
CPU + RAM limits |
Not fully working / stalled |
Jails can now have the amount of memory available to their processes' resident sets (RSS) |
|
Multi-IPv4/v6/no-IP jails |
done / commited |
Bjoern A. Zeeb |
As an alternate solution to full network stack virtualization, this work shall provide a lightweight solution for multi-IP virtualization. Perforce status overview SVN 188281 |
Wildcard IP (INADDR_ANY) should not bind inside a jail |
done / stalled |
Frank Behrens |
If there is a jail, the assigned IP address should not considered as valid address for INADDR_ANY in host. With this change an easy jail setup is possible without further modifications of hosts base configuration! PR 84215 patch |
jexec - selection by jail name |
done / commited |
Frank Behrens |
jexec(8) needs the numeric id of jail. This id is not constant and changes on jail restarts. Therefore it is desirable to select a jail by it's name. The attached patch is a sample implementation. PR 119305 patch |
Set priority in rc.d |
done / not commited |
Jan Srzednicki |
A simple rc.d jail patch to enable priority patch |
Do not keep JID incrementing forever |
done / commited / reverted |
Ed Schouten |
Jail numbers are incrementing forever, this makes jls/ps output a lot easier. No more 4-5 digit numbers in your test setup, where you only have 10-20 jails. PR 122270 |
Per prison process count limit |
done / stalled |
Alex Lyashkov |
Done as part of FreeVPS |
Separated UID hash |
done / stalled |
Alex Lyashkov |
Done as part of FreeVPS |
Separated SysV IPC |
done / stalled |
Alex Lyashkov |
Done as part of FreeVPS |
File handles usage limit |
done / stalled |
Alex Lyashkov |
Done as part of FreeVPS |
Hierarchical jails |
work in progress |
Jamie Gritton |
Hierarchical jails under the new framework (jails inside jail) |
Future plans for FreeBSD jails / virtualization
If you have any idea or request, please address the freebsd-jail maililng list.
freebsd-arch mailing list discussions:
kernel level virtualisation requirements (2007-10-11)
kernel level virtualisation requirements / quota system changes (2007-10-14)
kernel level virtualisation requirements / list of ideas (2007-10-18)
Well known or frequent requests, some of these will never be done with jails, some are (or will be) done with alternative technologies:
- be able to specify a different root for the virtual machine
be able to specify a different output for "uname" jail(8) (r279361 )
- be able to specify a different network address
be able to specify a different routing table setfib(1)
- be able to specify a completely different network universe
- be able to dedicate an interface to it
- be able to have a separate PID space for it
- be able to specify a separate UID space for it
- be able to specify a CPU maximum quota
be able to confine it to some set of CPUs cpuset(1)
- be able to have different mount tables for it
be able to specify a different security level for it [sysctl kern.securelevel is already per jail]
- be able to specify a disk IO limits for it
SNMP monitoring of jail resources port by Stef Walter http://memberwebs.com/stef/software/bsnmp-jails/ (broken)
- exposure of certain sysctl values inside jails
wildcard ip (INADDR_ANY) should not bind inside a jail PR 84215 this is known problem from beginning of the jails and is stated in manpage section bugs jail(8) [fixed by Multi-IPv4/v6/no-IP jails patch]
Unprivileged jail_attach / a way to do jail_attach as non-root; Done as port sysutils/jailme by Bill Moran - jailme
Hierarchical jails (Multilevel jails) mailing list discussion patches by Jamie Gritton: jh jhu
Netgraph-based networking for jails
Introduction
A few links to other resources:
Using jails with if_bridge(4) and epair(4) (Mason Loring Bliss)
Sample (and simple) VNET configurations in shell (Bjoern Zeeb)
There are three ways to do networking with jails: (i) using IP addresses (aliases) from the host, (ii) using if_bridge(4) and epair(4) interfaces, and (iii) using netgraph. I will start this how-to with the third method as the second method has already been explained by Mason Loring Bliss.
Before we kick off, we need to understand some new terms that netgraph introduces.
- Node
- Any object that can connect to another object. This can be a bridge, a virtual interface, but also a VLAN interface or a BPF object.
- Edge
- The connection between two nodes. You could consider it to be a virtual network cable.
- Hook
In netgraph, edges do not really exist per se. Instead, an edge is simply an association of two hooks, one from each node. A node's hooks define how that node can be connected. For a bridge node, the hooks are the virtual interfaces, though for a virtual interface node, hooks can be each protocol supported by that interface.
- Path
Every netgraph node is addressable via an ASCII string called a node address or path. Many nodes have names. When a node has a name, it can always be addressed usingthe absolute address consisting of the name followed by a colon. If a node does not have a name, you can construct one from the node's unique ID number by enclosing the number in square brackets. Relative addressing is also possible when two nodes are indirectly connected. A relative address uses the names of consecutive hooks on the path from the source node to the target node. Relative and absolute addressing can be combined.
This is quite confusing. See the example at https://people.freebsd.org/~julian/netgraph.html.
Let's start off with a small network consisting of two jails, called "test1" and "test2", which are bridged to an new, virtual interface on the host machine. We thus need three new virtual interfaces. We will use ngctl to manage all netgraph nodes, hooks, and edges. This tool can be used interactively or you can supply the command on the command line. Below we will use it interactively. Note that this command must be run as the root user.
# ngctl
mkpeer eiface ether ether mkpeer eiface ether ether mkpeer eiface ether ether name ngeth0: ng_host name ngeth1: ng_test1 name ngeth2: ng_test2
We create three interfaces of type eiface as we need Ethernet interfaces (as opposed to L3 interfaces when using type iface). The man page state they are named ngeth0, etc. We rename them to indicate their purpose.
Note that when referencing a node name, we must append it with a colon. That is because we're specifying a path, not just a node.
Next we need to bridge these interfaces together. We can either create the bridge first and then connect it to the three interfaces, or while creating the bridge, we can already connect it to an interface.
/* * OPTION A: * To only create the bridge but not connect it to any existing nodes */ mkpeer bridge link0 link0 /* * OPTION B: * Create a new node of type bridge and connect it to the * path 'ng_host:'. Use its hook called 'ether' to connect it to the * hook 'link0' of the new node (the bridge). */ mkpeer ng_host: bridge ether link0 /* * As opposed to the Ethernet interfaces, a bridge does not have a * name by default. So we name it here. */ name [1e]: br0 /* * Now connect the nodes together. The first connection is only needed * when option A was used. */ connect br0: ng_host: link0 ether connect br0: ng_test1: link1 ether connect br0: ng_test2: link2 ether
The above configuration is pretty straightforward, the exception being the name command. Since the bridge does not have a name, we can't use it to construct a path. Instead we must use its unique ID (between brackets). To determine this ID, we need the list command which I ommitted from the configuration above. Let's show it here.
+ list There are 6 total nodes: Name: ng_host Type: eiface ID: 00000007 Num hooks: 1 Name: ng_test1 Type: eiface ID: 00000009 Num hooks: 0 Name: ngctl85802 Type: socket ID: 0000002b Num hooks: 0 Name: <unnamed> Type: bridge ID: 0000001e Num hooks: 1 Name: ng_test2 Type: eiface ID: 0000000e Num hooks: 0 +
The above output shows the unique ID for the bridge (preceding zeros can be ommitted) as well as the number of hooks used, i.e. connections made. I thus created the bridge by directly connecting it to the ng_host interface.
Configuring the jails
I assume you know how to setup a jail, i.e. create a base jail, clone it, etc. If not, please read the Jails chapter of the FreeBSD Handbook.
The below configuration only includes the networking-specific configuration for the two jails. Although shown per jail, you can easily move these two lines to the default configuration using the variable $name to customize the interface name.
test1 { vnet; vnet.interface = ng_test1; } test2 { vnet; vnet.interface = ng_test2; }
Once the jails have been started, you can manually configure IP addresses on their interfaces, as well as on the host interface ng_host.
NAT to be covered in the future.
Known issues
impossible to kill a jail / zombie jail PR 89528 Fixing old TTY bugs
This will (hopefully) be fixed by new TTY layermultiple postgresql servers in multiple jails (SysV IPC problem) initial question continuation
In a short answer - use different UIDs for PostgreSQL in each jail
You can also use some loader.conf tunables for IPC kern.ipc.shmmni=2048 kern.ipc.shmseg=2048 kern.ipc.semmni=128 kern.ipc.semmns=512
And of course allow IPC in jails by jail_sysvipc_allow="YES" in hosts rc.conf (or security.jail.sysvipc_allowed=1 in hosts sysctl.conf)bug in cpuset(1) - At the moment it is possible for a superuser inside a jail to modify the root cpuset of that jail. PR 134050
Tips
- leak of system message buffer from host system to jails
By default jails can read the system message buffer (ie: the console buffer). In many cases this isn't desired. Add the following line to /etc/sysctl.conf to turn it off: security.bsd.unprivileged_read_msgbuf=0
- allow ping inside jail
If you need to use ping inside jail, you must set sysctl security.jail.allow_raw_sockets=1 (add security.jail.allow_raw_sockets=1 in hosts sysctl.conf).
- installing 32-bit jail in to 64-bit host
- Running CentOS 5.5 in a jail
FreeBSD Wiki Centos 5.5 jail running on a host running FreeBSD 8-STABLE
- Running Debian 6.0 (Squeeze) in a FreeBSD jail
Tutorial: Debian GNU/kFreeBSD in a FreeBSD jail
Can I run Debian GNU/kFreeBSD in a chroot under FreeBSD?
Jail management
Some of these utilities are usable only with ancient versions of FreeBSD (4.x / 5.x).
- bastille
- Pure shell system for automating deployment and management of containerized applications in jails
- BSDploy
- Container management and orchestration for FreeBSD jails using ansible
- cbsd
- FreeBSD jail and bhyve management utility
- crate
- An application containerizer utilizing jails
- ezjail
- A framework to easily create, manipulate and run FreeBSD jails
- focker
A jail-based image orchestration tool in the vein of Docker
- iocage
- Utility for managing jails
- iocell
iocage fork focused on supporting classic iocage experience and adding fixes and features requested by the iocage users
- jadm
- FreeBSD jail framework with zfs, vnet and jail.conf support
- jailadmin
- A system for managing a set of named jails
- jailaudit
- Script for generating portaudit reports for jails
- jailctl
- Jail management tool
- jailer
- Manage FreeBSD jail startup, shutdown and console
sysutils/jailer (This port expired on: 2014-03-10)
- jailme
- A setuid version of jexec to allow normal users access to jails
- jailrc
- Improved jail startup/shutdown script
- jailutils
- Several utilities for managing jails
- jest
- REST API for creating and managing jails
- jkill
- Shutdowns a running jail and all its processes
- jps
- Wrapper to ps(1) that maps pids to jails
- jruls
- A top-alike utility to display and update FreeBSD jails resource usage
- jtop
- Wrapper to top(1) that maps pids to jails
- mkjail
- Easy jail creation, updates and upgrades
- mod_jail
- Apache 1.3.x/2.0.xx module to enable an easy alternative to mod_chroot
www/mod_jail (This port expired on: 2017-04-30)
- p5-BSD-Jail-Object
- An object oriented perl interface to jail(2)
- ploy
- Command-line tool to provision, manage, and control servers, including jails
- pot
- Container framework for FreeBSD, based on jails, ZFS and pf
- qjail
- Utility to deploy large number of jails quickly
- quickjail
- utility to create transient single-command jails
- runj
- Proof-of-concept OCI-compatible runtime for FreeBSD jails
Other resources
Eclipse/BSD - based on FreeBSD 3.4 (QoS - hierarchical proportional-share cpu, disk and link schedulers) (Wayback Machine, 2013-10-23)
freebsd-virtualization mailing list (jail, VIMAGE, Xen Project …)
SysJail for NetBSD / OpenBSD (Wayback Machine, 2014-04-08)
An Introduction to Jails and Jail Networking (SKYFORGE, 2018-03-14)
- FreeVPS - jail2 implementation for FreeBSD kernels
CategorySecurity CategoryHowTo CategoryNeedsContent CategoryStale