• TomMarcoen
• JailNetworking

Title: Networking with Jails

Introduction

To start off, first a few links to other resources:

• Using jails with if_bridge(4) and epair(4) (Mason Loring Bliss)

• Jails (generic information on jail development)

• Sample (and simple) VNET configurations in shell (Bjoern Zeeb)

There are three ways to do networking with jails: (i) using IP addresses (aliases) from the host, (ii) using if_bridge(4) and epair(4) interfaces, and (iii) using netgraph. I will start this how-to with the third method as the second method has already been explained by Mason Loring Bliss. Later I will add an example of the first method.

Netgraph

I am migrating this section to TomMarcoen/Netgraph.

Before we kick off, we need to understand some new terms that netgraph introduces.

Node

Any object that can connect to another object. This can be a bridge, a virtual interface, but also a VLAN interface or a BPF object.

Edge

The connection between two nodes. You could consider it to be a virtual network cable.

Hook

In netgraph, edges do not really exist per se. Instead, an edge is simply an association of two hooks, one from each node. A node's hooks define how that node can be connected. For a bridge node, the hooks are the virtual interfaces, though for a virtual interface node, hooks can be each protocol supported by that interface.

Path

Every netgraph node is addressable via an ASCII string called a node address or path. Many nodes have names. When a node has a name, it can always be addressed usingthe absolute address consisting of the name followed by a colon. If a node does not have a name, you can construct one from the node's unique ID number by enclosing the number in square brackets. Relative addressing is also possible when two nodes are indirectly connected. A relative address uses the names of consecutive hooks on the path from the source node to the target node. Relative and absolute addressing can be combined.

This is quite confusing. See the example at https://people.freebsd.org/~julian/netgraph.html.

Let's start off with a small network consisting of two jails, called "test1" and "test2", which are bridged to an new, virtual interface on the host machine. We thus need three new virtual interfaces. We will use ngctl to manage all netgraph nodes, hooks, and edges. This tool can be used interactively or you can supply the command on the command line. Below we will use it interactively. Note that this command must be run as the root user.

# ngctl

mkpeer eiface ether ether
mkpeer eiface ether ether
mkpeer eiface ether ether
name ngeth0: ng_host
name ngeth1: ng_test1
name ngeth2: ng_test2

We create three interfaces of type eiface as we need Ethernet interfaces (as opposed to L3 interfaces when using type iface). The man page state they are named ngeth0, etc. We rename them to indicate their purpose.

Next we need to bridge these interfaces together. We can either create the bridge first and then connect it to the three interfaces, or while creating the bridge, we can already connect it to an interface.

/*
 * OPTION A:
 * To only create the bridge but not connect it to any existing nodes
 */
mkpeer bridge link0 link0

/*
 * OPTION B:
 * Create a new node of type bridge and connect it to the
 * path 'ng_host:'.  Use its hook called 'ether' to connect it to the
 * hook 'link0' of the new node (the bridge).
 */
mkpeer ng_host: bridge ether link0

/*
 * As opposed to the Ethernet interfaces, a bridge does not have a
 * name by default. So we name it here.
 */
name [1e]: br0

/*
 * Now connect the nodes together.  The first connection is only needed
 * when option A was used.
 */
connect br0: ng_host: link0 ether
connect br0: ng_test1: link1 ether
connect br0: ng_test2: link2 ether

The above configuration is pretty straightforward, the exception being the name command. Since the bridge does not have a name, we can't use it to construct a path. Instead we must use its unique ID (between brackets). To determine this ID, we need the list command which I ommitted from the configuration above. Let's show it here.

+ list
There are 6 total nodes:
  Name: ng_host         Type: eiface          ID: 00000007   Num hooks: 1
  Name: ng_test1        Type: eiface          ID: 00000009   Num hooks: 0
  Name: ngctl85802      Type: socket          ID: 0000002b   Num hooks: 0
  Name: <unnamed>       Type: bridge          ID: 0000001e   Num hooks: 1
  Name: ng_test2        Type: eiface          ID: 0000000e   Num hooks: 0
+

The above output shows the unique ID for the bridge (preceding zeros can be ommitted) as well as the number of hooks used, i.e. connections made. I thus created the bridge by directly connecting it to the ng_host interface.

Configuring the jails

I assume you know how to setup a jail, i.e. create a base jail, clone it, etc. If not, please read the chapter on jails in the FreeBSD handbook.

The below configuration only includes the networking-specific configuration for the two jails. Although shown per jail, you can easily move these two lines to the default configuration using the variable $name to customize the interface name.

test1 {
   vnet;
   vnet.interface = ng_test1;
}
test2 {
   vnet;
   vnet.interface = ng_test2;
}

Once the jails have been started, you can manually configure IP addresses on their interfaces, as well as on the host interface ng_host.

Network address translation

Again we're presented with a bunch of options. When using if_bridge you connect the bridge to the host machine using an epair and you configure NAT on the host's instance of PF. When using netgraph, a similar option can be achieved by creating a virtual ng_eiface, connecting this to the ng_bridge and then configuring NAT on the host's firewall. Netgraph provides a different option though: ng_nat.

Aliases

You can easily assign an IP address to your jail from the same range as your host. The host then creates an alias on its interface and directs all traffic to that IP address to your jail.

test1 {
   ip4.addr = "em0|203.0.113.80";
}

This method might not possible when you have a cloud-hosted VPS where it is either not possible to get additional public IP addresses for your VPS or where it is simply too expensive to get more public IP addresses. In this case it is possible however to combine this method with network address translation.

1. Assign an IP address range to be used locally on your server for your jails.
2. Create a new loopback interface to which all the jails will connect.
3. Create the necessary firewall rules for outbound network address translation and inbound port forwarding.