Setting up VNET jails with dual stack networking on Hetzner

Contents

Setting up VNET jails with dual stack networking on Hetzner

The Problem

Colo servers at Hetzner come with one fixed public IPv4 IP (which, by now, they also seem to charge for) and a dedicated IPv6 /64 prefix.

I wanted to run jails on FreeBSD on my colo server. For "untrusted" services, I wanted to do this with VNETs with an additional (paid) IPv4 address for each, and dual stack all the way so IPv6 connectivity as well.

This turned out to be less than simple, especially since I haven't ran FreeBSD since the early 2000's and never did anything productive with it before, and I wasted a lot of time being a wiseass and chasing ghosts before making it work so hopefully this blurb will save someone else that time.

My setup specifics:

Single-interface server (igb0)
FreeBSD 14.1-RELEASE
using Bastille (version 0.10.20231125) to create/manage the jails

Host: Getting IPv4 up and running

The first step was getting the host prepared. VNET jails work by plugging epair devices into a bridge; Bastille can do that for you (using bastille create -V), creating an igb0bridge device initially and plugging igb0 in it. While undoubtedly handy, this did have as side effect that you can't just go around and configure the bridge as required. Manual creation of the bridging setup is preferred instead.

Because Hetzner does care about the MAC address of the interface, the MAC should be cloned to the bridge. To do this, load bridging already at boot:

$ grep bridge /boot/loader.conf
if_bridge_load="YES"

Then tell sysctl to let bridges inherit the MAC of the first interface, and set some extra parameters for doing filtering - which is not what is recommended by Bastille, but is necessary regardless:

$ grep bridge /etc/sysctl.conf 
net.link.bridge.inherit_mac=1
net.link.bridge.pfil_bridge=1
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_member=1

Add the first part of the networking configuration of the bridge to /etc/rc.conf. The right details can be gotten from ifconfig/netstat after installing the machine with DHCP enabled on igb0 (Hetzner will hand out your dedicated IP over DHCP).

# Create bridge0 device
cloned_interfaces="bridge0"

# Force the primary interface up. Disable TSO to not mess with the bridge's head
ifconfig_igb0="up -tso -vlanhwtso"

# Configure the bridge with igb0 as first member, and already give it inet6 link-local address
create_args_bridge0="inet6 auto_linklocal -ifdisabled addm igb0"

# Set the primary IP details
ifconfig_bridge0="inet a.b.c.126 netmask 0xffffffc0 broadcast a.b.c.127"
defaultrouter="a.b.c.65"

To stress again: the options to disable TSO on the physical bridge member are important.

If you reboot now, you should end up with a working bridge0 with IPv4 connectivity.

Guest: Getting IPv4 up and running

I create the jails with Bastille, as (thick) VNET jails but attached to the existing bridge:

$ sudo bastille create -T -B boink 14.1-RELEASE a.b.c.120/26 bridge0

Bastille will create the jail, create and plug e0a_boink in bridge0, and e0b_boink in the jail. The a.b.c.120 is the extra IP I paid Hetzner for; this will be assigned to e0b_boink which Bastille will rename to vnet0 inside the jail. Routing information will be copied from the host.

Once the jail is up, IPv4 will not function. This is normal. By default, the IP will be linked to the MAC address of the main interface - but that's not what we want. In the Hetzner Robot site, you can request a separate MAC address for the extra IPv4 addresses. Do this, then create a start_if file inside the jail:

$ cat /etc/start_if.vnet0 
ifconfig vnet0 ether 00:50:x:y:z:DC

This will force the jail to use the requested MAC address for the internal vnet0 device which holds the IP. Restart the jail, and pronto, IPv4 working inside the jail. I suppose it could also be done by setting this MAC address in the jail config which would remove the need for this start_if.vnet0 inside the jail but that's cosmetics and I prefer not to customize things via Bastille any more than required.

IPv6: things that don't work

Getting IPv6 to work turned out to be a much more confusing endeavour. The /64 prefix is, as mentioned in Hetzner docs (but in a non-threatening, somewhat vague way) routed to the hardware-linked address of the primary network interface, aka our igb0. The gateway to use is fe80::1%interface.

Some specific things to note:

Adding IPv6 addresses straight to the jails, with the same gateway as the host, works for a little while only. You can ping6 the gateway, and Google. You are reachable from the internet. This does not last. After a while, connectivity just... stops. You can still ping fe80::1%vnet0 but nothing beyond. You are not reachable from the internet, even though you were for a good number of minutes before. I lost so much time chasing this.
A real reason for this is still not 100% clear to me since I have no insight in the setup of the Hetzner infra, but they did mention the link to the hardware address of the primary network interface. So I could have saved myself the time of thinking I could be smarter and get it to do what I want regardless.
My guess? At startup time the jail announces itself upstream as neighbour via ICMP6. Somehow, this gets accepted. Even though it really shouldn't. So packets flow. Once the initial announcement expires (but why would it expire if the initial packets are accepted?), the router reverts to what it knows, aka the entire prefix should go to... igb0's hardware address. Not your vnet0.
Some reports/writeups/blurbs/... refer to setting net.inet6.icmp6.nd6_onlink_ns_rfc4861=1 (see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263288). This does not work. Don't set it.
Some reports/writeups/blurbs/... refer to setting static ndp entries. This does not work. Don't bother.
Some reports/writeups/blurbs/... refer to running the rtsold service. This is not required and does not work. Don't bother.

Host: Getting IPv6 up and running

Configuring IPv6 on the host is now trivial: set the address on bridge0, and the gateway via the bridge0 device, in /etc/rc.conf.

ifconfig_bridge0_ipv6="inet6 2a01:p:q:r::1 prefixlen 64"
ipv6_defaultrouter="fe80::1%bridge0"

Keep the full /64 prefix. That's all.

Guest: Getting IPv6 up and running

Configuring guests for IPv6 is now as easy as setting the chosen address on vnet0, with /64 prefix, and using the host as a gateway. /etc/rc.conf:

ifconfig_vnet0_ipv6="inet6 2a01:p:q:r::120 prefixlen 64"
ipv6_defaultrouter="2a01:p:q:r::1"

In order to let the host route, don't forget to enable the sysctl on the host:

$ sudo sysctl net.inet6.ip6.forwarding=1
$ grep forward /etc/sysctl.conf 
net.inet6.ip6.forwarding=1

And that is it. Hetzner is happy because the IPv4 addresses arrive at network interfaces with the MAC address they like. The whole IPv6 prefix arrives at the hardware address of the physical igb0, which lives now on bridge0. The jail host will forward v6 IPs in the range to the respective jail epair devices. Jails can talk to other jails, and to the jail host.

Finishing touches

Don't forget to enable pf on both the host and the jails, and to give icmp6 a pass in /etc/pf.conf.

References

I got a lot of useful information, both on what I should and should not do, from a lot of different links. Some of them are these, in no particular order of importance:

Remarks/corrections

... are of course always welcome. I'm new to FreeBSD so it is definitely likely that there are a number of cockups higher up.

CategoryIpv6 CategoryVirtualization

LennertVanAlboom/DualStackHetznerVnetJails (last edited 2024-08-19T22:34:29+0000 by LennertVanAlboom)