Native MPLS Support for FreeBSD
Project Information
Student |
Subhash Murmu |
Mentor |
Bruce Simpson, Alexander Chernikov |
Organization |
FreeBSD Foundation |
Duration |
350 hours (Large Project) |
Period |
1 May 2026 – 24 August 2026 |
Repository |
Summary
This project implements native Multi-Protocol Label Switching (MPLS) support in the FreeBSD kernel. MPLS is a high-performance packet-forwarding technology that prepends a 32-bit shim header between the Layer 2 and Layer 3 headers, establishing Label Switched Paths (LSPs) where core routers perform simple fixed-length label swaps instead of expensive IP longest-prefix lookups. While OpenBSD, NetBSD, and Linux have shipped MPLS for years, FreeBSD lacks a native implementation. This project closes that gap using a bottom-up approach — starting at the wire and building upward through the control plane — and exceeds existing BSD implementations in two key areas: ECMP multipath via FreeBSD's modern nhgrp nexthop-group subsystem, and a Netlink-based control plane that replaces the rigid, extension-hostile rtsock interface.
Approach
Philosophy: Bottom-Up Implementation
Rather than designing a top-level architecture first and filling it in later, every layer is built on a fully verified layer below it — nothing above compiles or is tested until everything beneath it works. This is the opposite of how most kernel features are typically prototyped and is the primary ideological departure from prior BSD MPLS implementations (OpenBSD, NetBSD), which were designed top-down.
The layer stack, from bottom to top: [ Phase 5 ] Top-Level Routing & Multipath ← nhop/nhgrp, ECMP, userland tooling [ Phase 4 ] Netlink Integration ← AF_MPLS route messages, daemon hookup [ Phase 3 ] Forwarding Engine & LIB ← mpls_input / forward / output, PHP [ Phase 2 ] Minimalist LDP Daemon ← label negotiation, control-plane primer [ Phase 1 ] Link-Layer Encapsulation ← EtherType, shim header, reserved labels ───────────────────────────────────────────── THE WIRE
No phase begins until the exit criterion of the phase below it is verified via tcpdump, functional tests, or integration tests. This ensures that integration failures are always traceable to the current phase rather than buried in an untested layer beneath it.
Phase-by-Phase Implementation
Phase 1 lays the wire-level foundation: the 32-bit MPLS shim header is defined, EtherType 0x8847/0x8848 is hooked into ether_demux(), and a NETISR_MPLS stub handler is registered so the kernel can recognize and receive MPLS frames. Nothing above this makes sense until a frame can reach the stack.
Phase 2 builds the control-plane primer by porting OpenBSD's ldpd to FreeBSD. The wire-protocol logic (hello, init, keepalive, label mapping) is reused as-is; the key ideological change is a full rewrite of kroute.c, replacing OpenBSD's proprietary SIOCADDMPLSLABEL ioctl with FreeBSD Netlink. The daemon is built at this stage — before the kernel forwarding engine exists — so that the full label lifecycle (allocate → advertise → install → withdraw) is reasoned through before the LIB API is designed, resulting in a cleaner boundary between the control plane and the data plane.
Phase 3 implements the data plane inside the kernel: the per-VNET Label Information Base (LIB), mpls_input() for ingress label lookup and pop, mpls_forward() for transit swap with RFC 3443 TTL handling, and Penultimate Hop Popping (PHP) with mandatory S-bit validation. At this point a full LSP can be manually programmed and verified in a VNET-jail lab without any Netlink dependency.
Phase 4 wires everything together via Netlink: AF_MPLS RTM_NEWROUTE / RTM_DELROUTE / RTM_GETROUTE handlers are added to the kernel so the Phase 2 LDP daemon can install and withdraw label routes through a standard socket interface, replacing the manual test harness used in Phase 3. All Netlink interactions — including atomic LIB state dumps — are built on top of the snl(3) library, which handles the construction and parsing of Tag-Length-Value (TLV) structures internally; raw rtnetlink(4) TLVs are never assembled by hand.
Phase 5 delivers production quality: MPLS metadata is embedded directly into FreeBSD's nhop_object to eliminate double lookups in the forwarding path, ECMP is enabled via nhgrp nexthop groups with per-flow hashing, and userland tooling (route(8), netstat(8), mpe(4)) is added to make the feature fully accessible to operators.
Deliverables
Pre-Midterm (Phases 1–3)
Phase 1 — Link-Layer Encapsulation (done)
Phase 1 established the MPLS link-layer foundation in the FreeBSD kernel. A new sys/netmpls/ subsystem was created with mpls.h (RFC 3032 shim header, bitmasking macros, reserved labels, and operation constants), mpls_input.c (NETISR_MPLS stub handler with SYSINIT/VNET_SYSINIT/VNET_SYSUNINIT registration), and mpls_output.c (mpls_encap() egress helper with software checksum finalization and M_PREPEND shim insertion). Existing files were modified to wire everything together: ether_demux() in if_ethersubr.c gained EtherType dispatch for 0x8847/0x8848, netisr.h gained NETISR_MPLS = 11, and the build system was updated via conf/options and conf/files. A custom MPLS kernel config activates the entire subsystem.
Exit Criterion: A raw MPLS frame injected via BPF across an epair(4) pair reached mpls_input() without a kernel panic (confirming NETISR_MPLS registration), and a tcpdump hex dump of a frame produced by mpls_encap() decoded correctly to label=100, tc=0, BoS=1 ([S]), ttl=64 — with zero errors or warnings in the final build log.
Phase 2 — Minimalist LDP Daemon (done)
Phase 2 ports OpenBSD's ldpd to FreeBSD as a minimalist LDP control-plane daemon, preserving the three-process architecture (ldpd, ldpe, lde). A force-included compat.h translates the OpenBSD API surface without touching upstream sources. Key OS-level fixes: fetchifs() rewritten with getifaddrs() to replace an OpenBSD-specific NET_RT_IFLIST sysctl; multicast detection replaced with IP_RECVDSTADDR ancillary data since FreeBSD does not set MSG_MCAST; and multicast joins switched to struct ip_mreqn for ifindex-based binding. A startup sequencing bug where addr_ok was permanently 0 was resolved by introducing kr_ifstatus_rebroadcast() to ensure IMSG_IFSTATUS precedes IMSG_NEWADDR in the ldpe startup dispatch.
Exit Criterion: A complete RFC 5036 LDP session reaching OPERATIONAL state against an independent peer — verified against FRRouting 8.4.4. The FSM traversed PRESENT → INITIALIZED → OPENSENT → OPENREC → OPERATIONAL, bidirectional Label Mapping PDUs were exchanged, and FRR's show mpls ldp neighbor independently confirmed State: OPERATIONAL with sustained uptime, validating that the Keepalive mechanism was functioning correctly on both sides.
OpenBSD ldpd vendor import: Commit diff, Browse files
Phase 3 — Forwarding Engine & LIB (done)
Phase 3 implements the in-kernel MPLS forwarding engine and per-VNET Label Information Base, replacing Phase 1's stub with the real mpls_input()/mpls_forward() pair. mpls_lib_lookup() copies matching entries out under the table mutex instead of returning a pointer, avoiding use-after-free in the fast path. Key implementation details: PHP is encoded as MPLS_OP_SWAP to the implicit-null label rather than a fourth opcode; reserved labels 0–15 are classified in mpls_shim.c ahead of the LIB lookup; and SWAP/PHP egress is built by hand and sent via ether_output_frame(), since ether_output() would stamp the wrong EtherType. An ordering bug initially let reserved labels skip TTL expiry; moving the ttl <= 1 guard ahead of the reserved-label branch in mpls_input() fixed it.
Exit Criterion: A manually-programmed LSP transiting a three-VNET-jail lab end-to-end, swapped at transit node P and popped at egress node PE2 — confirmed via tcpdump at each hop. All thirteen functional and integration test cases (TC-POP-01 through TC-MULTIHOP-01) passed, including sustained 1,000-frame load with zero faults and clean per-VNET teardown via jail -r.
Post-Midterm (Phases 4–5)
Phase 4 — Netlink Integration
not started
Phase 5 — Top-Level Routing & Multipath
not started