Adopt -fbounds-safety in the FreeBSD Kernel TCP/IP Stack
Abhijeet Sharma, funded by the FreeBSD Foundation. Technical monitor and reviewer: Rui Paulo.
Project page at the FreeBSD Foundation
Contents
Overview
The networking stack is the kernel's largest remotely reachable attack surface, and most of its exploitable bugs are out of bounds accesses. Clang's -fbounds-safety extension turns such accesses into clean runtime traps, and Apple has already deployed it across the xnu TCP/IP stack, which shares 4.4BSD ancestry with FreeBSD's.
This project introduces the annotation macros and build support for -fbounds-safety in FreeBSD and adopts it across the TCP/IP stack, file by file, with each change remaining fully compatible with the stock toolchain.
How it works
In C a pointer and its length are usually passed together, but the compiler cannot connect them. __counted_by(n) and its siblings tie the two, so the compiler can bounds-check accesses and elide the check where it can prove the access is safe.
The bound travels as type information in the shared header, not as data in the pointer. __single and __counted_by keep the ordinary one-word pointer layout, so struct layouts and calling conventions are unchanged and instrumented files link freely against uninstrumented ones. The wider forms (__indexable, __bidi_indexable) do change layout, so this project keeps them confined to function locals and never lets them cross a boundary between instrumented and uninstrumented code.
The ABI-visible annotation macros expand to nothing under a compiler without the extension, so the annotated tree still builds unchanged with the stock toolchain. The wide forms are left undefined rather than stubbed, so a misuse is a compile error rather than a silent loss of bounds.
Scope
Annotated in priority order, most exposed to attacker-controlled input first:
tcp_ecn.c -- smallest file, validates the toolchain and build glue
tcp_sack.c -- attacker supplied SACK blocks
tcp_reass.c -- reassembly of untrusted segments
tcp_fastopen.c -- TFO cookie validation
tcp_syncache.c -- SYNs and syncookies from unauthenticated peers
tcp_input.c -- tcp_dooptions(), the most exposed routine in the stack
tcp_output.c -- tcp_addoptions() writes into a TCP_MAXOLEN buffer
tcp_subr.c -- tcp_respond() and the pcblist sysctl
The IP layer parsers ip_options.c and ip_reass.c follow as the schedule permits. The alternate RACK and BBR stacks and tcp_lro.c are out of scope.
Getting started
The code generation for -fbounds-safety is not yet in the Clang shipped in the FreeBSD base system, so building annotated files with checks enabled needs an external toolchain. These steps reproduce the M1 toolchain and kernel build.
1. Prerequisites
FreeBSD:
1 pkg install cmake ninja git python3
Debian or Ubuntu:
macOS:
1 brew install cmake ninja
2. Build the toolchain
1 git clone --depth 1 -b stable/21.x \
2 https://github.com/swiftlang/llvm-project.git
3 cd llvm-project
4 cmake -S llvm -B build -G Ninja \
5 -DCMAKE_BUILD_TYPE=Release \
6 -DLLVM_ENABLE_PROJECTS="clang;lld" \
7 -DLLVM_TARGETS_TO_BUILD="X86;AArch64"
8 ninja -C build clang lld clang-cpp llvm-objdump llvm-nm llvm-objcopy
stable/21.x is a moving branch and the recipe clones its tip, so record the commit you built and quote it with any results. sysctl -n kern.compiler_version reports it from a running kernel. The builds below used clang 21.1.6, commits 098fd15f41be and 7227bcf78b0a.
3. Verify the toolchain
Base Clang parses the annotations without acting on them, so a clean compile proves nothing. Confirm the check runs. The function below is the documentation's own __counted_by example; <ptrcheck.h> is the toolchain's header and expands the annotations to nothing when the flag is absent.
1 /* bs-check.c */
2 #include <ptrcheck.h>
3
4 void fill_array_with_indices(int *__counted_by(count) p, unsigned count) {
5 // off-by-one error (i < count)
6 for (unsigned i = 0; i <= count; ++i) {
7 // bounds check inserted:
8 // if (i >= count) trap();
9 p[i] = i;
10 }
11 }
12
13 int main(void) {
14 int a[5]; /* one element of slack */
15 fill_array_with_indices(a, 4); /* count is 4, so p[4] is out of bounds */
16 return 0;
17 }
The array has a spare element, so the final write lands in real memory. Only the annotation makes it a violation, which keeps the result attributable to the bounds check rather than to a stack canary.
The first run must crash on the inserted trap. If it exits 0 the compiler is ignoring the flag and nothing built with it is checked.
4. Build a kernel
MAKEOBJDIRPREFIX must already exist. The examples build amd64; for arm64 use TARGET=arm64 TARGET_ARCH=aarch64.
On FreeBSD, describe the toolchain in a makefile and pass it as CROSS_TOOLCHAIN, which build(7) accepts as either a full path or the base name of a file in ${LOCALBASE}/share/toolchains:
On Linux or macOS, cross-build with tools/build/make.py, which sets XCC on its own command line and so overrides whatever CROSS_TOOLCHAIN sets. Use --cross-bindir there instead:
1 T=/path/to/llvm-project/build/bin
2 cd src
3 env MAKEOBJDIRPREFIX="$OBJPREFIX" python3 tools/build/make.py \
4 --cross-bindir=$T \
5 TARGET=amd64 TARGET_ARCH=amd64 -j8 \
6 -DWITHOUT_CLANG_BOOTSTRAP -DWITHOUT_LLD_BOOTSTRAP \
7 -DWITHOUT_CLANG -DWITHOUT_LLD -DWITHOUT_LLDB -DNO_MODULES \
8 kernel-toolchain buildkernel
-DNO_MODULES builds the GENERIC kernel image without loadable modules.
The resulting kernel is at:
Verified on
Host |
Toolchain |
Kernel build |
macOS 26 arm64 |
builds |
cross, amd64 and arm64 |
Debian 12 x86_64 |
builds |
cross, amd64 |
FreeBSD 14.3 amd64 |
builds |
native via CROSS_TOOLCHAIN, amd64 |
Milestones and status
✅ marks an approved or committed item.
M1: toolchain and recipe
- ✅ External toolchain builds an unmodified GENERIC kernel on amd64
✅ Recipe documented: see Getting started
✅ D58943 tools/build: stage stdckdint.h's dependencies for non-FreeBSD hosts (committed, rG805c5004fa86)
M2: annotation macros and build knob
Reviews, in the order they should be read. Each builds on the one before.
D58983 cdefs: add bounds-safety annotation macros
D58984 kern.mk: add per-file bounds-safety build knob
D58985 sys: build core headers under bounds-safety
D58986 contrib/ck: keep dcas parameters thin under bounds-safety
D58987 kern: add bounds-safety soft-trap runtime
Also outstanding:
tcp_ecn.c compiling with checks enabled
- Document the knob
M3: mbuf accessor, tcp_sack.c, tcp_reass.c
- mbuf accessor design agreed with the reviewer
tcp_sack.c annotated
tcp_reass.c annotated, booting and passing the netinet tests on amd64 and arm64
- kyua test that deliberately triggers a bounds violation and verifies the trap fires
M4: tcp_fastopen.c, tcp_syncache.c, CI
tcp_fastopen.c annotated
tcp_syncache.c annotated and passing on both architectures
- CI job definition for the instrumented build, submitted to the CI maintainers
- Documentation for reproducing a CI failure locally
M5: tcp_input.c, tcp_output.c
tcp_input.c annotated
tcp_output.c annotated and passing on both architectures
M6: tcp_subr.c, report, adoption guide
tcp_subr.c annotated
- Performance report, instrumented versus stock
- Adoption guide for other kernel subsystems
Contact
Abhijeet Sharma <abhijeetsharma2002@gmail.com>
Rui Paulo <rpaulo@FreeBSD.org>, technical monitor and reviewer