What Is Infiniband?
InfiniBand (IB) is a high-speed (2–96 Gbit/s) low-latency (140–2600 ns) switched-fabric interconnect, developed primarily for HPC, but as of now widely adopted where its properties are in demand.
InfiniBand is supported on most major operating systems: Windows, Linux, (Open)Solaris, AIX, HP-UX, z/OS, among others. OpenFabrics Alliance (founded in 2004 as OpenIB Alliance) is developing a cross-platform IB/RDMA stack for Linux and Windows. The stack is released as OFED (OpenFabrics Enterprise Distribution) under both GPL and BSD licenses.
Infiniband in FreeBSD
The OFED stack was merged in March 2011 (http://jeffr-tech.livejournal.com/23467.html) and is available in FreeBSD 9.0 and later.
Proprietary IB stacks are known to exist outside of FreeBSD proper. In particular, three major storage appliance vendors -- EMC/Isilon, NetApp, and Panasas -- ship products based on FreeBSD and use InfiniBand for high-speed interconnects.
Building OFED in FreeBSD
- Add the following options to your KERNCONF to compile the necessary protocols/protocol options into your kernel:
options OFED # Infiniband protocol stack and support (needed for FreeBSD 9.X only) options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib
- Add the OFED drivers to your KERNCONF (alternately you could use the MODULES_OVERRIDE variable in /etc/src.conf):
device ipoib # IP over IB devices device mlx4ib # ConnectX Infiniband support device mlxen # ConnectX Ethernet support device mthca # Infinihost cards
- Run buildkernel/installkernel per usual process.
Add WITH_OFED='yes' to /etc/src.conf.
- Run buildworld/installworld per usual process.
- Set the "sys.device.mlx4_coreX.mlx4_portY" sysctl to either "eth" or "ib" depending on how you want the device ports to be configured.
There are some outstanding work items for Infiniband/OFED on FreeBSD.
Short term items
- Upgrade to 1.5.x (Isilon). Delivery date: TBD.
Long term items
- Upgrade to 3.x (Isilon). Can't be done immediately because of refactoring done in the kernel.org branch and with the userland utilities, and the fact that things are still in a large degree of flux.
- WITH_OFED doesn't cover all OFED stack utilities; a number of them need to be ported from Linux and contributed back to OFED (preferred) or rewritten to work with FreeBSD.
There's an infiniband mailing list. If you're having issues or would like to provide feedback about the Infiniband stack on FreeBSD, you should subscribe and poke the mailing list.