What Is Infiniband?

InfiniBand (IB) is a high-speed (2–96 Gbit/s) low-latency (140–2600 ns) switched-fabric interconnect, developed primarily for HPC, but as of now widely adopted where its properties are in demand.

InfiniBand is supported on most major operating systems: Windows, Linux, (Open)Solaris, AIX, HP-UX, z/OS, among others. OpenFabrics Alliance (founded in 2004 as OpenIB Alliance) is developing a cross-platform IB/RDMA stack for Linux and Windows. The stack is released as OFED (OpenFabrics Enterprise Distribution) under both GPL and BSD licenses.

Infiniband in FreeBSD

The OFED stack was merged in March 2011 (http://jeffr-tech.livejournal.com/23467.html) and is available in FreeBSD 9.0 and later.

Proprietary IB stacks are known to exist outside of FreeBSD proper. In particular, three major storage appliance vendors -- EMC/Isilon, NetApp, and Panasas -- ship products based on FreeBSD and use InfiniBand for high-speed interconnects.

Building OFED in FreeBSD

Kernel

options        OFED            # Infiniband protocol stack and support
options        SDP             # Sockets Direct Protocol for infiniband
options        IPOIB_CM        # Use connect mode ipoib

device         ipoib           # IP over IB devices
device         mlx4ib          # ConnectX Infiniband support
device         mlxen           # ConnectX Ethernet support
device         mthca           # Infinihost cards

Userland

  1. Add WITH_OFED='yes' to /etc/src.conf.

  2. Run buildworld/installworld per usual process.

Status

There are some outstanding work items for Infiniband/OFED on FreeBSD.

Short term items

Long term items

Wishlist

Feedback

There's an infiniband mailing list. If you're having issues or would like to provide feedback about the Infiniband stack on FreeBSD, you should subscribe and poke the mailing list.

References

InfiniBand (last edited 2013-05-26 22:13:18 by GarrettCooper)