What Is Infiniband?

InfiniBand (IB) is a high-speed (10-300Gb/s) low-latency (140–2600 ns) switched-fabric interconnect, developed primarily for HPC, but as of now widely adopted where its properties are in demand.

InfiniBand is supported on most major operating systems: Windows, Linux, (Open)Solaris, AIX, HP-UX, z/OS, among others. OpenFabrics Alliance (founded in 2004 as OpenIB Alliance) is developing a cross-platform IB/RDMA stack for Linux and Windows. The stack is released as OFED (OpenFabrics Enterprise Distribution) under both GPL and BSD licenses.

Infiniband in FreeBSD

The OFED stack was merged in March 2011 (http://jeffr-tech.livejournal.com/23467.html) and is available in FreeBSD 9.0 and later.

Proprietary IB stacks are known to exist outside of FreeBSD proper. In particular, three major storage appliance vendors -- EMC/Isilon, NetApp, and Panasas -- ship products based on FreeBSD and use InfiniBand for high-speed interconnects.

Infiniband hardware support by release

Card series

Driver

FreeBSD 9.x

FreeBSD 10.x

FreeBSD 11.x

Comments

InfiniHost

mthca

Yes

Yes

Yes

Not maintained

ConnectX

mlx4

Yes

Yes

Yes

ConnectX-2

mlx4

Yes

Yes

Yes

ConnectX-3/Pro

mlx4

Yes

Yes

Yes

ConnectX-4 (Eth)

mlx5

No

Yes

Yes

ConnectX-4 (IB)

mlx5

No

Not Yet

Not Yet

In development

Connect-IB

mlx5

No

Not Yet

Not Yet

In development

Building OFED in FreeBSD

Kernel

options        OFED            # Infiniband protocol stack and support
options        SDP             # Sockets Direct Protocol for infiniband
options        IPOIB_CM        # Use connect mode ipoib

device         ipoib           # IP over IB devices
device         mlx4ib          # ConnectX Infiniband support
device         mlxen           # ConnectX Ethernet support
device         mthca           # Infinihost cards

Kernel Modules

As an alternative to building a new kernel, recent releases provide IB drivers as kernel modules.

Note: If your /usr/src tree is newer than your kernel, you must do a buildkernel/installkernel along with buildworld/installworld. See the FreeBSD handbook for full details.

The script below will build, install, and load the appropriate drivers:

 #!/bin/sh -e

cd /usr/src/sys/modules
for module in mlx4 ibcore mlx4ib ipoib; do
    cd $module
    make
    make install
    sync
    kldload $module || true
    printf "${module}_load=\"YES\"\n" >> /boot/loader.conf
    cd ..
done
kldstat

Userland

  1. Add WITH_OFED='yes' to /etc/src.conf.

  2. Run buildworld/installworld per usual process.
  3. Set the "sys.device.mlx4_coreX.mlx4_portY" sysctl to either "eth" or "ib" depending on how you want the device ports to be configured.

Status

Mellanox engineers, et. al., have made substantial improvements to the IB stack in 2015/2016 to support the latest ConnectX HCAs. The freebsd-infiniband email archives contain discussions of progress and issues.

There are some outstanding work items for Infiniband/OFED on FreeBSD.

Short term items

Long term items

Wishlist

Feedback

There's an infiniband mailing list. If you're having issues or would like to provide feedback about the Infiniband stack on FreeBSD, you should subscribe and poke the mailing list.

References


Everything below is "work in progress". It will be moved into the correct place on this page when it's ready B)

Where are the sources?

We should point out what the sys/ofed directory is for, the related IB driver directory, and any other related bits

Compile time settings

We should explain all relevant compile time settings

Configuration / runtime settings

We should explain the relevant sysctl.conf, loader.conf, and rc.conf settings.

First thoughts on ones to include:

/etc/sysctl.conf

/etc/loader.conf

/etc/rc.conf

Operating modes

We should add information here about the 2 main operating modes for the Mellanox Infiniband adapters:

  1. Native infiniband mode
  2. Native ethernet mode

It would be useful to clarify that in Infiniband mode, IP can be encapsulated and sent using IPoIB.

Cables & connectors

We should add info and photo's here for the different cables & connector types

Maximum cable length

It would also be useful to explain the maximum length of cables, according to type, connector, data rate (and whatever else).

InfiniBand (last edited 2016-04-24 13:06:09 by JasonBacon)