• InfiniBand

What Is Infiniband?

InfiniBand (IB) is a high-speed (10-300Gb/s) low-latency (140–2600 ns) switched-fabric interconnect, developed primarily for HPC, but as of now widely adopted where its properties are in demand.

InfiniBand is supported on most major operating systems: Windows, Linux, (Open)Solaris, AIX, HP-UX, z/OS, among others. OpenFabrics Alliance (founded in 2004 as OpenIB Alliance) is developing a cross-platform IB/RDMA stack for Linux and Windows. The stack is released as OFED (OpenFabrics Enterprise Distribution) under both GPL and BSD licenses.

Infiniband in FreeBSD

The OFED stack was merged in March 2011 (http://jeffr-tech.livejournal.com/23467.html) and is available in FreeBSD 9.0 and later.

Proprietary IB stacks are known to exist outside of FreeBSD proper. In particular, three major storage appliance vendors -- Dell EMC Isilon, NetApp, and Panasas -- ship products based on FreeBSD and use InfiniBand for high-speed interconnects.

Infiniband hardware support by release

Building OFED in FreeBSD

Kernel

1. Add WITH_OFED='yes' to /etc/src.conf. This will build all of the OFED kernel modules, e.g. ibcore, alongside the static kernel image.

2. Add the following options to your KERNCONF to compile the necessary protocols/protocol options into your kernel:

options        OFED            # Infiniband protocol stack and support
options        SDP             # Sockets Direct Protocol for infiniband
options        IPOIB_CM        # Use connected mode IPoIB

• Add the OFED drivers to your KERNCONF (alternately you could use the MODULES_OVERRIDE variable in /etc/src.conf):

device         ipoib           # IP over IB devices
device         mlx4ib          # ConnectX Infiniband support
device         mlxen           # ConnectX Ethernet support
device         mthca           # Infinihost cards

• Run buildkernel/installkernel as per the usual process.

Note: If your /usr/src tree is newer than your kernel, you must do a buildkernel/installkernel along with buildworld/installworld. See the FreeBSD handbook for full details.

Userland

* This is no longer necessary as OFED support is on by default now, but is still documented here for those who need to do a buildworld anyway.

1. Add WITH_OFED='yes' to /etc/src.conf.

2. Run buildworld/installworld per usual process.
3. Set the "sys.device.mlx4_coreX.mlx4_portY" sysctl to either "eth" or "ib" depending on how you want the device ports to be configured.

Kernel Modules

It is now possible to enable and upgrade Infiniband support on FreeBSD in a few minutes.

As an alternative to building a new kernel, recent releases provide IB drivers as kernel modules, and userland now supports Infiniband by default, so there is no need to perform a buildworld.

The script below will build, install, and load the appropriate drivers:

 #!/bin/sh -e

usage()
{
    printf "Usage: $0 4|5 datagram|connected\n"
    printf "* Use 4 for ConnectX 3 and earlier, 5 for later HCAs.\n"
    printf "* Specify connected if you want to use connected mode ipoib.\n"
    exit 1
}

if [ $# != 2 ]; then
    usage
fi

driver_version=$1
ipoib_mode=$2

case $ipoib_mode in
datagram)
    ;;

connected)
    export CFLAGS="-DIPOIB_CM"
    ;;

*)
    usage
    ;;

esac
    
cd /usr/src
make clean  # Rebuild all objs in case src was updated

cd sys/modules
for module in mlx$driver_version ibcore mlx${driver_version}ib ipoib; do
    rm -f /boot/modules/$module.ko
    cd $module
    make clean
    make DEBUG_FLAGS="-DVIMAGE=1"
    make install
    # Minimize damage if kldload triggers a panic.  Unlikely, but has happened
    # with experimental drivers.
    sync
    kldload $module || true
    auto-append-line ${module}'_load="YES"' /boot/loader.conf $0
    cd ..
done
kldstat
cat << EOM

To disable Infiniband support, remove or comment out the following device
loads in /boot/loader.conf:

mlx$driver_version ibcore mlx${driver_version}ib ipoib

EOM

Following updates to /usr/src, the IB kernel modules can be upgraded using the script below:

 #!/bin/sh -e

usage()
{
    printf "Usage: $0 4|5 datagram|connected\n"
    printf "* Use 4 for ConnectX 3 and earlier, 5 for later HCAs.\n"
    printf "* Specify connected if you want to use connected mode ipoib.\n"
    exit 1
}

if [ $# != 2 ]; then
    usage
fi

driver_version=$1
ipoib_mode=$2

for interface in $(ifconfig | awk -F : '$1 ~ "^ib" { print $1 }'); do
    ifconfig $interface down
done

kldunload ipoib mlx${driver_version}ib ibcore mlx$driver_version || true

auto-build-infiniband-modules $1 $2

for interface in $(ifconfig | awk -F : '$1 ~ "^ib" { print $1 }'); do
    ifconfig $interface up
done

Both of these scripts will be included in sysutils/autoadmin in the near future.

OpenSM

To start OpenSM automatically when the system boots, add opensm_enable="YES" to /etc/rc.conf, e.g.

# sysrc opensm_enable="YES"

Status

Mellanox engineers, et. al., have made substantial improvements to the IB stack in 2015/2016 to support the latest ConnectX HCAs. The freebsd-infiniband email archives contain discussions of progress and issues.

There are some outstanding work items for Infiniband/OFED on FreeBSD.

Wishlist

• WITH_OFED doesn't cover all OFED stack utilities; a number of them need to be ported from Linux and contributed back to OFED (preferred) or rewritten to work with FreeBSD.

Feedback

There's an infiniband mailing list. If you're having issues or would like to provide feedback about the Infiniband stack on FreeBSD, you should subscribe and poke the mailing list.

References

• http://en.wikipedia.org/wiki/InfiniBand

• FreeBSD9

• GlusterFS

• http://www.openfabrics.org/

Everything below is "work in progress". It will be moved into the correct place on this page when it's ready

Where are the sources?

• contrib/ofed - all of the userspace libraries and tools, e.g. opensm(8).
• sys/ofed - all of the kernel code and drivers, e.g. mlx4(4).

Configuration / runtime settings

TODO: We should explain the relevant sysctl.conf, loader.conf, and rc.conf settings.

/etc/sysctl.conf

NOTE (ngie): these items should be documented in the device driver/network stack specific manpages to reduce duplication.

• net.inet.sdp.pcblist
• mlx4 (this :
• Mode switching
  • sys.device.mlx4_core0.mlx4_portX (valid values "ib" or "eth")

/boot/loader.conf

• See driver-specific manpages for more details, e.g. "man 4 mlx4en".

/etc/rc.conf

• IP configuration settings (ifconfig_ibX, ifconfig_mlxenX)
• opensm_enable

Operating modes

We should add information here about the 2 main operating modes for the Mellanox Infiniband adapters:

1. Native infiniband mode
2. Native ethernet mode

It would be useful to clarify that in Infiniband mode, IP can be encapsulated and sent using IPoIB.

Cables & connectors

We should add info and photo's here for the different cables & connector types

• Explain passive vs active

• Copper & optical cable

• CX4 connectors ← JustinClift has some of these. Can take photos

• SFP

• QSFP connectors ← JustinClift has some of these. Can take photos

Maximum cable length

It would also be useful to explain the maximum length of cables, according to type, connector, data rate (and whatever else).