bhyve PCI Passthrough

bhyve supports passing of host PCI devices to a virtual machine for its exclusive use of them.

Requirements

CPU supports Intel IOMMU (a.k.a. VT-d) feature
PCI device (and driver) supports MSI/MSI-x interrupts

Host VT-d support can be determined by searching for a DMAR table in the ACPI tables with acpidump -t | grep DMAR

PCI card MSI/MSI-x support can be determined with pciconf -lc | grep MSI

Instructions

1. Configure vmm.ko to be preloaded at boot-time in loader.conf, adding:

```
      vmm_load="YES"
```

2. Determine the bus/slot/function of the device that you want to pass through to the guest with pciconf -v -l

For example, to pass through a Atheros Gigabit Ethernet adapter:

alc0@pci0:2:0:0:        class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00
    vendor     = 'Atheros Communications'
    device     = 'AR8151 v2.0 Gigabit Ethernet'
    class      = network
    subclass   = ethernet

The bus/slot/function is for this devices is 2/0/0 (from the end of 'alc0@pci0:2:0:0')

3. Once the desired PCI device is identified, it and any others must be masked from the host early in the boot process with a PCI passthrough devices entries or "pptdevs" in loader.conf

The PCI device at bus/slot/function 2/0/0 would be masked from the host with:

```
    pptdevs="2/0/0"
```

Multiple devices can be added, space-separated, to the "pptdevs" entry

```
    pptdevs="2/0/0 1/2/6 4/9/0"
```

The "pptdevs" entry supports up to 128 characters in the string. Any additional entries must be added with additional, numbered "pptdevsN" entries:

```
    pptdevs2="123/2/0 123/3/0"
```

4. Rebooting the system will cause the requested PCI passthrough devices to be masked from the system and become available for use in a virtual machine

For example, to the Atheros Gigabit Ethernet adapter will be associated with device ppt0 in the virtual machine:

ppt0@pci0:2:0:0:        class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00
    vendor     = 'Atheros Communications'
    device     = 'AR8151 v2.0 Gigabit Ethernet'
    class      = network
    subclass   = ethernet

5. The slot option to the bhyve(8) command is used to bind the masked host PCI device to the guest using the b/s/f as the identifier. The guest slot/function has no correlation to the host b/s/f. For example, to assign the Atheros Gigabit Ethernet adapter to a guest at slot 7, the bhyve(8) command line would contain:

```
   -s 7,passthru,2/0/0
```

Caveats: multi-function devices are not always independent, and may have to be assigned to guests with the functions being the same. An example is QLogic FC adapters, where function 0 is used for firmware loading, while the other functions are for port data transfer. For these, the mappings of functions must be the same in the guest e.g.

   -s 7:0,passthru,4/0/0 -s 7:1,passthru,4/0/1 -s 7:2,passthru,4/0/2

Intel network adapters *do not* have this issue and can be split out to different guest slots e.g. an Intel adapter at host 6/5/0 and with 2 functions could be setup in the guest as two separate devices at slot 3 and slot 8:

   -s 3:0,passthru,6/5/0   -s 8:0,passthru,6/5/1

Additional Notes

The "-S" option must be passed to both bhyveload(8) and bhyve(8) to wire guest memory.
bhyve support AMD-Vi/IOMMU by change r317600. By default AMD-Vi passthrough support is disabled, set hw.vmm.amdvi.enable and reload vmm.ko to enable it.

CategoryVirtualization CategoryHowTo