bhyve PCI Passthrough

The bhyve hypervisor supports the passing of PCI devices belonging to the host through to a virtual machine for its exclusive use of them. This ability requires the Intel IOMMU (a.k.a. VT-d) CPU feature and that the PCI device supports MSI/MSI-x interrupts.

Host VT-d support can be determined by searching for a DMAR table in the ACPI tables with 'acpidump -t | grep DMAR'

PCI card MSI/MSI-x support can be determined with 'pciconf -lc | grep MSI'

If neither the device nor its driver support MSI/MSI-x interrupts, the device will not be supported.

1. Configure vmm.ko to be preloaded at boot-time in loader.conf, adding:

      vmm_load="YES"

2. Determine the bus/slot/function of the device that you want to pass through to the guest with 'pciconf -v -l'

alc0@pci0:2:0:0:        class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00
    vendor     = 'Atheros Communications'
    device     = 'AR8151 v2.0 Gigabit Ethernet'
    class      = network
    subclass   = ethernet

The bus/slot/function is for this devices is 2/0/0 (from the end of 'alc0@pci0:2:0:0')

3. Once the desired PCI device is identified, it and any others must be masked from the host early in the boot process with a PCI passthrough devices entries or "pptdevs" in loader.conf

The PCI device at bus/slot/function 2/0/0 would be masked from the host with:

    pptdevs="2/0/0"

3a. Note that multiple devices can be added to the "pptdevs" entry:

    pptdevs="2/0/0 1/2/6 4/9/0"

The "pptdevs" entry supports up to 128 characters in the string. Any additional entries must be added with additional, numbered "pptdevsN" entries:

    pptdevs2="123/2/0 123/3/0"

4. Rebooting the system will cause the requested PCI passthrough devices to be masked from the system and become available for use in a virtual machine

ppt0@pci0:2:0:0:        class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00
    vendor     = 'Atheros Communications'
    device     = 'AR8151 v2.0 Gigabit Ethernet'
    class      = network
    subclass   = ethernet

5. The slot option to the bhyve(8) command is used to bind the masked host PCI device to the guest using the b/s/f as the identifier. The guest slot/function has no correlation to the host b/s/f. For example, to assign the Atheros Gigabit Ethernet adapter to a guest at slot 7, the bhyve(8) command line would contain:

   -s 7,passthru,2/0/0

Caveats: multi-function devices are not always independent, and may have to be assigned to guests with the functions being the same. An example is QLogic FC adapters, where function 0 is used for firmware loading, while the other functions are for port data transfer. For these, the mappings of functions must be the same in the guest e.g.

   -s 7:0,passthru,4/0/0 -s 7:1,passthru,4/0/1 -s 7:2,passthru,4/0/2

Intel network adapters *do not* have this issue and can be split out to different guest slots e.g. an Intel adapter at host 6/5/0 and with 2 functions could be setup in the guest as two separate devices at slot 3 and slot 8:

   -s 3:0,passthru,6/5/0   -s 8:0,passthru,6/5/1

6. Note that post r284539 the "-S" option must be passed to both bhyveload(8) and bhyve(8) to wire guest memory. This is only required in FreeBSD 11.0 and above.

7. bhyve does not support VGA passthrough devices at this time

8. bhyve support AMD-Vi/IOMMU by change r317600. By default AMD-Vi passthrough support is disabled, set hw.vmm.amdvi.enable and reload vmm.ko to enable it.


CategoryVirtualization

bhyve/pci_passthru (last edited 2017-11-08 19:05:36 by MichaelDexter)