bhyve PCI Passthrough

The bhyve hypervisor supports the passing of PCI devices belonging to the host through to a virtual machine for its exclusive use of them. The ability requires the Intel IOMMU (a.k.a. VT-d) CPU feature. The availability of this feature can be determined by seeing if there is a DMAR table in the ACPI tables i.e. acpidump -t | grep DMAR

1. Configure vmm.ko to be preloaded at boot-time in loader.conf, adding:

      vmm_load="YES"

2. Determine the bus/slot/function of the device that you want to pass through to the guest with 'pciconf -v -l'

alc0@pci0:2:0:0:        class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00
    vendor     = 'Atheros Communications'
    device     = 'AR8151 v2.0 Gigabit Ethernet'
    class      = network
    subclass   = ethernet

The bus/slot/function is for this devices is 2/0/0 (from the end of 'alc0@pci0:2:0:0')

3. Once the desired PCI device is identified, it and any others must be masked from the host early in the boot process with a PCI passthrough devices entries or "pptdevs" in loader.conf

The PCI device at bus/slot/function 2/0/0 would be masked from the host with:

    pptdevs="2/0/0"

3a. Note that multiple devices can be added to the "pptdevs" entry:

    pptdevs="2/0/0 1/2/6 4/9/0"

The "pptdevs" entry supports up to 128 characters in the string. Any additional entries must be added with additional, numbered "pptdevsN" entries:

    pptdevs2="123/2/0 123/3/0"

4. Rebooting the system will cause the requested PCI passthrough devices to be masked from the system and become available for use in a virtual machine

ppt0@pci0:2:0:0:        class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00
    vendor     = 'Atheros Communications'
    device     = 'AR8151 v2.0 Gigabit Ethernet'
    class      = network
    subclass   = ethernet

5. The slot option to the bhyve(8) command is used to bind the masked host PCI device to the guest using the b/s/f as the identifier. The guest slot/function has no correlation to the host b/s/f. For example, to assign the Atheros Gigabit Ethernet adapter to a guest at slot 7, the bhyve(8) command line would contain:

   -s 7,passthru,2/0/0

Caveats: multi-function devices are not always independent, and may have to be assigned to guests with the functions being the same. An example is QLogic FC adapters, where function 0 is used for firmware loading, while the other functions are for port data transfer. For these, the mappings of functions must be the same in the guest e.g.

   -s 7:0,passthru,4/0/0 -s 7:1,passthru,4/0/1 -s 7:2,passthru,4/0/2

Intel network adapters *do not* have this issue and can be split out to different guest slots e.g. an Intel adapter at host 6/5/0 and with 2 functions could be setup in the guest as two separate devices at slot 3 and slot 8:

   -s 3:0,passthru,6/5/0   -s 8:0,passthru,6/5/1

6. Note that post r284539 the "-S" option must be passed to both bhyveload(8) and bhyve(8) to wire guest memory. This is only required in FreeBSD 11.0 and above.

7. bhyve only supports passthrough devices that use MSI/MSI-x interrupts. If the PCI device does not have MSI/MSI-x capability or if the driver only uses legacy interrupts then this device will not work with bhyve passthrough

8. bhyve does not support VGA passthrough devices but work is under way to support them

9. bhyve support AMD-Vi/IOMMU by change r317600. By default AMD-Vi passthrough support is disabled, set hw.vmm.amdvi.enable and reload vmm.ko to enable it.


CategoryVirtualization

bhyve/pci_passthru (last edited 2017-05-08 23:48:32 by AnishGupta)