bhyve PCI Passthrough
bhyve supports passing of host PCI devices to a virtual machine for its exclusive use of them.
Note: VGA / GPU pass-through devices are not currently supported.
- CPU supports Intel IOMMU (a.k.a. VT-d) feature
- PCI device (and driver) supports MSI/MSI-x interrupts
Host VT-d support can be determined by searching for a DMAR table in the ACPI tables with acpidump -t | grep DMAR
PCI card MSI/MSI-x support can be determined with pciconf -lc | grep MSI
1. Configure vmm.ko to be preloaded at boot-time in loader.conf, adding:
2. Determine the bus/slot/function of the device that you want to pass through to the guest with pciconf -v -l
- For example, to pass through a Atheros Gigabit Ethernet adapter:
alc0@pci0:2:0:0: class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00 vendor = 'Atheros Communications' device = 'AR8151 v2.0 Gigabit Ethernet' class = network subclass = ethernet
The bus/slot/function is for this devices is 2/0/0 (from the end of 'alc0@pci0:2:0:0')
3. Once the desired PCI device is identified, it and any others must be masked from the host early in the boot process with a PCI passthrough devices entries or "pptdevs" in loader.conf
The PCI device at bus/slot/function 2/0/0 would be masked from the host with:
Multiple devices can be added, space-separated, to the "pptdevs" entry
pptdevs="2/0/0 1/2/6 4/9/0"
The "pptdevs" entry supports up to 128 characters in the string. Any additional entries must be added with additional, numbered "pptdevsN" entries:
4. Rebooting the system will cause the requested PCI passthrough devices to be masked from the system and become available for use in a virtual machine
For example, to the Atheros Gigabit Ethernet adapter will be associated with device ppt0 in the virtual machine:
ppt0@pci0:2:0:0: class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00 vendor = 'Atheros Communications' device = 'AR8151 v2.0 Gigabit Ethernet' class = network subclass = ethernet
5. The slot option to the bhyve(8) command is used to bind the masked host PCI device to the guest using the b/s/f as the identifier. The guest slot/function has no correlation to the host b/s/f. For example, to assign the Atheros Gigabit Ethernet adapter to a guest at slot 7, the bhyve(8) command line would contain:
Caveats: multi-function devices are not always independent, and may have to be assigned to guests with the functions being the same. An example is QLogic FC adapters, where function 0 is used for firmware loading, while the other functions are for port data transfer. For these, the mappings of functions must be the same in the guest e.g.
-s 7:0,passthru,4/0/0 -s 7:1,passthru,4/0/1 -s 7:2,passthru,4/0/2
Intel network adapters *do not* have this issue and can be split out to different guest slots e.g. an Intel adapter at host 6/5/0 and with 2 functions could be setup in the guest as two separate devices at slot 3 and slot 8:
-s 3:0,passthru,6/5/0 -s 8:0,passthru,6/5/1
- The "-S" option must be passed to both bhyveload(8) and bhyve(8) to wire guest memory.
bhyve support AMD-Vi/IOMMU by change r317600. By default AMD-Vi passthrough support is disabled, set hw.vmm.amdvi.enable and reload vmm.ko to enable it.