PCI passthru with bhyve works on systems that have the Intel IOMMU (aka VT-d). This can be determined by seeing if there is a DMAR table in the ACPI tables i.e. acpidump -t | grep DMAR
1. Set up vmm.ko to be preloaded at boot-time.
- - edit /boot/loader.conf and put in a line
2. Determine the bus/slot/function of the device that you want to pass through to the guest. This can be done with 'pciconf -v -l'
- For example, to pass through the Atheros gigE adapter on my test machine:
alc0@pci0:2:0:0: class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00 vendor = 'Atheros Communications' device = 'AR8151 v2.0 Gigabit Ethernet' class = network subclass = ethernet
... the bus/slot/function is 2/0/0 (from the end of 'alc0@pci0:2:0:0')
3. Set up a loader environment variable so that vmm.ko's passthru module will grab the PCI device early in the boot sequence and prevent FreeBSD's driver for that device from attaching. The env variable contains the b/s/f of the device to pass through.
- - edit /boot/loader.conf and add
3a. Note that multiple devices can be added to this variable
pptdevs="2/0/0 1/2/6 4/9/0"
- .. up to a length of 128 characters in the string, at which point additional variables have to be used
4. After a reboot, the system will be ready to run with passthru.
For example, to the Atheros gigE adapter will be associated with ppt0:
ppt0@pci0:2:0:0: class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00 vendor = 'Atheros Communications' device = 'AR8151 v2.0 Gigabit Ethernet' class = network subclass = ethernet
5. The slot option to bhyve must be used to bind the host PCI device to the guest, using the b/s/f as the identifier. The guest slot/function has no correlation to the host b/s/f. For example, to assign the Atheros gigE adapter to a guest at slot 7, the bhyve commmand line would contain
Caveats: multi-function devices aren't always independent, and may have to be assigned to guests with the functions being the same. An example is QLogic FC adapters, where function 0 is used for f/w loading, while the other functions are for port data transfer. For these, the mappings of functions must be the same in the guest e.g.
-s 7:0,passthru,4/0/0 -s 7:1,passthru,4/0/1 -s 7:2,passthru,4/0/2
Intel network adapters *don't* have this issue and can be split out to different guest slots e.g. an Intel adapter at host 6/5/0 and with 2 functions could be setup in the guest as two separate devices at slot 3 and slot 8:
-s 3:0,passthru,6/5/0 -s 8:0,passthru,6/5/1
6. Note that post r284539 the "-S" option must be passed to both bhyveload and bhyve to wire guest memory. Also note that this is required only in FreeBSD 11.0 and above.
7. bhyve only supports passthru devices that use MSI/MSI-x interrupts. If the PCI device does not have MSI/MSI-x capability or if the driver only uses legacy interrupts then this device will not work with bhyve passthru.