Introduction

bhyve, pronounced "beehive" is a hypervisor/virtual machine manager for FreeBSD that supports a wide range of guest operating systems on Intel and AMD processors that support the "POPCNT" (POPulation Count) feature, and experimentally ARM64/aarch64 processors that support the gic0: <ARM Generic Interrupt Controller v3.0> feature (visible in dmesg(8)).

bhyve supports multiple storage and networking back-ends, UEFI, FreeBSD loader, and GRUB booting, PCI Pass-Through (PPT), integrated VNC and 9pfs servers, and many more features.

Contents

  1. Introduction
  2. Events
  3. Presentations
  4. Frequently Asked Questions
    1. General
      1. Q: Is it "bhyve", "Bhyve", "BHyVe" or BHyve?
      2. Q: How do I configure bhyve and install a VM operating system?
      3. Q: Has bhyve been ported to other operating systems?
      4. Q: What VM operating systems does bhyve support?
      5. Q: What does bhyve "look" like?
      6. Q: What is the easiest way to try bhyve?
      7. Q: What is required to support Graphical UEFI and operating systems like Microsoft Windows?
      8. Q: Does libvirt support bhyve?
      9. Q: Can I run multiple bhyve hosts under VMware nested VT-x EPT?
      10. Q: Can I run multiple bhyve hosts under Linux KVM with nested VT-x EPT?
      11. Q: Can I use disk images from QEMU, VMware, or other virtualization platforms on bhyve?
    2. Technical Questions
      1. Q: What comprises bhyve?
      2. Q: Does bhyve support VT-d PCI device pass-through?
      3. Q: Does bhyve support AHCI devices?
      4. Q: Does bhyve support TPM emulation?
      5. Q: Does bhyve have an "out-of-band"/"lights-out management" serial console?
      6. Q: Is there any way to determine if an OS is virtualized?
      7. Q: How is a FreeBSD VM 9pfs share configured?
      8. Q: How is a FreeBSD root-on-9pfs VM configured?
    3. Troubleshooting
      1. Q: What will I see if my CPU does not support EPT?
      2. Q: What will I see if EPT is disabled in BIOS?
      3. Q: What will I see if a VM name is over 31 characters long?
      4. Q: Why does my system appear to stop booting just after "Starting background file system checks in 60 seconds."?
      5. Q: What is the string I need to add to "/etc/ttys" for older versions of FreeBSD?"
      6. Q: Why won't vmrun.sh VMs archived with tar?
      7. Q: Why can't I network VMs over a wireless host interface?
      8. Q: Why does my VM fail with "vm_run error -1, errno 25" ?
      9. Q: Are there any debugging options for bhyve?
      10. Q: Why can't my nested VMs reach the network?
    4. Participation
      1. Q: Is there a mailing list or IRC channel?
      2. Q: How can I help?
      3. Q: How do I report a panic?

Events

Presentations

Frequently Asked Questions

General

Q: Is it "bhyve", "Bhyve", "BHyVe" or BHyve?

A: The developers mercifully retired "BHyVe" and simply refer to it as "bhyve", after the utility.

Q: How do I configure bhyve and install a VM operating system?

A: Begin with the FreeBSD Handbook chapter and manual page.

Q: Has bhyve been ported to other operating systems?

Yes. bhyve is available in illumos and its distributions, plus Project ACRN.

Q: What VM operating systems does bhyve support?

A: bhyve supports any version of FreeBSD i386/amd64. OpenBSD, NetBSD, illumos and GNU/Linux are supported using the UEFI and the sysutils/grub2-bhyve port.

Tested GNU/Linux distributions include:

Tested other Operating systems include:

Most guest operating systems will boot with the UEFI-GOP EDK2 firmware, but see the grub2-bhyve page for detailed sysutils/grub2-bhyve instructions.

Some operating systems will require the -w bhyve flag.

Q: What does bhyve "look" like?

To load a FreeBSD kernel from disk image vm0.img with 256M RAM and the name vm0 on and AMD64 system: 

/usr/sbin/bhyveload -m 256 -d ./vm0.img vm0

This will show the FreeBSD loader screen and you should see the device /dev/vmm/vm0

To boot the VM with 2 vCPUs, the same 256M RAM and the tap0 network interface: 

/usr/sbin/bhyve -c 2 -m 256 -A -H -P \
  -s 0:0,hostbridge \
  -s 1:0,virtio-net,tap0 \
  -s 2:0,ahci-hd,./vm0.img \
  -s 31,lpc -l com1,stdio \
  vm0

After the VM has been shutdown, its resources can be reclaimed with: 

/usr/sbin/bhyvectl --destroy --vm=vm0

Destruction of a VM can also be completed by bhyve when powering off a VM by adding the -D flag. To simplify bhyve calls, one can put configuration settings into a configuration file instead of specifying a list of flags and options; this config file can be referenced via the -k <filename> option. To construct a configuration file, call bhyve like you usually would and add the flag -o config.dump=1: 

/usr/sbin/bhyve -c 2 -m 256 -A -H -P \
  -s 0:0,hostbridge -s 1:0,virtio-net,tap0 \
  -s 2:0,ahci-hd,./vm0.img \
  -s 31,lpc -l com1,stdio \
  -o config.dump=1 vm0 > configfile

The config file then contains the relevant options: 

memory.size=256
x86.strictmsr=true
x86.vmexit_on_hlt=true
x86.vmexit_on_pause=true
cpus=2
acpi_tables=true
pci.0.0.0.device=hostbridge
pci.0.1.0.device=virtio-net
pci.0.1.0.backend=tap0
pci.0.2.0.device=ahci
pci.0.2.0.port.0.type=hd
pci.0.2.0.port.0.path=./vm0.img
pci.0.31.0.device=lpc
lpc.com1.path=stdio
config.dump=1
name=vm0

Before using this file with the -k option, remove the config.dump=1 line.

ARM64 virtual machines require the u-boot-bhyve-arm64 package and can be booted with: 

bhyve -o bootrom=/usr/local/share/u-boot/u-boot-bhyve-arm64/u-boot.bin \
-s 1,virtio-blk,./FreeBSD-15.0-CURRENT-arm64-aarch64-ufs.raw \
-c console=stdio \
vm0

The U-Boot loader will boot the first VirtIO block device.

Q: What is the easiest way to try bhyve?

The included method booting FreeBSD virtual machines is the /usr/share/examples/bhyve/vmrun.sh script: 

fetch https://download.freebsd.org/snapshots/VM-IMAGES/14.0-STABLE/amd64/Latest/FreeBSD-14.0-STABLE-amd64.raw.xz

unxz FreeBSD-14.0-STABLE-amd64.raw.xz

sudo sh /usr/share/examples/bhyve/vmrun.sh -d FreeBSD-14.0-STABLE-amd64.raw vm0

There are numerous helper scripts and applications in ports, that help you set up more complex environments, i.e.

Q: What is required to support Graphical UEFI and operating systems like Microsoft Windows?

See the bhyve/UEFI and bhyve Windows Virtual Machines wiki pages.

Q: Does libvirt support bhyve?

A: Yes

Q: Can I run multiple bhyve hosts under VMware nested VT-x EPT?

A: Yes, but you must enable it: look for the GUI or vhv.allow or vhv.enable options, depending on your version of VMware. VMware Fusion: Processors & Memory: Advanced options: Enable hypervisor applications in this virtual machine

Confirmed with VMware Workstation, VMware Player, VMware Fusion and ESXi. Note that you must stop all running VMs before you can enable EPT pass-through.

Q: Can I run multiple bhyve hosts under Linux KVM with nested VT-x EPT?

A: Maybe. You must enable it and note the known limitations. Please let us know if you are successful.

Q: Can I use disk images from QEMU, VMware, or other virtualization platforms on bhyve?

A: bhyve does not currently support reading foreign disk image formats directly, so one will have to convert existing VMDK, QCOW, QCOW2, and VDI disk images to a raw image using qemu-img. This can be accomplished by doing the following on a FreeBSD system: 

$ pkg install qemu-tools
$ qemu-img convert -f vmdk -O raw vmware_image.vmdk bhyve_raw_image.raw

Technical Questions

Q: What comprises bhyve?

A: The vmm.ko loadable kernel module, the libvmmapi library, the bhyve, bhyveload and bhyvectl utilities.

In total these binaries are under 500K in size.

Q: Does bhyve support VT-d PCI device pass-through?

A: Yes, on Intel CPUs. See bhyve/pci_passthru.

Q: Does bhyve support AHCI devices?

A: Yes, and they provide non-blocking I/O for higher performance. CD: -s 4,ahci-cd,/path/to/image.iso HD: -s 4,ahci-hd,/path/to/disk.img

Q: Does bhyve support TPM emulation?

As of 14.0-RELEASE, bhyve supports TPM passthrough and TPM emulation - see bhyve(8) man page and YouTube for details.

Q: Does bhyve have an "out-of-band"/"lights-out management" serial console?

A: Yes, and it can be accessed several ways including as a nmdm nullmodem terminal or a pty. stdio output can also be sent to a terminal multiplexer like tmux or screen. Example nmdm syntax is:

-s 31,lpc -l com1,stdio Becomes: -s 31,lpc -l com1,nmdm0A Before starting the VM, run: kldload nmdm Start the VM but note that it will not display any output when booted. Connect to the serial interface with: cu -l /dev/nmdm0B -s 9600

Notes: A is connected to B nmdmN is a a 32 bit unsigned integer so you have plenty to use. ptys can be created with socat. This functionality is included in the FreeBSD 10.0 release.

Q: Is there any way to determine if an OS is virtualized?

A: Yes, the kern.vm_guest sysctl will report if a FreeBSD guest is under KVM or bhyve.

Q: How is a FreeBSD VM 9pfs share configured?

A: Using a FreeBSD 15.0-CURRENT host from 2024-06-19 or later:

Create a directory to share such as /root/9p

Virtual Machine /boot/loader.conf: 

virtio_p9fs_load="YES"

Virtual Machine /etc/fstab: 

sharename / p9fs rw 0 0

Boot the VM adding a device for the 9pfs directory: 

-s 3,virtio-9p,sharename=/root/9p/

The share can be manually mounted with: 

kldload virtio_p9fs
mount -t p9fs sharename /mnt

Q: How is a FreeBSD root-on-9pfs VM configured?

A: Using a FreeBSD 15.0-CURRENT host from 2024-07-10 or later:

Create a directory to share such as /root/9p and populate it with a bootable userland, such as one copied from a VM-IMAGE.

Virtual Machine /boot/loader.conf: 

virtio_p9fs_load="YES"
vfs.root.mountfrom="p9fs:sharename"

Virtual Machine /etc/fstab 

sharename / p9fs rw 0 0

Load and boot the VM using the directory: 

bhyveload -m 1024 -h /root/9p vm0

bhyve -m 1024 -A \
        -l com1,stdio \
        -s 0,hostbridge \
        -s 3,virtio-9p,sharename=/root/9p/ \
        -s 31,lpc \
        vm0

Troubleshooting

Q: What will I see if my CPU does not support EPT?

# kldload vmm
vmx_init: processor does not support VMX operation
module_register_init: MOD_LOAD (vmm, 0xffffffff81e17169, 0) error 6

Q: What will I see if EPT is disabled in BIOS?

# sh /usr/share/examples/bhyve/vmrun.sh ...
Launching virtual machine "...
vm_create: Device not configured

Q: What will I see if a VM name is over 31 characters long?

vmcreate: Invalid argument

Q: Why does my system appear to stop booting just after "Starting background file system checks in 60 seconds."?

A: This indicates that your /etc/ttys is not properly configured for serial output for use with bhyve.

Q: What is the string I need to add to "/etc/ttys" for older versions of FreeBSD?"

ttyu0 "/usr/libexec/getty 3wire.9600"  vt100  on secure

Q: Why won't vmrun.sh VMs archived with tar?

A: vmrun.sh creates disk images using the truncate(1) command and as per the tar(1) manual page, "There is not yet any support for multi-volume archives or for archiving sparse files."

Q: Why can't I network VMs over a wireless host interface?

A: Access points cannot determine the destination VMs and only see the host.

Q: Why does my VM fail with "vm_run error -1, errno 25" ?

A: This can happen if your kernel and world are out of sync.

Q: Are there any debugging options for bhyve?

A: Yes: The gdb interface.

Q: Why can't my nested VMs reach the network?

A: The bottom most hypervisor needs to run the virtual interface in promiscuous mode in order for packets destined for an ethernet address other than that of the first level guest to be received by the virtual interface.

Enabling Promiscuous mode in VMWare ESX

Participation

Q: Is there a mailing list or IRC channel?

A: The most popular places to discuss bhyve are the freebsd-virtualization@freebsd.org mailing list and the #bhyve channel on irc.libera.chat.

Q: How can I help?

A: First and foremost, by testing bhyve on your unique hardware and in every configuration you can imagine. There could easily be remaining bugs and it is in everyone's best interests for these to be resolved ASAP.

If you think you have an issue, considering asking about it on IRC. If you know you have an issue, consider posting it on the FreeBSD Virtualization mailing list.

Testing ideas:

Q: How do I report a panic?

A: If your host or VM panics and enters the debugger, the information you provide can be very useful for revealing and reproducing bugs. Typing "bt" at the debugger will produce a backtrace and "show thread" on various hex strings may yield more. Please capture this information and use your best judgement in sending it to the bhyve developers or the freebsd-virtualization mailing list. If in doubt, I am happy to relay reports (address below).

Do consider using the sysutils/panicmail port with bhyve.

CategoryHowTo CategoryProject CategoryVirtualization

bhyve (last edited 2024-07-15T08:57:36+0000 by MichaelDexter)