bhyve, the BSD Hypervisor
bhyve is a hypervisor/virtual machine manager for FreeBSD that was introduced in FreeBSD 10.0
bhyve Frequently Asked Questions
Q: Is it "bhyve", "BHyVe" or BHyve?
A: The developers mercifully retired the CamelCase name "BHyVe" and simply refer to it as "bhyve".
Q: When was bhyve first released?
bhyve was officially released on January 20th, 2014 as part of FreeBSD 10.0.
Q: What hardware does it run on?
A: bhyve supports Intel processors with Extended Page Tables. Processor EPT compatibility can be determined at ark.intel.com but most Atom C2000, Core i3, i5, i7 and related Xeon processors are supported. Presence of the "POPCNT" (POPulation Count) processor feature in dmesg(8) indicates EPT support.
AMD RVI (Rapid Virtualization Indexing, formerly Nested Page Tables) was merged into FreeBSD CURRENT on October 21st, 2014 with SVN release tag r273375
"Barcelona" class and newer AMD processors include the required RVI extension and as with Intel processors, presence of the "POPCNT" (POPulation Count) processor feature in dmesg(8) indicates RVI support. Note some processors such as Kuma core have POPCNT feature but lacks required "NRIPS" (Next RIP Save) feature.
Q: Does bhyve support ARM A15/v8 virtualization extensions?
A: Not yet. This is being investigated.
Q: What versions of FreeBSD does bhvye support as a host?
A: bhyve is included in FreeBSD/amd64 10.0-RELEASE amd64, with development being done in CURRENT.
Q: Could bhyve be ported to other operating systems?
Yes, and commit records of contributed patches suggest that it already has been.
Q: Does bhyve support i386 VMs?
A: Yes, as of February 5th, 2014: r261504
Q: What VM operating systems does bhyve support?
A: bhyve supports any version of FreeBSD i386/amd64 with VirtIO support, plus OpenBSD, NetBSD and Linux using the sysutils/grub2-bhyve port to load them:
- FreeBSD 8.4-RELEASE amd64/i386 and 8-STABLE amd64
- FreeBSD 9.*-RELEASE amd64/i386 9-STABLE amd64/i386
- FreeBSD 10 (All amd64/i386 versions)
- FreeBSD 11-CURRENT amd64/i386
- OpenBSD amd64/i386 5.2 and newer
- Linux amd64/i386
- NetBSD amd64 6.1 and newer
Tested GNU/Linux distributions include:
- CentOS/RHEL 6.2, 6.3, 6.4, 6.5 and 7.0
- Debian 6.0.7, 7.0, 7.6 netinstall i386/amd64
- Fedora 20
- OpenSUSE 12.3 and 13.1 amd64
- Ubuntu 10.04, 12.04, 13.04, 14.04 and 14.10 server i386/amd64
See the grub2-bhyve page for detailed sysutils/grub2-bhyve instructions and for best results use FreeBSD CURRENT with the -w bhyve flag.
Illumos VM support is under development.
Q: What does bhyve "look" like?
To load a FreeBSD kernel from disk image vm0.img with 256M RAM and the name vm0:
/usr/sbin/bhyveload -m 256 -d ./vm0.img vm0
This will show the FreeBSD loader screen and you should see the device /dev/vmm/vm0
To boot the VM with 2 vCPUs, the same 256M RAM and the tap0 network interface:
/usr/sbin/bhyve -c 2 -m 256 -A -H -P \ -s 0:0,hostbridge \ -s 1:0,virtio-net,tap0 \ -s 2:0,ahci-hd,./vm0.img \ -s 31,lpc -l com1,stdio \ vm0
After the VM has been shutdown, its resources can be reclaimed with:
/usr/sbin/bhyvectl --destroy --vm=vm0
Q: What is the easiest way to try bhyve?
Starting June 15th, 2014, the official FreeBSD 11.0-CURRENT snapshots include a bhyve-aware console and raw VM images that can be used with the /usr/share/examples/bhyve/vmrun.sh script:
fetch ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/11.0-CURRENT/amd64/Latest/FreeBSD-11.0-CURRENT-amd64.raw.xz unxz FreeBSD-11.0-CURRENT-amd64.raw.xz sudo sh /usr/share/examples/bhyve/vmrun.sh -d FreeBSD-11.0-CURRENT-amd64.raw vm0
Q: How do I configure bhyve and install a VM operating system?
The official way to run bhyve is with the /usr/share/examples/bhyve/vmrun.sh script (instructions) written by bhyve developer neel@. The latest version can always be found at svnweb.freebsd.org and it has seen a few updates since the 10.0-RELEASE.
Q: What was that string I need to add to "/etc/ttys" again?"
ttyu0 "/usr/libexec/getty 3wire.9600" vt100 on secure
Q: Is this requirement scheduled to go away?
Yes. The r260913 patches lay the groundwork for auto-enabling the serial console and FreeBSD 9.3, 10.1 and 11-CURRENT all include the permanent fix. The same goes for the snapshot VM-IMAGES.
Q: Does libvirt support bhyve?
Q: Are there other frameworks for bhyve?
A: There is vmrc among others.
Q: Can I run multiple bhyve hosts under VMware nested VT-x EPT?
A: Yes, but you must enable it: look for the GUI or vhv.allow or vhv.enable options, depending on your version of VMware. VMware Fusion: Processors & Memory: Advanced options: Enable hypervisor applications in this virtual machine
Confirmed with VMware Workstation, VMware Player, VMware Fusion and ESXi. Note that you must stop all running VMs before you can enable EPT pass-through.
Q: Can I run multiple bhyve hosts under Linux KVM with nested VT-x EPT?
Q: Can I run bhyve and VirtualBox on FreeBSD at the same time?
A: No. See this thread for more information.
Q: Does VirtualBox support nested VT-x EPT?
Q: Does Hyper-V support nested VT-x EPT?
Q: Does bhyve support nested VT-x EPT (bhyve under bhyve)?
A: Not yet. This is being investigated.
Q: Does Debian GNU/kFreeBSD support bhyve?
Q: Can I use disk images from VirtualBox or other virtualization platforms on bhyve?
A: bhyve does not currently support reading foreign disk image formats directly, so one will have to convert existing disk images to a raw image using qemu-img. This can be accomplished by doing the following on a FreeBSD system:
$ pkg install qemu-devel $ qemu-img convert -f vmdk -O raw virtual_box_image.vmdk bhyve_raw_image.img
Here we first ensure qemu-devel is installed as that contains the qemu-img utility. Next take a .vmdk file from a VirtualBox machine and covert it into a raw disk image file after installing qemu-devel. This same process should also work if for converting QCOW, QCOW2, VDI or other image formats supported by qemu-img.
Q: What comprises bhyve?
A: The vmm.ko loadable kernel module, the libvmmapi library, the bhyve, bhyveload and bhyvectl utilities.
In total these binaries are under 500K in size.
Q: Does bhyve have any dependencies?
A: The bhyve kernel module requires CPU EPT support and bhyve VMs uses the VirtIO (MSI and MSI-X) set of devices for virtualized storage and network devices. The FreeBSD GENERIC kernel has included VirtIO drivers beginning with the 8.4, 9.2 and 10.0 amd64 releases and their equivalent STABLE branches.
Q: Does bhyve support VT-d PCI device pass-through?
A: Yes, on Intel CPUs. See wiki.freebsd.org/bhyve/pci_passthru
Q: Does bhyve support AHCI devices?
A: Yes, and they provide non-blocking I/O for higher performance. CD: -s 4,ahci-cd,/path/to/image.iso HD: -s 4,ahci-hd,/path/to/disk.img
Q: What boot media are supported?
A: Raw disk images and any block device such as ZFS zvols and iSCSI targets, plus ISOs using the AHCI driver.
Q: Does bhyve have its own MAC address allocation?
A: Yes, as of commit r258141 on November 14th, 2013, FreeBSD has its own allocation of IEEE Organizationally Unique Identifiers (hex: 58-9C-FC / base16: 589cfc) and bhyve will be allocated a sub-set of these.
Q: Does bhyve have an "out-of-band"/"lights-out management" serial console?
A: Yes, and it can be accessed several ways including as a nmdm nullmodem terminal or a pty. stdio output can also be sent to a terminal multiplexer like tmux or screen. Example nmdm syntax is:
-s 31,lpc -l com1,stdio Becomes: -s 31,lpc -l com1,nmdm0A Before starting the VM, run: kldload nmdm Start the VM but note that it will not display any output when booted. Connect to the serial interface with: cu -l /dev/nmdm0B -s 9600
Notes: A is connected to B nmdmN is a a 32 bit unsigned integer so you have plenty to use. ptys can be created with socat. This functionality is included in the FreeBSD 10.0 release.
Q: Does bhyve have ACPI support?
A: Yes, support for a proper "power down" at shutdown arrived with SVN commit r259826. Full suspend and resume is under development.
Q: Is there any way to determine if an OS is virtualized?
A: Yes, the kern.vm_guest sysctl will report if a FreeBSD guest is under KVM or bhyve.
Q: Where are the sources?
A: The source is in FreeBSD SVN source repository: sys/amd64/vmm/, usr.sbin/bhyve/, usr.sbin/bhyveload/, usr.sbin/bhyvectl/, and lib/libvmmapi/
Q: What will I see if my CPU does not support EPT?
# kldload vmm vmx_init: processor does not support VMX operation module_register_init: MOD_LOAD (vmm, 0xffffffff81e17169, 0) error 6
Q: What will I see if EPT is disabled in BIOS?
# sh /usr/share/examples/bhyve/vmrun.sh ... Launching virtual machine "... vm_create: Device not configured
Q: What will I see if a VM name is over 31 characters long?
vmcreate: Invalid argument
Q: Why does my system appear to stop booting just after "Starting background file system checks in 60 seconds."?
A: This indicates that your /etc/ttys is not properly configured for serial output for use with bhyve.
Q: Why is my VM only booting once I attach to the "nmdm" device?
A: This is a bug has been fixed with r262884.
Q: Why won't vmrun.sh VMs archive with tar?
A: vmrun.sh creates disk images using the truncate(1) command and as per the tar(1) manual page, "There is not yet any support for multi-volume archives or for archiving sparse files."
Q: Why can't I network VMs over a wireless host interface?
A: Access points cannot determine the destination VMs and only see the host. Workarounds are possible using pf(4).
Q: Why does mounting my FreeBSD with ZFS on root VM fail with "error 45" when "Mounting from zfs:" ?
A: You are probably attempting to boot an older version of FreeBSD on a zpool that you created with a newer version of FreeBSD and the available ZFS features are not available. A safe set of features to use with FreeBSD 9.2 and 10.0 are:
zpool create -d -o feature@async_destroy=enabled -o feature@empty_bpobj=enabled -o feature@lz4_compress=enabled -o ...
Q: Why does my VM fail with "vm_run error -1, errno 25" ?
A: This can happen if your kernel and world are out of sync.
Q: Are there any debugging options for bhyve?
A: Yes: The gdb interface.
Q: Why can't my nested VMs reach the network?
A: The bottom most hypervisor needs to run the virtual interface in promiscuous mode in order for packets destined for an ethernet address other than that of the first level guest to be received by the virtual interface.
Q: What is needed to support additional VM operating systems?
All operating systems load and boot differently and a simple grub2 "chainload" is not possible. Source availability and modifiability are obviously important when addressing this issue on other operating systems. VirtIO drivers are also important for a meaningful bhyve experience. bhyve supports MSI-X and MSI interrupts although MSI interrupts are not part of the VirtIO specification. We appreciate your research into other operating systems!
Q: Could bhyve support UEFI/BIOS?
A: Yes, this is a priority because it would simplify the booting of non-FreeBSD OSs and implementing integrated video support.
Q: Could bhyve support VGA graphics?
A: Yes, this is closely related to UEFI/BIOS support.
Q: Is there a mailing list or IRC channel?
Q: How can I help?
A: First and foremost, by testing bhyve on your unique hardware and in every configuration you can imagine. There could easily be remaining bugs and it is in everyone's best interests for these to be resolved ASAP.
If you think you have an issue, considering asking about it on IRC. If you know you have an issue, consider posting it on the FreeBSD Virtualization mailing list.
- High vCPU numbers
- High RAM allocations
- High CPU, network and storage loads
- Every piece of compatible hardware you can find
- PCI pass-through with every device you can find
- Every storage strategy you can think of including ZVOLs and iSCSI targets
- Watch for clock drift
- Test every possible VM OS
- Countless things only you will think of!
Q: How do I report a panic?
A: If your host or VM panics and enters the debugger, the information you provide can be very useful for revealing and reproducing bugs. Typing "bt" at the debugger will produce a backtrace and "show thread" on various hex strings may yield more. Please capture this information and use your best judgement in sending it to the bhyve developers or the freebsd-virtualization mailing list. If in doubt, I am happy to relay reports (address below).
Do consider using the sysutils/panicmail port with bhyve.
bhyvecon Tokyo 2014 Presentations
BAFUG March 2014 - Jenkins and bhyve
AsiaBSDCon 2014 - Nested Paging in bhyve
AsiaBSDCon 2014 - Visualizing Unix: Graphing bhyve, ZFS and PF with Graphite
bhyvecon Tokyo 2014 - bhyvecon.org
FOSDEM 2014 - What's new in FreeBSD 10? - Video
BSDNow Episode 020 - bhyve Mind
CapBUG - Live Demo
Sysadmin #5 in Paris - PDF
TechSNAP 111 - BSDCan Recap
AsiaBSDCon 2013 - Implements BIOS emulation support for BHyVe: A BSD Hypervisor
bhyve on BSDTalk - MP3 and OGG
MeetBSD California 2012 - PDF
BSDCan 2012 - BSD Multiplicity