bhyve, the BSD Hypervisor

bhyve is a hypervisor/virtual machine manager for FreeBSD that was introduced in FreeBSD 10.0 and supports most Intel and AMD processors that report the "POPCNT" (POPulation Count) processor feature in dmesg(8).

Most Intel Atom C2000, Core i3, i5, i7 and related Xeon processors support bhyve but early "Core" processors may only support one virtual machine.

Most "Barcelona" class and newer AMD processors support bhyve but some, notably the "Kuma" core processors inclue POPCNT but lack the required "NRIPS" (Next RIP Save) feature.

bhyve Feature History by Source Reference and Date

Feature

Reference

Date

Identify missing back storage device in bhyve(8)

r202321

2016-05-27

UEFI-GOP Project Branch

r300829

2016-05-26

FreeBSD 10 Stable UEFI VM BIOS Support

r295124

2016-02-01

bhyvectl man page

r293745

2016-01-12

netmap VirtIO

r293459

2016-01-09

FreeBSD HEAD UEFI VM BIOS Support

r288522 & r288524

2015-10-02

xhyve OS X

GitHub

2015-06

AMD CPU Support

r273375

2014-10-21

i386 VM Support

r261504

2014-02-05

Automatic Serial tty Detection

r260913

2014-01-20

FreeBSD 10 Release

r260789

2014-01-16

ACPI Power-off Support

r259826

2013-12-24

nmdm Console Support

r258668

2013-11-27

MAC Address Allocation

r258141

2013-11-14

grub2-bhyve Port Added for Foreign VMs

r332589

2013-11-03

FreeBSD HEAD Import

r245652

2013-01-19

FreeBSD bhyve Project Import

r221906

2011-05-14

FreeBSD bhyve_ref Project Import

r221828

2011-05-13

bhyve Features Under Development

Feature

Status

Activity Date

Live VM Migration

Future

Future

VM Suspend and Resume

Future

Future

VMDK, VHD & QCOW2 Support

Note vmdktool

Future

Support null storage devices with bhyve(8)

Future

Future

Nested EPT Support

Import from OpenBSD?

Future

USB Pass-through

ETA 2016

2016

ATA Device Emulation

Phabricator

2016

VirtFS Support

GitHub

2016

ARM CPU Support

GSoC Project

2015

NE2000 Network Emulation

GSoC Project

2015

VM PXE Boot Support

GSoC Project

2015

bhyve Frequently Asked Questions

General Questions

Q: Is it "bhyve", "Bhyve", "BHyVe" or BHyve?

A: The developers mercifully retired "BHyVe" and simply refer to it as "bhyve", after the utility.

Q: How do I configure bhyve and install a VM operating system?

A: Begin with the FreeBSD Handbook chapter and manual page.

Q: Could bhyve be ported to other operating systems?

Yes. See the xhyve, Pluribus Netvisor efforts.

Q: What VM operating systems does bhyve support?

A: bhyve supports any version of FreeBSD i386/amd64 with VirtIO support. OpenBSD, NetBSD and GNU/Linux are supported using the using the sysutils/grub2-bhyve port. UEFI support is available for FreeBSD, GNU/Linux, SmartOS and Windows:

Tested GNU/Linux distributions include:

See the grub2-bhyve page for detailed sysutils/grub2-bhyve instructions and for best results use FreeBSD CURRENT with the -w bhyve flag.

Q: What does bhyve "look" like?

To load a FreeBSD kernel from disk image vm0.img with 256M RAM and the name vm0:

/usr/sbin/bhyveload -m 256 -d ./vm0.img vm0

This will show the FreeBSD loader screen and you should see the device /dev/vmm/vm0

To boot the VM with 2 vCPUs, the same 256M RAM and the tap0 network interface:

/usr/sbin/bhyve -c 2 -m 256 -A -H -P \
-s 0:0,hostbridge \
-s 1:0,virtio-net,tap0 \
-s 2:0,ahci-hd,./vm0.img \
-s 31,lpc -l com1,stdio \
vm0

After the VM has been shutdown, its resources can be reclaimed with:

/usr/sbin/bhyvectl --destroy --vm=vm0

Q: What is the easiest way to try bhyve?

The included method booting FreeBSD virtual machines is the /usr/share/examples/bhyve/vmrun.sh script:

fetch ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/11.0-CURRENT/amd64/Latest/FreeBSD-11.0-CURRENT-amd64.raw.xz

unxz FreeBSD-11.0-CURRENT-amd64.raw.xz

sudo sh /usr/share/examples/bhyve/vmrun.sh -d FreeBSD-11.0-CURRENT-amd64.raw vm0

Q: What is required to support Graphical UEFI and operating systems like Microsoft Windows?

See the bhyve UEFI and bhyve Windows Virtual Machines wiki pages.

Q: Does libvirt support bhyve?

A: Yes

Q: Are there other frameworks for bhyve?

A: There is vmrc among others, a search of Github will reveal several.

Q: Can I run multiple bhyve hosts under VMware nested VT-x EPT?

A: Yes, but you must enable it: look for the GUI or vhv.allow or vhv.enable options, depending on your version of VMware. VMware Fusion: Processors & Memory: Advanced options: Enable hypervisor applications in this virtual machine

Confirmed with VMware Workstation, VMware Player, VMware Fusion and ESXi. Note that you must stop all running VMs before you can enable EPT pass-through.

Q: Can I run multiple bhyve hosts under Linux KVM with nested VT-x EPT?

A: Maybe. You must enable it and note the known limitations. Please let us know if you are successful.

Q: Can I run bhyve and VirtualBox on FreeBSD at the same time?

A: No. See this thread for more information.

Q: Do VirtualBox or Hyper-V support nested VT-x EPT?

A: VirtualBox does not, but Hyper-V has nested EPT available to Windows Insiders.

Q: Does Debian GNU/kFreeBSD support bhyve?

A: Good question!

Q: Can I use disk images from VirtualBox or other virtualization platforms on bhyve?

A: bhyve does not currently support reading foreign disk image formats directly, so one will have to convert existing disk images to a raw image using qemu-img. This can be accomplished by doing the following on a FreeBSD system:

$ pkg install qemu-devel
$ qemu-img convert -f vmdk -O raw virtual_box_image.vmdk bhyve_raw_image.img

VMDK, QCOW, QCOW2, VDI and more images can be converted to raw images using the qemu-img utility in the qemu-devel port.

Technical Questions

Q: What comprises bhyve?

A: The vmm.ko loadable kernel module, the libvmmapi library, the bhyve, bhyveload and bhyvectl utilities.

In total these binaries are under 500K in size.

Q: Does bhyve support VT-d PCI device pass-through?

A: Yes, on Intel CPUs. See wiki.freebsd.org/bhyve/pci_passthru

Q: Does bhyve support AHCI devices?

A: Yes, and they provide non-blocking I/O for higher performance. CD: -s 4,ahci-cd,/path/to/image.iso HD: -s 4,ahci-hd,/path/to/disk.img

Q: Does bhyve have an "out-of-band"/"lights-out management" serial console?

A: Yes, and it can be accessed several ways including as a nmdm nullmodem terminal or a pty. stdio output can also be sent to a terminal multiplexer like tmux or screen. Example nmdm syntax is:

-s 31,lpc -l com1,stdio Becomes: -s 31,lpc -l com1,nmdm0A Before starting the VM, run: kldload nmdm Start the VM but note that it will not display any output when booted. Connect to the serial interface with: cu -l /dev/nmdm0B -s 9600

Notes: A is connected to B nmdmN is a a 32 bit unsigned integer so you have plenty to use. ptys can be created with socat. This functionality is included in the FreeBSD 10.0 release.

Q: Is there any way to determine if an OS is virtualized?

A: Yes, the kern.vm_guest sysctl will report if a FreeBSD guest is under KVM or bhyve.

Q: Where are the sources?

A: The source is in FreeBSD SVN source repository: sys/amd64/vmm/, usr.sbin/bhyve/, usr.sbin/bhyveload/, usr.sbin/bhyvectl/, and lib/libvmmapi/

Troubleshooting

Q: What will I see if my CPU does not support EPT?

# kldload vmm
vmx_init: processor does not support VMX operation
module_register_init: MOD_LOAD (vmm, 0xffffffff81e17169, 0) error 6

Q: What will I see if EPT is disabled in BIOS?

# sh /usr/share/examples/bhyve/vmrun.sh ...
Launching virtual machine "...
vm_create: Device not configured

Q: What will I see if a VM name is over 31 characters long?

vmcreate: Invalid argument

Q: Why does my system appear to stop booting just after "Starting background file system checks in 60 seconds."?

A: This indicates that your /etc/ttys is not properly configured for serial output for use with bhyve.

Q: What is the string I need to add to "/etc/ttys" for older versions of FreeBSD?"

ttyu0 "/usr/libexec/getty 3wire.9600"  vt100  on secure

Q: Why is my VM only booting once I attach to the "nmdm" device?

A: This is a bug has been fixed with r262884.

Q: Why won't vmrun.sh VMs archive with tar?

A: vmrun.sh creates disk images using the truncate(1) command and as per the tar(1) manual page, "There is not yet any support for multi-volume archives or for archiving sparse files."

Q: Why can't I network VMs over a wireless host interface?

A: Access points cannot determine the destination VMs and only see the host. Consider a simple pf(4) pf.conf configuration:

nat on wlan0 from bridge0:network to any -> (bridge0)
pass out all
pass in all

Q: Why does mounting my FreeBSD with ZFS on root VM fail with "error 45" when "Mounting from zfs:" ?

A: You are probably attempting to boot an older version of FreeBSD on a zpool that you created with a newer version of FreeBSD and the available ZFS features are not available. A safe set of features to use with FreeBSD 9.2 and 10.0 are:

zpool create -d -o feature@async_destroy=enabled -o feature@empty_bpobj=enabled -o feature@lz4_compress=enabled -o ...

Q: Why does my VM fail with "vm_run error -1, errno 25" ?

A: This can happen if your kernel and world are out of sync.

Q: Are there any debugging options for bhyve?

A: Yes: The gdb interface.

Q: Why can't my nested VMs reach the network?

A: The bottom most hypervisor needs to run the virtual interface in promiscuous mode in order for packets destined for an ethernet address other than that of the first level guest to be received by the virtual interface.

Enabling Promiscuous mode in VMWare ESX

Q: Why can't my VM acquire an address via DHCP?

Q: Why is the host and VM unable to communicate across the network?

Q: Why is the VM and systems on the network unable to communicate with each other?

Q: Why are connections interrupted, seemingly at random?

A: It sounds like you are having more or less random routing issues. If your hosts IP is assigned to a member of a bridge (for example the NIC itself), try moving the IP to the bridge itself instead.

Participation

Q: Is there a mailing list or IRC channel?

A: The most popular places to discuss bhyve are the freebsd-virtualization@freebsd.org mailing list and the #bhyve channel on freenode.net.

Q: How can I help?

A: First and foremost, by testing bhyve on your unique hardware and in every configuration you can imagine. There could easily be remaining bugs and it is in everyone's best interests for these to be resolved ASAP.

If you think you have an issue, considering asking about it on IRC. If you know you have an issue, consider posting it on the FreeBSD Virtualization mailing list.

Testing ideas:

Q: How do I report a panic?

A: If your host or VM panics and enters the debugger, the information you provide can be very useful for revealing and reproducing bugs. Typing "bt" at the debugger will produce a backtrace and "show thread" on various hex strings may yield more. Please capture this information and use your best judgement in sending it to the bhyve developers or the freebsd-virtualization mailing list. If in doubt, I am happy to relay reports (address below).

Do consider using the sysutils/panicmail port with bhyve.

bhyve Presentations


CategoryVirtualization

bhyve (last edited 2016-06-12 07:30:11 by MichaelDexter)