Improving libvirt support for bhyve
Student: Fabian Freyer (fabs@FreeBSD.org)
Mentor: Roman Bogorodskiy (novel@FreeBSD.org)
Project description
The primary aim of this project is to implement missing calls and functionality in the libvirt bhyve driver. According to the libvirt API Support Matrix, there are a large number of calls not yet implemented. While some missing API calls are not applicable to bhyve, a number of them are, and these are aimed to be implemented in the scope of the project.
Deliverables
This is a list of ideas of implementable features, however it is by no means complete, nor is the goal to implement all of these. It should rather be considered a braindump of possibilities:
General calls
virConnectDomainXMLFromNative: This would mostly be an argument parser for a bhyve(8) and bhyvectl(8) command line.
- virConnectGetCPUModelNames: This needs some research: bhyve is not very flexible in configuration of what CPU model is exposed to the guest, need to figure out if that’s worth implementing now.
Connection calls
Most of these include some form of authentication handling and are therefore not applicable. However, the following do apply to bhyve and are easy to implement:
General Domain calls
- virDomainGetMaxMemory: Since bhyve does not support memory ballooning, just return the amount of memory allocated here
virDomainGetMaxVcpus, virDomainGetVcpus: I would use the approach described in this mailing list thread: “Until the vCPU state is exposed by bhyvectl, or we provide a sysctl, you can use heuristics: the number of vCPU threads for the bhyve process, or scan all vCPUs and only count those that have a non-zero RIP.”
virDomainGetCPUStats: There are two ways to implement this, depending on whether usage should include or exclude the hypervisor time. If usage should include the hypervisor time, use standard CPU usage reporting as used by top et al. to determine CPU usage. If usage should exclude hypervisor time and include only vCPU time, use bhyvectl --get-stats to get the vcpu total runtime, and monitor this. I would have to do some research as to what to set this total runtime in relation to, and how to best monitor it.
virDomainInjectNMI: Call bhyvectl --inject-nmi
virDomainReset: Reset a bhyve VM with bhyvectl --force-reset and then clean things up using bhyvectl --destroy; update bhyve monitor code to handle exit code 0 from bhyve(8) that’s corresponding to reset (0 - reset / reboot, 1 - shutdown, 2 - halt) to trigger re-starting of the VM.
virDomainReboot: Reboot a bhyve VM. Since bhyve does not support any facilities to trigger an ACPI G2 state by killing the domain with SIGTERM, then polling for the domain to shut down and restart it afterwards. This would be a rather hackish solution, so I’d like to hear feedback on this. However, the QEMU driver does it this way too, so I'm guessing that will be ok.
Block-Device level calls
These would implement access to the vdev block storage layer. I plan to implement support for both file-backed and zvol-backed virtual machines for the following API calls:
- virDomainGetBlockInfo
- virDomainBlockPeek
- virDomainBlockCopy
- virDomainBlockStats
- virDomainBlockStatsFlags
Going further, since zvols support snapshotting, I plan to implement the following for zvol-backed storage
- virDomainBlockCommit
- virDomainBlockPull
VirtFS layer
This is blocked by getting virtfs-9p merged upstream I would like to create patches to support specifying filesystems when creating the domain as well as the following calls to be merged at a later time when VirtFS-9p support for bhyve becomes ready:
- virDomainFSFreeze
- virDomainFSThaw
Memory inspection
virDomainMemoryPeek: A guest’s memory space is exposed in /dev/vmm, so this call would have to read from that.
Virt-host-validate(1) bhyve support: Add bhyve support to virt-host-validate
- Check support of CPU supports features required for bhyve
Check if the vmm module is available
- Check if essential networking things are available (if_bridge(4), if_tap(4)) + nmdm(4) for console
Networking support
Currently the bhyve’s libvirt driver (as well as libxl/FreeBSD driver) only supports L2 interface bridging. There’s no support for upper level schemas like NAT for example. This is a huge task that involves research of what firefall is more applicable (ipfw or pf), designing of the firewall rules and the actual implemtation
PCI passthrough support
I.e. the bhyve ... -s 7:0,passthru,4/0/0 thing. Probably that will involve the HAL nodedev driver modifications.
Milestones
Communtiy Bonding Period
22 April - 23 May
- Improve ideas list
- Build a testing environment
Read HACKING and look through the K&R style guide again
- Write simple patches to get accustomed to the patch submitting process
- Get an overview of the codebase
First Work Period
23 May - 20 June
In this period I aim to better familiarize myself with the codebase and get a better overview of what features are realistic to implement within the scope of GSoC.
Goals for this period are:
- Implement basic calls
Implement virConnectDomainXMLFromNative
- Implement domain reset
Start work implementing support booting via UEFI. This will entail:
Introducing a new capability checking whether BHYVE supports -l bootrom.
- Researching the correct way to define the bootrom in the definition XML. At the moment several possibilities come to mind:
Checking whether the file specified in the <bootloader> tag is executable or not (if not executable, use it as a bootrom)
Specifying the firmware blob in a <loader type="pflash"> tag within the <os> tag.
Implementing the feature in virConnectDomainXMLToNative - especially omitting the loader command when a boot rom is specified needs some research if cases exist where bhyveload or grub-bhyve have to be called while booting using UEFI.
Implementing the feature in virConnectDomainXMLFromNative
- Research other features and define a plan for remaining time
Second Work Period
27 June - 15 August
Finish work on virConnectDomainXMLFromNative
Implement virConnectGetDomainCapabilities
Finish work on UEFI support
Update domain capabilities - here we could iterate over a list of UEFI firmwares, should they be installed on the system. Since it isn't really sure where they would be installed, do some research.
- Probably creating a port containing uefi firmwares would be a good idea.
- Implement domain reboot and reset
- Implement inject-nmi
Implement VNC display support for bhyve_graphics
Last Week
15 August - 23 August
Test Plan
Where possible, tests will be added to the libvirt tests.
Some testing ideas:
virConnectDomainXMLFromNative: Create a round trip converting XML to native and then back, as well as native to xml and back. This isn't really practicable since we lose information.
Tests have been implemented and/or updated for:
- bootrom support
The Code
WIP
I have a Fork on GitHub of the libvirt project, where I am pushing my code.
The following branches are currently used
libvirt:
gsoc/bhyve_lpc_bootrom: implement support for booting via UEFI firmware. Posted to Mailinglist: v1
gsoc/bhyve_gop: implement support for VNC framebuffer
gsoc/fix/bhyveCapsBuild: fix emulator binary in virBhyveCapsBuild
virt-manager
Ports:
D7223/#211154: allow USE_GCC dependencies to be specified individually for {BUILD_,RUN_,TEST_}DEPENDS.
I have also created a patch to bhyve_graphics:
Upstreamed
Connection Calls
126e630 virConnectGetType: Trivially return "BHYVE"
a1efc94 virConnectIsAlive: Trivially return 1, since "A connection will be classed as alive if it is [...] local" and /dev/vmm is local.
32aa9ed virConnectIsEncrypted: Trivially return 0, since bhyve does not support encrypted interfaces.
ef45eb9 virConnectIsSecure: Trivially return 1, since /dev/vmm is considered a "local connection".
b436a8a to dd23c38 gsoc/bhyveConnectDomainXMLFromNative Posted to Mailinglist: v1 v2 v3 v4 v5
32916b1 virConnectGetDomainCapabilities Posted to Mailinglist: v1
Bugfixes
843a70a gsoc/fix/test-wrap-argv.pl Posted to Mailinglist: v1
Ports
Useful links
The following contributions are related to my work:
[bhyve][patch] print list of supported PCI and LPC devices by Roman Bogorodskiy
D6999: Make kern.argmax boot tunable fixes make syntax-check on FreeBSD.