Porting bhyve to ARM-based platforms

Project description

Nowadays virtualization is an important mechanism being approached by both, hardware designers and software developers. Given this necessity, the BSD community created a new hypervisor supporting hardware assisted virtualization. It is known under the name of bhyve. The current version of bhyve supports only the x86-based platforms (Intel and AMD, both having support for hardware assisted virtualization). In the last two years ARM has made significant progress developing virtualization extensions, this being implemented with success on other well-known hypervisors (KVM and Xen). Given the fact that ARM is the leader in embedded devices market and they target on long-term the server market, it will be a necessity for bhyve supporting ARM virtualization extensions. This projects proposes to implement support for hardware assisted virtualization on ARM platforms. It is not a trivial porting project due to the fact that ARM virtualization extensions doesn't allow access to normal CPU operation, like x86-based architectures.

In general, virtualization extensions adds a new privilege level for being able to isolate different guest OSes. On x86-based architectures this new privilege level comes with a new set of instructions in order to manage multiple guests. At the same time, the hypervisor has access to all other normal instructions, as running in supervisor mode (kernel mode). This scenario is the best fit for a hosted hypervisor (Type-2) [1], as bhyve is. In a Type-2 hypervisor both, guest OSes and normal processes, can run at the same time. The new level of privilege in ARM-based architectures it's called HYP-MODE. Compared to the x86 architecture, this mode only enables us the virtualization extensions instructions and we don't have access to the normal supervisor-mode ones. Running a Type-2 hypervisor in HYP-MODE isn't possible. The HYP-MODE is a best fit for Type-1 (baremetal) hypervisors like XEN [2] (it can run only virtual machines). Not being able to run a hosted hypervisor in HYP-MODE brings an important drawback: one cannot use the mechanisms (e.g. like memory management) already in-place in a hosted hypervisor and need to implement new ones.

Approach to solving the problem

bhyve was originally developed only for 64bit x86 platforms due to the fact that 32bit processor are too old and doesn't have virtualization support (or if they have its incomplete and not widespread at all). When talking about ARM platforms, the story changes: all the market devices have only 32bit processors. The 64bit ARM architecture was released by ARM but there isn't any development board supporting 64bit processors (only some emulators: Qemu and VExpress from ARM). Also FreeBSD support for 64bit is under heavy development. However ARM released the Virtualization Extensions on 32bit processors and this will have aproximatively full compatibility with the 64bit platforms. We will go developing bhyve support for ARM on a 32bit platform (see bellow).

Problems that will need to be tackled in order to be able to run an unmodified GuestOS on top of bhyve using an ARM platform: - configure u-boot bootloader to remain in hyp-mode, not drop to svc. - cpu virtualization: write the code that switches between the highvisor and guest OS - memory virtualization: this is similar to x86 memory virtualization with a difference: the HYP-mode lowvisor needs to maintain its own pagetable being in another address space than the highvisor - interrupt virtualization: expose a vGIC for each GuestOS - timer virtualization: ARM offers support for this, but these are likely to be shared between the hypervisor and the Guest OS, thus the configuration by the GuestOS will trap in the hypervisor in order to ensure isolation.


Week 1

Week 2

Week 3-5

MIDTERM (week 6)

Week 7-9

Week 10-11

FINAL (week 12)

Test Plan

This kind of project is not suitable for a generic test plan (like unit tests, or action-based tests, etc). The real final goal of this project is running a FreeBSD GuestOS (a virtual machine) on top of bhyve using a CubieBoard2 platform. While coding different mechanisms of the virtualization infrastructure, we will verify the correctness by printing the state of the system at each important step. The progress will be given by the state of the GuestOS (how far it can boot).

Final Results and thoughts

The Code

SummerOfCode2015/PortingBhyveToArm (last edited 2016-02-11 16:03:06 by MihaiCarabas)