A FreeBSD/bhyve version of the netmap virtual passthrough (ptnetmap) for VMs.
Student: StefanoGarzarella
Mentor: Luigi Rizzo
Project description
To avoid VMs networking bottlenecks, I recently worked with my advisors on ptnetmap: a netmap virtual passthrough for VMs (ptnetmap).
ptnetmap allows userspace applications running in a guest VM to safely use any netmap port with near-native performance (physical devices [14.88 Mpps], software switches [20 Mpps], shared memory channels [50 Mpps]).
I developed a preliminary version for linux/KVM hosts. For this GSoC project I'll develop host ptnetmap support for bhyve/FreeBSD.
Approach to solving the problem
Virtual Machines are widely used in Cloud services and Network Function Virtualization. Unfortunately, they can introduce performance bottlenecks, especially for intensive I/O operations that are very common in networking services. To avoid these bottlenecks, we present ptnetmap: a netmap virtual passthrough for VMs.
The preliminary version of ptnetmap for linux/KVM hosts is already done, and on this platform both linux and FreeBSD guest are able to send and receive 14.88 Mpps on 10Gbps physical link (line rate), talk at over 20 Mpps through VALE ports, and over 75 Mpps through netmap pipes.
Now I would like to add host ptnetmap support on bhyve and FreeBSD. The code will be completely BSD licensed. (We wrote ourselves the entire ptnemtap code for linux, so we own the copyright on it).
The implementation of ptnetmap will be based on our previous prototype above, which requires work in the following areas:
- netmap framework modifications both for the guest part and the host part, to implement the virtio-like communication and the shared netmap memory; (existing code ~ 1900 loc in netmap framework + 300 loc OS specific; estimated new code ~ 300 + 300 loc)
- guest device driver modifications for the setup of the notification mechanism and to share netmap memory; (estimated new code ~800 loc for virtio, assuming the existing netmap support for virtio developed by my advisor)
- bhyve extensions to expose netmap host memory to the guest and to exchange notifications. We need a mechanism like linux's eventfd to exchange notifications and a way to expose the host memory to the guest through PCI device. (estimated new code unknown)
Deliverables
D1 (due by week 3):
modify virtio device driver to allow the FreeBSD guest to use ptnetmap. (We can test it on linux-KVM where ptnetmap is already done.) Since the device driver is used for the setup, we expect to have the same performance of e1000.
D2 (due by mid-term):
map netmap host memory into the guest. We need to expose the memory of the netmap instance running in the host to the netmap instance running in the guest. Risk factor: In linux we used PCI BAR to expose the host memory to the guest, but we don't know if it is possible in easy way also in bhyve.
D3 (due by week 9):
kernel thread in netmap host adapter to implement ptnetmap on FreeBSD host. We expect to have the same performance of KVM (over 20 Mpps on VALE port; over 50 Mpps on netmap-pipe; line rate [14.88 Mpps] on Physical [10Gbps] NICs). The code that the kthread runs, is already done. We need to implement only the functions to create/delete kernel thread and to handle notifications. (~300 loc on linux).
D4 (due by the end of project):
implement notification mechanism in bhyve/FreeBSD. We want to have a similar mechanism to ioeventfd and irqfd available in linux, to exchange notifications between guest and host threads. Risk factor: We don't know how long it takes because in linux we had eventfd already implemented.
Milestones
Start date: 2015/05/25
Estimated end dates: 2015/08/17-21
Timetable:
Week 1-2: add ptnetmap support to virtio device driver and QEMU frontend [D1]. --- DONE ---
Week 3: test, bugfix, and performance evaluation [D1]. --- DONE ---
Week 4-5: write code to map netmap host memory into the guest [D2]. --- DONE ---
Week 6: test and bugfix [D2]. prepare documents to mid-term evaluation. --- DONE ---
Week 7-8: implement kernel thread in netmap host adapter [D3]. --- DONE ---
Week 9: test and bugfix [D3]. --- DONE ---
Week 10: implement notification mechanism in bhyve/freebsd [D4]. --- DONE ---
Week 11: test and performance evaluation [D4]. --- DONE ---
Week 12: clean code and prepare documents to final evaluation. --- DONE ---
The Code
Useful links
ptnetmap slides - AsiaBSDCon 2015
Virtual device passthrough for high speed VM networking Stefano Garzarella, Giuseppe Lettieri, Luigi Rizzo