Grant-Table User-Space Device
Student: Akshay Jaggi (<jaggi AT SPAMFREE FreeBSD DOT org>)
Mentor: Roger Pau Monné (<royger AT SPAMFREE FreeBSD DOT org>)
Project description
To ensure efficiency there is a need to share page frames between Xen domains, which can then be used for purposes like bulk data transfer between domains, inter-domain communication, direct DMA of data from hardware devices into the target domain by a driver domain, etc. Dom0, being the privileged domain, has access to all the pages of all the domains, but the same is not true for domU unprivileged domains. Grant-Tables come to the rescue! Grant tables provide a mechanism to share and transfer pages between domains.
Each domain has its own grant table, a data structure that is shared with Xen, and which contains information about what kind of permissions other domains have on this domain’s pages. Two kinds of operations are supported: granting a page to another domain, and mapping the grant from another domain. Entries in the grant table are indexed by an integer - grant reference, which is generated by the granting domain and used by the receiving domain to map the shared page.
Although the kernel can utilise grant tables, there is no way for user-space applications to use them in FreeBSD. Hence, the goal of creating a grant table userspace device, which can be utilised by userspace applications to grant local memory and map grants from other domains.
Approach to solving the problem
Write a character device which will do all the required tasks. We will try to expose an API which is very similar to the linux user-space grant-table device, so that we can take advantage of existing handlers as much as possible.
The API exported by the device is comprised of two components:
Grant Allocator - Manages creation/deletion of grants from local memory to be shared with other domains.
Allocate Grant - Allocates a new page of kernel memory and creates a new grant reference for this page using the Xen Control Library (libxc).
Deallocate Grant - Deallocate the grant reference. If the page is not mapped by anyone anymore, free it.
Set Unmap Notification
Grant Mapper - Manages grants from other domains.
Map Grants - Prepare grants from other domains to allow them to be mapped to user-space.
Unmap Grants - Unmap grants previously mapped.
Virtual Address to Offset - return offset (used as an index to refer to the appropriate grants (explained later)) using virtual address where the start of this grants range was mapped. Used by programs which do not keep track of offset.
Set Max Grants
Set Unmap Notification
Both Allocate Grants and Map Grants return an offset which can be passed along with mmap() by the user-space application. The appropriate pages (allocated, or the ones received as a grant) are then mapped into the user-space application’s memory. We need to write a device that exposes this API; an interface to talk to the Xen control library to manage the grants; and a pager to manage allocation and deallocation of pages.
Grant tables are used extensively by QEMU in order to implement para-virtualized backend drivers, for instance, qdisk, in user-space. This project will allow us to utilise these backend drivers in a FreeBSD dom0 and thus also provide us a way to benchmark and test our implementation.
Deliverables
- Grant Table User Space Device
- Working QEMU qdisk backend for FreeBSD Dom0
Milestones
Community Bonding Period (April 22 - May 22)
I would like to spend this period learning more about FreeBSD and it’s people. Learning more about the different components (especially Xen) and talking to people who worked on those components. This will help me get more involved with FreeBSD as a whole. Apart from this, this period would also be utilised to come up with a detailed plan with respect to the project execution, so that we do not end up wasting time during Coding Period. Reading similar implementations (privcmd), and resolving any design issues that need to be sorted out, etc. should also be done in this period. Also, I’ll utilise this period to go through the style guidelines, etc.
Coding Period (May 23 - August 23)
Pre Midterm (May 23 - June 21)
My goal for midterms is to submit the grant table device for testing. This will pave the way for post-midterm work, and at the same time keep us on track. Work involves writing the device, pager and the libxc interface, with basic testing so as to make it capable enough for it to be tested by the community.
Midterm Evaluation (June 21 - June 27)
Post Midterm (June 28 - August 15)
Work involves making QEMU backend qdisk works with the newly designed device. Begin benchmarking and testing the implementation. Lookout for bugs, and bug-fixes. Post benchmarking, prepare and submit the device for release, along with the benchmarking results.
Pencils Down (August 16 - August 23)
Write project report and documentation. Finish up any remaining work, tidy up things, and submit code for final evaluation.
Test Plan
The best way to test the implementation would be to use existing user-space applications which utilise grant tables.
XEN can utilise the para-virtualized qemu-xen backend, qdisk, as a disk backend for DomU guests. This is only possible if the Dom0 host supports user-space grant table access, as qdisk is implemented in userspace.
With this project a FreeBSD Dom0 would be able to utilise these backends for guests, and it also gives us a robust way to test out implementation.
Possible tests include:
- Filesystem consistency checks after multiple read/write operations on a guest with different media sizes, using a FreeBSD Dom0 and qdisk as disk backend.
- Disk benchmarks (read/write speed, etc.) on guests with different media sizes and comparison amongst following configurations:
- FreeBSD Dom0, qdisk
- FreeBSD Dom0, tap
- Linux Dom0, qdisk
- Linux Dom0, tap
Final Submission
The three patches that completely summarise my work:
Usage Instruction:
Patch FreeBSD src with D7429 and D7293, buildkernel and installkernel.
- Add the xen-patch to /usr/ports/sysutils/xen-tools/files/ and modify /usr/ports/sysutils/xen-tools/MAKEFILE to use the newly added patch.
make and make install xen-tools.
- Reboot.
Now you can use and test the qdisk as a backend for DomUs, and can also test the gntdev device manually by granting and mapping grants from other domains.
Apart from these three main patches, I have added a few small patches/bug-fixes here and there.