Xen bus_dma Grant Table Handler

Student: Vaibhav Kumar Gautam vaibhav@FreeBSD.org

Mentor: Roger Pau Monné royger@FreeBSD.org

Project Description

The OpenBSD has integrated grant references into its bus_dma(9) subsystem, allowing Xen drivers to use a more transparent integration with the rest of the OS. The aim of this project is to bring the OpenBSD bus_dma(9) grant table handlers from OpenBSD into FreeBSD, expand them in order to support all the grant operations (share and map), and modify the existing frontends and backend in FreeBSD in order to make use of them.

Grant tables represent a mechanism of passing references to pages of memory allocated by the guest across domains. This allows drivers for paravirtualized devices to take advantage of a standard approach to DMA memory management.

Approach to Solving the Problem

1. Study the existing implementation of bus_dma(9) in OpenBSD and FreeBSD and find out the similarities in data structures, functions, and APIs that could be directly reused.

2. Study the grant table implementation in OpenBSD and FreeBSD, and map the APIs as per their functionalities.

3. Implement the prototype and test it on the Kernel.

4. Replicate the implementation to other areas using bus_dma (9).

Deliverables

Mid-term deliverable: Have a Xen-specific implementation of bus_dma(9) in order to map and share grants with domains.

Final deliverable: Have the interface fully integrated with all the frontends and backends present in FreeBSD.

Milestones

Starts

Ends

Description

Comments

Status

May 15

May 30

Preparation work, community bonding, create wiki page, setup accounts and working environment, read FreeBSD coding guidelines, checked-out the FreeBSD code

This activity took time because I couldn’t setup suitable coding environment for myself. Since I did not download the GUI for FreeBSD, so I maintained two working environments, one for coding which is Ubuntu (preferred) and another for testing which is FreeBSD so, had to run OS on virtual machine. Therefore, it took time to setup right environment for myself suiting my needs.

(./)

May 30

June 20

FreeBSD code walkthrough (bus_dma, grant table), OpenBSD code walkthrough (xen.c), understanding virtualization, Xen, device drivers, debugging kernel, understanding code APIs - https://www.freebsd.org/cgi/man.cgi?query=busdma&sektion=9

During this period, the biggest challenge was to understand the technical terms used in the literature [1], [2] and their relevance implementation wise [1]. For example, [1] directly talks about functions and data structures, which I had no idea. Before beginning the project, I tried to enhance my understanding on key concepts like virtualization [6], hypervisor [6], Xen [6], Xenstore [6]. After the above exercise, I could comprehend the terms used in the codebase and be able to relate the code’s functionality through comments. The [1] helped me in understanding the sys/dev/pv/xen.c in OpenBSD. When I looked at FreeBSD code the issue was what files to look at. I started with few random files with xen and bus_dma in its name or code. [5] and xen.c of OpenBSD helped me structuring my understanding of code flow, some trivial APIs and data structures common across BSD OS. After confirming from my mentor, now, I had identified the area I need to work on. I mapped the common functions like _bus_dmamap_*, referred [1] and started discovering the code. For grant table, I used the predefined FreeBSD APIs and didn’t create my own as in OpenBSD. Note: Numbers in the bracket represents the order number of the like in the useful link section below.

(./)

June 20

June 26

First prototype of integration of grant table into bus_dma(9) and seek improvements to make it a part of the first deliverable

In this period, I also designed several prototypes and tried to solve the problem in numerous ways. I crashed Kernel several times, even stuck in between because the idea required more work or the required information is missing and need to store it in some data structure. I debugged [7] the code only when it crashed to find its cause. By this time, I knew every small thing about the code I was working on and what was talked in [1] and [2].

(./)

June 26

June 30

First deliverable to be delivered

I have delivered a small implementation of the project. My changes are limited to the busdma_bounce.c. I am able to store grant table references, assign them to the DMA maps, loading them and then destroying them. My implementation is similar to what it is done in OpenBSD but I have more ideas around implementation. I would be planning to extend it to other areas of code using bus_dma(9).

{o}

June 30

July 20

Resume coding and integrate the functionality to all frontends and backends

{o}

July 20

July 24

Stress testing, bugs removal, further enhancements, and code review

{o}

July 24

July 28

Second deliverable to be delivered

{o}

July 28

August 21

Documentation, code review, comments incorporation, performance improvements, testing, and bugs removal

{o}

The Code

https://svnweb.freebsd.org/socsvn/soc2017/vaibhav/

Xen bus_dma grant table handler

https://www.openbsd.org/papers/asiabsdcon2016-xen-paper.pdf

http://www.openbsd.org/papers/asiabsdcon2016-xen-slides.pdf

http://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/src/sys/dev/pv/xen.c?content-type=text/plain

https://xenbits.xen.org/docs/unstable/misc/grant-tables.txt

The above papers explained everything on integration of grant table in OpenBSD. The papers discussed data structures and APIs in detail which was helpful as some of the APIs could be reused. It also talked about correct way of handling the data (also how it is supposed to be done in FreeBSD) like maintaining a cookie, saving grant table references and updating addresses on ds_addr. Which again helped me in understanding the grant table implementation in OpenBSD.

Device Drivers

http://www.unixag-kl.fh-kl.de/~jkunz/projekte/NetBSD-driver_writing-1.0.1e.pdf.gz

The above paper talked about writing device drivers. Though, it talked about NetBSD, it was helpful in debugging the code as some of the file structures and data structures are like FreeBSD and OpenBSD. For example, how device is attached and the use of _SOFTC structure for future reference is well explained in the paper and could be help in debugging. Also, since very less literature is available elsewhere, it is a good thing to start with.

Xen

https://www.netbsd.org/gallery/presentations/jym/eurobsdcon2011/BSDs_and_Xen.pdf

Really helpful in getting an overview on Xen and virtualization. It also helped in understanding Xenstore, device drivers and domains which are widely used in the current project.

Kernel Debugging

https://people.freebsd.org/~syrinx/presentations/FreeBSD-debugging-draft.pdf

SummerOfCode2017/Xenbus_dmaGrantHandler (last edited 2018-03-13T08:09:09+0000 by MarkLinimon)