Virtual Memory Compression


Virtual memory compression is a memory management technique implemented by multiple operating systems ranging from Windows to Darwin. Depending on the implementation, it can either reduce swapping to a physical storage medium or completely replace it. In the vast majority of systems, compression and decompression are generally known to be faster than disk IO. It has also been reported to offer energy savings.

Prior art

The target audience for the project

Project Plan

Primary goal: “zram for FreeBSD”

The primary goal is to implement a compressed in-memory disk that can be utilized as a swap storage. The current plan is to integrate it to existing md(4) and the corresponding userland configuration tool mdconfig(8). Lz4 will be the default algorithm due to its good speed/compression ratio, while zstd with a configurable compression level is also available as an option. I will also investigate an idea that I’ve never seen implemented before: a compressed memory disk that by default uses a very fast compression algorithm (lz4) but changes to a more compact but slower one (zstd with an aggressive compression level) when under extreme memory pressure. I call this adaptive compression. The idea is to use a fast algorithm untill x % of RAM is utilized and to actively start recompressing existing data using a more compact algorithm when x + y % of RAM is utilized.

Secondary goal: “zswap for FreeBSD”

A secondary goal (only once the first one is considered ready and there's still time) is to implement a compressed cache that sits in front of a physical swap device. I am unsure how far I am able to go with this, since it involves some fairly complex parts, such as finding compressed page pairs (or groups) that fit nicely to a single physical block on the disk and are likely to be fetched at the same time.


Development and testing

Development and testing will happen in a 13.0-CURRENT system that I already have inside a bhyve virtual machine. As the project progresses, testing will gradually move to physical hardware that I have at disposal. That includes two ThinkPads (X230 and T480) and two Raspberry Pis (2 and 3). Testing will consist of real-world applications, such as running Poudriere builds, databases, and serving web content.

The Code

SummerOfCode2019Projects/VirtualMemoryCompression (last edited 2019-06-23T20:37:27+0000 by PaavoEinariKaipila)