Linuxulator (Linux Emulation)
Linux binary compatibility, commonly referred to as Linuxulator, is a mechanism to run unmodified Linux binaries under FreeBSD. It does not involve virtual machines or emulation; instead, it provides the binaries with kernel interfaces identical to those provided by a real Linux kernel. Technically it's similar to the way 32-bit FreeBSD binaries run on top of 64-bit FreeBSD kernel.
There are two main ways to use Linuxulator. First, documented in the FreeBSD Handbook, uses Linux applications provided as FreeBSD ports and packages, to be installed using pkg(8). Second, described at LinuxJails, provides a full Debian/Ubuntu userspace. The former is more user-friendly, but provides a limited number of available applications. The latter method is more involved, but is handy when doing development work. Both can be used at the same time.
Status
We currently claim compatibility with Linux 3.2.0 in 12-STABLE and 4.4.0 in 14-CURRENT.
The list of currently tested applications can be found at LinuxApps. For Steam, we suggest using linux-steam-utils package. The status of tested games can be found here. Instructions on how to watch Netflix can be found in this forum post.
Development
The goal is to implement as many Linux syscalls as possible, except for ones that are obsolete or only used for management purposes.
The canonical list of unimplemented syscalls can be found at sys/compat/linux/linux_dummy.c. Those marked UNIMPLEMENTED() are not supposed to be ever supported, usually because they have been obsoleted upstream. A somewhat stale list can be found at LinuxulatorMissing.
The Linux syscalls(2) man page lists all syscalls including references to the Linux kernel version each syscall first appeared. Note that in Linux, there are differences in syscalls supported by each architecture, there's a website tracking this.
Testing
Linux Test Project: README and INSTALL are straightforward, don't forget to mount devfs, linprocfs and linsysfs to the linux-chroot where you compile/test
GNU libc nptl regression test: not as easy as LTP, but doable; precompiled stuff available
mmap fingerprinter: little tool by Marcin Cieslak, extended by Jung-uk Kim to determine the fingerprint of the mmap() call. Could be used to extend the FreeBSD regression tests too. Note: Marcin confirmed that mmap_test.c is public domain.
Darren Hart's futex test suite: this suite will run various futex related functionality, performance and stress tests.
Trinity: a Linux system call fuzz tester: a package to call syscalls at random, with random arguments, to find eventual vulnerabilities and bugs.
strace(1) test suite - make check to run, focused on ptrace(2)
- The ¨ballista¨ test cases of the Linux Test Project should be run when we are finished with fixing bugs.
- The Open POSIX test suite (comes with the LPI); we need a reference output from Linux to be able to compare this.
- Some of the LTP test cases that are not run by default.
- The glibc regression tests.
Applications listed at LinuxApps and WantedLinuxApps
LTP
Very easy to set up:
- Install devel/linux-ltp port, or linux-c7-ltp package
- cd into /compat/linux/opt/ltp and initiate a full LTP test run with "./runltp -p -l/var/tmp/results-log -o/var/tmp/results-output -C/var/tmp/results-failed -d/tmp"
Keep an eye on /compat/linux/tmp between tests and clean it up. You should also run "ipcs" and remove some leftover (compare before and after each run).
Current LTP results can be found here. Results for some of the older releases can be found here.
Debugging
FreeBSD's truss(1) and ktrace(1) can trace Linux binaries and decode the syscall names, but they don't support decoding Linux arguments into human-readable form. Linux strace(1) does, though - it can be installed from the linux-c7-strace package, or by apt install strace in Ubuntu chroot.
Linux core dumps are supported starting with 14.0-CURRENT. Use Linux gdb to open them; you might want to consult Ubuntu documentation on debug symbol packages.
Roadmap
- Implement PI Futexes
Implement linprocfs /proc/self/task/$TID/
Check setpriority/getpriority - seems that Linux allows set/get prio per thread
Use uma instead of M_LINUX
Check __WALL, __WCLONE in wait syscalls since Linuxulator's threading model has changed. The __WALL since Linux 4.7 automatically implied if the child process is being ptraced
- add the compat.linux.strict_emu sysctl (the name is not set in stone) and use it to enable/disable some functionality we allow in FreeBSD but are not allowed to do in Linux, e.g., allow to open more than one million files, or allow to read() directories
- add detection of unhandled flags to syscalls (e.g. add each detected flag to a variable and compare this variable to the input, if there's a difference print the difference)
- print a message if a known but unhandled flag (e.g. LINUX_O_NOATIME for open()) is used (maybe protected by bootverbose or debug; depending upon the severity and frequency of the use of this flag)
- change the non P1003_1B_MQUEUE part of linux_mq_*() into its own module (the changes for linux aio in p4@114975 can serve as an example)
- add in the normal device driver entries into the device_handler stuff that are common to all Linux machines
- have a look at the linux dev_t issue (HEADSUP message from phk to arch@ in March of 2005)
- cleanup/review of existing code
- Verify that each copyin()/copyout() is handled correctly. There are at least 5 copyout()s where a quick grep revealed ignorance of the return value.
there are places where a function A calls function B, then calls copyin() on the result of function B, and then checks the return value of function B for errors -> BOOM on error when calling copyin()
- safety-net like in rt_sigpending (but no "new" return values, KASSERT if possible, rt_sigpending needs to be reviewed regarding this)
- CLK_TCK still valid? / XXX comment: both in linux_misc.c
- add doxygen comments to the code
- follow r237433/aea810386d8e and use kernel timekeeping data in the Linuxulator's vdso
Bugs
A note to users: Feel free to read the following list of things to fix, but do not make conclusions out of it. The linux compatibility environment runs just fine. There are some broken edge cases that don't affect daily use. The following list is only meant for developers. Just because something is marked as a bug, it doesn't mean it doesn't work. It may be the case that some obscure error condition does not return the expected error value, or that a seldomly used feature is not implemented.
There's an umbrella ticket for tracking Linuxulator-related PRs: 247219
Linux apps require at least drm-devel-kmod-5.4.62.g20201109; otherwise they segfault. Use export LIBGL_DRI3_DISABLE=1 as a workaround.
the linux-ldd doesn't work when not run with the linux compat shell; if it is run within the linux compat shell, it does print nothing for dynamic executables (it correctly works for libs); because it is a shell script which calls ld-linux.so.2 it may be caused by something which is tested in the LTP... or not. It works inside a Linux jail/chroot, though.
- linux_lseek() silently truncates the offset modulo 2^32, lseek() in 2.6.x has error handling for this (EOVERFLOW). Unlike the FreeBSD lseek() (open(), ...) the Linux one can't handle 64bit file sizes.