Linux Emulation on FreeBSD - 2014/07/09 BSDCam DevSummit session

Session Chair : Vassilis Laganakos. Attendees list : (incomplete, room was busy :) )

Notes compiled by Vassilis and Pricetx.

Executive summary

We need to move to a more recent linux_base port (such as f{19,20} and centos{6,7}), but there are commercial apps that we use that require f10 to work and they won't be updated any time soon (if ever) to work with newer linux_base ports.

We should investigate whether we can turn the hard-coded f10 linux_base in the ports to something modular that will allow us to use any of the available linux_base ports. (USE_LINUX=f{10,19,20}/c{6,7}

Regarding linux64 lemul, dchagin has done a lot of work on it. We should ask for formal review and security review requests and lots of testing; we think that then it could make it to go in FreeBSD 11 (perhaps :) ).

A list of Linux "most wanted apps" that we want them to work on top of linux and linux64 is available at WantedLinuxApps.

More raw data

We need 64bit emulation since for some apps (eg. video) require 64bit to operate.

Dmitri has worked on 64-bit emulation on linux, having pushed to SVN in lemul branch. Has been tested a bit, “seems to be working fine”. A working port based on centos6 has been produced (Allan, Vassilis) and a f19 (Dmitry).

Xmj has been working on linux emulation for skype, now has all ports available for f10 and c6. The work is mostly by xmj. Work needs doing is code review. Vassilis asked if “Would it be possible to have a “superport” that fetched all of the linux ports for a supported distro?”.

Allan has made some changes to c6 64-bit to allow installation of linux version of oracle jdk7, including all dependencies. Crashing issues had occured but Allan is unsure whether this is due to the pmap issue.

According to xmj, you can combine C6_32 with f-10 userland.

Some argument as to why F19 is an option, as it has such a short support window, one argument is to instead do a C7 port.

Additionally, Allan brought up the question as to whether the 64-bit CentOS was already providing 32-bit support. Nobody had a definitive answer.

There was the question raised whether there is any proof that some Linux apps perform better on FreeBSD with Linux emulation than natively on Linux. Would be good to investigate whether this is the case or not. Based on one account from game performance comparison, the comparison was flawed as missing graphical features caused unfair testing.

On the question "What are we waiting for 64bit lemul to be pushed on CURRENT", the reply was more "testing and code ans security review". Explicit code review should be requested and should discuss this with the Foundation (check with Ed). Dmitry is working on the lemul branch, so he could tell us how far are we from linux64 prime time.

It was also stated that the 64-bit linux compat diff also introduce changes to the existing 32-bit compat layer, so we have to be careful not to break existing functionality.

If we push it to head, and make a to-do list on the wiki which people can tick off completed tasks on, that would promote contribution.

For linux64 to be functional it needs some 64bit linux_base and it should land in CURRENT along with one, e.g. the 64bit version of f{19,20} of c{6,7}, to make it is possible to test it.

Regarding the support/dependency of multiple linux_base distros in ports, Allan brought up an issue with the main Mk file in that every linux base has individual entries for every port, which will only become worse when potentially adding the 3 new linux bases (f19 both and c6 64-bit).

Linux should be able to coexist with Linux64. There was no final conclusion whether there should be one port that installs both 32 and 64 bit Linux libraries in /compat/linux - which would also take care of conflicts in installation paths (such as man) -, or if they should be separate ports. We should decide what to do with this; there was an inclination to the second option. There was then discussion as to whether there should be a “multiarch” port.

It was noted that the new syscalls for linux64 are extracted from 3.8, whereas the default sysctl is still 2.6. This is just a hard-coded test in the kernel module, and if you set the respective sysctl in the command line the app that complains should work ok.

Question was asked if any syscalls are missing. Answer is “don’t know, and do we know if all of the syscalls that claim to be implemented, work properly?”. It is agreed that more thorough testing is needed.

“Do we want native FreeBSD apps to be able to use linux syscalls?”. The case given was kqueue vs inotify (linux). It is agreed that kqueue is superior, but there were some advantages to inotify, and maybe something similar could be added to FreeBSD.

There was a discussion clarifying that we are not re-producing an entire linux environment. Rather, it is just providing the minimum possible environment to run linux binaries. It does have the one downside that it doesn’t make installing commercial linux software not available in the ports tree simple.

Centos 7 switched to 64bit only, and other Linux distributions may switch to coming out of the box with one supported architecture, so you will have to manually install base libraries from the other arch. Should we do try to present to the user/admin what he expects to see from other Linux distros?

Allan asked “Do we have a system in place to automatically convert an RPM into a port?”. We don’t. People that want to add arbitrary Linux apps, looks like for now they'll have to go through the procedure of creating a new port. Compared it to autoplist which is currently in use for the ports tree.

Probably we should not try to track installed app/packages using RPM, since mixing the two packages system is probably a recipe for headache :)

With regards to audio, since most apps on Linux side depend on ALSA we already handle that okay. Regarding video we had video calls through skype working before.

“Do we have openGL acceleration for 64-bit linux?”. Our mesa is 9.1 whereas C6 is 9.2. This means we’re missing some hardware context which is needed to get 3d acceleration right. - Google earth was tested and it was painfully slow. This applies to both 32-bit and 64-bit binaries.

There were concerns regarding the automated testing, especially when the app requires human interaction to validate that the application works. We need to make sure that we test everything properly!

In the past we had some security issues with linux emulation layer, so we need to make sure that we make a good review of linux64.

One interesting point raised was that the linuxulator could be used for performance testing just the kernel, as the userspace could remain identical.

Questions

Wish list

It was asked if there had been any progress with linux emulation on ARM. It appears that no progress has been made yet.

Some Actions:

Below are Pricetx's raw notes, need to incorporate them into the above.

==Linux Emulation==

Dimitri has worked on 64-bit emulation on linux, having pushed to SVN in ELMUL? Which has been tested, “seems to be working fine”. A working port based on centos6 has been produced.

Xmj has been working on linux emulation for skype, now has all ports available for f10 and c6. The work is mostly by xmj. Work needs doing is code review. Mk reviews for Allan work needed. Allan has made some changes to c6 64-bit to allow installation of linux version of oracle jdk7, including all dependencies. Crashing issues were had but Allan is unsure whether this is due to the pmap issue.

According to xmj, you can combine C6_32 with f-10 userland.

Apparently some legacy software will not run on the “newer” C6 linux base, so the F10 base will not be phased out immediately.

It is noted that the new syscalls are extracted from 3.8, whereas the default sysctl is still 2.6.

Question. Can we get the ports to be modular? (not dependent on a specific version of Linux compat). General consensus is this is better than hardcoded f10.

“Would it be possible to have a “superport” that fetched all of the linux bases?”. A new, interesting idea that sounds possible.

Allan brings up an issue with the main Mk file in that every linux base has individual entries for every port, which will only become worse when potentially adding the 3 new linux bases (f19 both and c6 64-bit).

Additionally, Allan brought up the question as to whether the 64-bit CentOS was already providing 32-bit support. Nobody had a definitive answer.

An issue raised by xmj is that you can’t have 32-bit and 64-bit versions of the same package installed at the same time, as you’ll have directory and man page conflicts.

ETA for Dimitri’s branch being committed to head? Couldn’t get hold of him, but hopefully completed for FreeBSD 11. It was stated that it would probably be best to get it pushed to head sooner rather than later as 11 (reason given for not already being there is lack of testing) is still far from release. It is also stated that the 64-bit linux compat diff also makes changes to the 32-bit c6 base, so we have to be careful not to break existing functionality.

If we push it to head, and make a to-do list on the wiki which people can tick off completed tasks on, that would promote contribution.

Question asked if any syscalls are missing. Answer is “don’t know, and do we know if all of the syscalls that claim to be implemented, work properly?”. It is agreed that more thorough testing is needed.

“Do we want native FreeBSD apps to be able to use linux syscalls?”. The case given was kqueue vs inotify (linux). It is agreed that kqueue is superior, but there were some advantages to inotify. “Perhaps we should just expand kqueue to add support for inotify functionality?”

Allan asked “Do we have a system in place to automatically convert an RPM into a port?”. We don’t. Compared it to autoplist which is currently in use for the ports tree.

Some argument as to why F19 is an option, as it has such a short support window, one argument is to instead do a C7 port. There seems to be agreement that the best long term solution is C7.

Noted that there should be a security review of the linuxulator, as there have been previous security vulnerabilities.

“Do we have openGL acceleration for 64-bit linux?”. Our mesa is 9.1 whereas C6 is 9.2. This means we’re missing some hardware context which is needed to get 3d acceleration right. Google earth was tested and it was painfully slow. This applies to both 32-bit and 64-bit binaries.

It was asked if we could either prove/disprove the long held rumour that the linux compat layer under FreeBSD was faster than native linux. Based on one account from game performance comparison, the comparison was flawed as missing graphical features caused unfair testing.

One interesting point raised was that the linuxulator could be used for performance testing just the kernel, as the userspace could remain identical.

It was discovered that the 64-bit CentOS does not ship with 32-bit libraries. There have been arguments for making the 32-bit compat libraries as a separate port, and alternatively as a port flag. It is agreed that a separate port would be more beneficial as it allows for a more seamless experience with pkgng. There was then discussion as to whether there should be a “multiarch” port. It was then discovered that CentOS 7 is 64-bit only, which simplifies matters.

There was a discussion clarifying that we are not re-producing an entire linux environment. Rather, it is just providing the minimum possible environment to run linux binaries. It does have the one downside that it doesn’t make installing commercial linux software not available in the ports tree simple.

It was asked if there had been any progress with linux emulation on ARM. It appears that no progress has been made yet.

new notes based on the discussion on whiteboard

Why?

Video app need to be 64-bit. (Commercial apps in general rather)

How?

List of desired applications is on the WantedLinuxApps page.

201407DevSummit/LinuxEmulation (last edited 2014-07-18 16:51:21 by GavinAtkinson)