• DevSummit
• 201405
• Ports

FreeBSD Developer Summit: Ports

Thursday May 15, 09:00-12:00

Overview

The FreeBSD project has provided pre-built ready-to-install binary packages for many years on a best-effort basis. While these packages do work in a large number of cases, there are too many inconsistencies and failure combinations, from the unpredictable update frequency to dependency handling across upgrades, for them to be used on a wider scale. This session centers on a round-table brainstorm that begins with a summary of the tremendous progress made in the last 12 months, and closes with a discussion of the roadmap on how to improve binary package creation, distribution, installation and upgrading.

If you would like to participate, contact the working group chairs below and CC devsummit@. You will be then added to this page. Please include a list of things you want to talk about or the areas you are interested in. This helps us in planning the session and to bring people together with common interests.

It is possible to bring in people who cannot attend in person via video conference or chat tools. Notes during the session will be published later on for the whole community to see what we discussed.

Goals

In particular, we would like to cover the following topics. This is not an exhaustive list and if you feel there is something missing that you want to talk about, contact one of the session chairs and we will include your topic here. Note that the numbering of the topics does not represent an ordering or importance indication of any kind, but rather a reference to the second table with the "topic of interest" column.

Topics

Final agenda

P Note: General presentations about work you have done that does not require further discussions should be submitted for the FreeBSD Developer Summit track at BSDCan (see the general developer summit page).

Attending

In order to attend you need register for the developer summit as well as by email for the session and be confirmed by the working group organizer. Follow the guidelines described on the main page or what you received by email. For questions or if in doubt ask the session chairs.

Notes

Results

bapt:

• === Status update on package building ===

Building all packages with poudriere on large machines. 16h to build full tree. each Wed at 01:00UTC snapshot tree and do incremental build (full build if there is a security problem). All supported releases + 2x stable branches (9, 10) Quarterly branches for 9.x-RELEASE and 10.x-RELEASE

Catalogue is signed on signing machine. 4 pkgrepo mirrors (2 US, 1 UK, 1 Russia). Most people seem happy -- can even run head with binary packages.

coordination with src -- abi breakage.

Supported arch -- i386, amd64. To add arm and one MIPs variant.

Don't mirror distfiles any more

Building a pkg of the ports tree? No progress, but it's a good idea. pkg-1.3 will have repo metadata including the revision of the prots tree.

PKGNG Status

We should drop the NG bit (bapt: "it's not me. I'm not writing pkgng for years now")

New solver in pkg-1.3 -- 'pkg upgrade' understands it needs to delete an old package and replace it with a new one, so no more need for 'pkg set' or portmaster chicanery. All due to new solver. cebka's talk on Saturday.

Added bits to cross-build ports. 'make XBUILD=armXX' some bits are in the ports tree already, rest is waiting on pkg-1.3

Security: every dangerous part of pkg upgrade/install/etc. is sandboxed

pkg can upgrade itself in jails, by forking itself

pkg_tools decomissioning

Errata notice has added default pkg.conf and keys to all supported branches, delays added when building from ports

Aim is to deprecate pkg_tools in 6 months. Usage of old style pkg_tools generates warning messsage

1st Sept for death of pkg_tools. (Same date as staging deadline)

Quarterly branches

Evaluation based on 3 months support. do we have enough manpower etc.

Snapshot of ports tree: only allow seurity, build time fixes and upgrades to pkg

MFH tag in commit messages, auth requests approval from portmgr. MFH tool in scripts to make it all easier.

So far has worked pretty well. Only a couple of problems due to insufficient testing.

commits to Q-branch should just merge the security fix: other changes

auto merge of updates to vuxml ? Or just remove vuxml from Q-branch; just have one per repository.

JA: Release notes for the quarterly branch? What is/isn't available etc.

We do extra testing etc. before branches. Use UPDATING file in pkg metadata? Not really needed -- pkg -1.3 should obviate the need for UPDATING.

per-package News planned for pkg-1.4.

Machine readable info.

pkg messages: differentiate between what you need for installing from ports and what you need with binary packages

Context sensitive messsages

Ports infrastructure

Various changes: staging, LIB_DEPENDS, users=foo

Huge migration affecting all ports. Expect committers to have port buildable without root, to pass QA tests etc. Still ongoing.

Needed to speed up. Maintainer lockout was slowing things down too much. New policy -- easy fixes and stuff related to infrastructure changes, any committer can just do without reference to the maintainer. "Just do it" Took time for people to get used to this. Hasn't been much complaining from maintainers.

About 3800 still to be staged. We did stage twice as many ports in one year than pkgsrc managed. Success!

Committters should have DEVELOPER=yes. Enables warnings for every infrastructure change in tree, and how to convert. Only special case is staging, which refers you to the wiki.

erwin:

MASTER_SITE_BACKUP

Change default to http: --- send announcement, and do it (maybe not in that order)

distfiles on the new pkg distribution network? yes, no, ?

ben kaduk:

MIT's Athena Environment

Athena is not yet FreeBSD based but would like it to be

started in 1983.

Long history of different hardware platforms and OS. Currently debian/ubuntu.

100s of machines, including ~400 publicly accessible machines (can get root password for the asking)

Software used to be monolithing; now split up into about 150 different packages. 4 popular metapackages What does a debian package look like? debian adds a subdir to the upstream source tarball, inside out from the ports way.

Diversions -- heavily used feature. replace another package's file with my own version.

https://github.com/mit-athena/

Is there a VM? Don't think there's one currently.

Diversions and pkg(8)? Possible, but not implemented. All of what was described is doable using the ports tree now.

Athena on FreeBSD depends on AFS, which needs work.

des:

Changes to RUN_DEPENDS

Now we have staging, split run dependencies into two different lists. What is needed at install time and what is needed at run time. Problem is we can't change meaning of RUN_DEPENDS overnight.

eg. anything that installs a DTD needs to have libxml installed so it can register DTD in catalogue.

Some dependencies have to be present for 'make stage' others only needed for 'make install'

STAGE_DEPENDS vs INSTALL_DEPENDS

bapt: STAGE_DEPENDS should just be part of BUILD_DEPENDS

More intelligent dependency tracking coming with pkg-1.3 and death of pkg_tools Will break portupgrade / portmaster

'pkg add' is a problem: without a repo, we don't have all the necessary meta-data. Merge of pkg add and pkg install was announced and implemented for 1.3, but reverted.

Certificate Transparency

Idea proposed to FreeBSD Foundation. Idea only, invitation to discussion, no code yet.

Initial idea: Google's Certificate Transparency. Detect doctored certificates, certificates illegitimately issued to people who don't own domain.

Works by logging certificate issuance to log server. Can be checked by public. Domain owners can search for certs issued for their own domains.

Extend this idea to binary package management.

Model is applicable to individually signed packages. Can verify packages issued that claim to be signed by FreeBSD are authentic.

Can we do this with existing infrastructure?

sson:

Qemu user mode

Working with Qemu 2.0.0

static + dynamic binaries supported

Handful of system calls not supported (will never be)

Supported arches: MIPS{32,64} ARMv6 PPC easy to add.

32bit on 64bit host now working, some lingering issues

Future:

• Fix lingering issues for 32 on 64 Add PPC and/or Aarch64 Pushing code upstream Improved debugger support for cross development

Kernel module allows eg. suing native compiler for cross building. Good performance improvement.

Kernel module already committed to head. will be in 11 and MFC'd to head

Can I use this to build Java for power pc without a pre-existing javavm?

yes.

also could use jam java interpreter. Close to being able to build java.

• ~~~~ break ~~~~

bapt:

Pre/Post install scripts

Currently scripts are not reviewed and can easily be buggy. Idea is to provide library of services (casper like)for common actions and reduce ability of maintainer to perform arbitrary actions. Also, services can be sandboxed easily.

Needs someone to work on this area in pkgng. Will be big security and reliability improvement.

Performance improvement: not calling the same script over and over for a sequence of packages, but run it once at the end. Is related, and should be done via services.

Cross-install of packages: services can run host-native binaries to modify guest arch content.

Cross building

Should be able to cross build ~80% of ports simply by 'make package'. xdev port critical to provide sysroot.

Major blockers:

1./ perl -- cross building "creative". Need ssh account on target machine. Not nice. Provide preseeded configure cache files for perl.

1./ python -- started by using autotools (good) but needs it's own python binary. Doesn't build native as well as target binary. Can we have the python ports cleaned up, as they are currently v. ugly?

Build everything twice -- native and target. pkg(8) has grown some relocation abilities

pkg.conf is a problem. We're using pkg.config alternative.

Toolchain is a problem. What you get (gcc vs clang, etc, etc,) depends on arch and OS version. Change ports to look for capabilities of toolchain (eg. c++11 compiler)

'make XDEV' doesn't allow bare sysroot without toolchain. Want to separate that, so can provide crossbuild tool chain provieded around features. Hoping to have this for official package builds by next BSDCon.

bdrewery:

PortsCI

Project for managing jobs in the cluster -- Exp-runs, patch testing against the ports tree.

Currently using cron and we often fail to build packages for some environment

Types of job: Exp-run, Reference, QAT, Port, Package -- idea is to be able to queue any of these and have simple UI for managing cluster servers.

code exist as design document: not much implmented yet.

We're building the entire tree every day to see what's broken. Similarly QAT.

Make PRs with patches into something more like pull requests. Can automate testing builds of patches.

Instead of asking portmgr, any committer can queue an Exp-run or port test

How hard would it be to have extra tests for test builds? Can use the QA framework to add arbitrary test... Hooks for adding you own script into stage? Should be do-able.

swills:

Jenkins PortsCI

Name clash -- FreeBSD_ports_ci

Polls SVN for committs. HAs 8 bhyve VMs difference combinations of arch, OS. Slow but it runs.

Tends to batch up several changes together. LEarning experience in bhyve... Tests right now use default poudriere tests, developer mode not turned on yet.

Should this replace tinderbox? It probably could with some changes to the way fail-mails work. Need to break it up so you do one build per commit?

Sean Bruno: How much hardware needed? swills: How much do you have?

Lots of copies of ports tree -- could use NFS or nullfs ?

We have 3 or 4 package build clusters now. Can we unify them? Can we replace QAT? Reuse QAT hardware...

Jenkins as frontend to Bryan's build system. Integration with phabricator.

Split up Jenkins -- individual "builds" for each port. Ports tree change: install dependencies from 'pkg install'. hook it into poudriere too: package seeding from repos.

erwin:

freebsd-version(1)

ports tree needs to find the version of the userland currently running. Told we "can't use freebsd-version that way" Is designed so that we can apply vuxml entries for the base system. uname -r becomes incorrect if a security fix doesn't touch the kernel

How to check the userland version from the ports tree? freebsd-version(1) isn't for that. Only works with freebsd-update.

Eg. OpenJDK marked as broken generally, but only actually on certain versions.

David Chisnall: Don't use OS Version at all? It's an ugly hack...

Currently version number is only practical solution. Actually need a file of feature flags to show what has been fixed where.

Who is responsible for maintaining feature flags file? Not just secteam -- bottleneck.

mat:

Licenses

Right now we have generic license tag in port, but should really include raw license text, as dates and copyright holder names change.

Framework should pull the licence file from the port and install automatically.

bapt:

Packaging Base

Target for FreeBSD 11 -- have the base system totally as series of pkgs. Can do make release-pkg generates all the packages as a normal user.

pkg(8) needs some extra capabilities: handle chflags

what granularity for base packages? Currently tends to be broad package with full toolchain etc. Should be finer grained. Toolchain, binutils, headers (required by some languages), one package per option in the build system (eg. src.options.conf)

some options like MK_NIS change how certain programs are linked not just what is built. How ambitious do we want to be... (Approx 10 out of 100 options are problematic like this) Provide/require system and different flavours of base system.

bsdimp: "our buildsystem is not a paragon of configurability, but a bunch of hacks that annoyed people the most."

Where do we get metadata for building this? How do we version that?

Current build system is not a paragon.

Base will be a meta-package that depends on split up different packages

Will give file tracking automatically.

Eliminates the obsolete files problems.

How will upgrades work? We can delete the entirety of base and reinstall.

Replacing all the libraries in multiuser -- expect things to "go wonky"

---

CategoryHistorical