• DevSummit
• 201309
• Ports

FreeBSD Developer Summit: Ports Working Group

We would like to cover the following topics. This is not an exhaustive list and if you feel there is something missing that you want to talk about, contact the session chair and your topic will be included here.

Agenda

Please send talks, brain storm topics, or other proposals to the session leader, ErwinLansing.

Discussion Topics

• Package set policy
• Security branch
• pkg_* removal

Results

(Add a list or attach slides detailing the achieved results here.)

Announcement: We now have INDEX-10 from portsnap

PCBSD Package CDN - Allan Jude

Packages sets would take mirrors weeks to upload.

pkgng: the master needs to be atomic. If the pkg checksum doesn't match will fall back to alternative

New high bandwidth master and using CDN

rsync --delay-update in two passes (files, then sym-links)

zfs replication. using zxfer.

zfs-snapshot-mgmt -- avoid having to resync the entire dataset even if offline for a few days. zxfer -- hasn't been touched since FreeBSD 8.2. Fails on later ZFS. Forked on github with many fixes, error checking, extra options To integrate: fdpv for progress bar animation

Buildfarm only has 5Mbps -- is uploading constantly. Servers sync last 15mins of traffic within seconds. Staggered updates across servers.

Automatic mirror selection. GSLB from ScaleEngine. LB checks mirrors for freshness on images and drops them from the real hosts.

Conclusion: ZFS make replication easier and allows atomicity.

PCBSD servers pull about 36TiB per month -- iso.cdn about 1 TiB/day, pkg.cdn ~85 GiB/day of that.

bapt: want to look at this for FreeBSD pkg repo and steal any applicable ideas.

Need to get pkg signing working -- need signing server.

Compare portsnap is about 5TiB per month

SSP - Jeremy Le Hen

Reprise of what was in the security session.

SSP mitigation fro stack based overflows. First appeared in FreeBSD 8, only in base system. Finally is here for ports.

The first 99% is easy. Easiest path simply to patch CFLAGS.

Needs a lot of polishing and testing before it can be default.

Showstoppers:

on i386 gcc special symbols addressed in specific way and needs to link binary in special way. Some ports do not honour LDFLAGS. Solution: turned libc.so into an ldscript: Propose libssp_nonshared for linking in 10.x. Now enabled by default for i386/amd64 in 10.x amd64 usable on 9.x by adding WITH_SSP_PORT knob (doesn't do anything on i386). Idea is to activate by default for official packages but not yet.

New archs need work.

New knob for port maintainers: SSP_UNSAFE -- for ports that do weird things or for kernel modules.

Staging - bapt

Every pkg building system uses staging *except* FreeBSD. staging means install somewhere else rather than braking your system. Also create package without installing. Can create packages without being root, except for some packages doing weird things, for which a new NEED_ROOT flag has been added.

Plan to create a new fakeroot thing using ptrace, but that is v. architecture dependent. Overwrites BINOWN etc. as current user to allow package-as-a-user

Can improve package building system so we don't need to wast time trying to figure out of pkg building system is doing stuff outside staging area.

Won't have packages with broken plist -- pkg register reads the plist to assemble the pkg. We also test installing as a pkg every time, which didn't happen before. (Many broken package scripts were broken but never tested)

Most of post-install targets can be ripped out of ports as they'll be done by the pkg.

Standard post-installtion tasks lke mkinfo no longer need to be run. Save time on pkg build systems, allows more EXP runs.

Added NO_STAGE to all ports so default for new ports now is to use staging. Don't want to spend years switching like pkgsrc. Plan is something like 6 months. Many packages work already, or require fairly simple changes.

Proposal: in 6 months time, mark as deprecated all packages still marked as NO_STAGE. Allow 3 months deprecation time. Will highlight what is still maintained in the ports tree.

Is this a bit aggressive? Yes, but having to maintain everything in dual is painful. 6 months is probably optimistic, may need to extend. Long deprecation period should allow maintainers to produce updates. Ultimate target is for stage project to complete in 1 year.

Have ripped out the MAN1 macros. Were needed for compress man -- new implementation doesn't need this. Separate Makefile build code from PLIST style stuff. Also sub-packages: will have several plists for one port (MAN1... how do you know which sub package this belongs to).

redports should already have staging spport, but may need to wait for updates on Monday.

Monthly mail to maintainers with ports still with NO_STAGE, wiki page

AUTO_PLIST as an option? Can do now (as soon bapt gets connectivity...)

EXP runs with FORCE_STAGE to find what is already stage ready. Usually the port works, but post-install needs work.

stage is the last major modification to ports we can do without breaking support for pkg_install.

STAGE give opportunity to add QA testing heuristics which analyze the stage dir and suggest eg. adding dependencies

Create debug sub-packages from stage dir (ie. compile everything with debug symbols) Conversely warn about installing binaries with debug flags when not intended.

RUNPATH in binaries -- know where to find shlibs

developer sub-package: should this be automatic or manual setup of sub-packages? cf. PORTDOCS -- start by doing things manually, then provide more automation once it has matured.

Package Sets -- Erwin

Decouple src from package releases.

Done:

convert ports from CVS to SVN new distribution infrastructure new build cluster (not yet for old packages) pkgng

Package/Ports releases:

Wanted?

How frequently? biweekly + monthly

Do we want a security branch?

portssecteam@ mailing list -- but no formal ports sec team, no charter etc. Needs to be formalised. Should they answer to portmgr or SO? SO technically could just appoint a ports sec team (SO charter is surprisingly wide...)

We rarely get advance notice of issues in ports. So portssecteam does not need to be a small closed private team. Needs to expedite handling vulns that are already out in public.

This is secteam responsibility, but t's already hard enough to stay on top of kernel+userland. secteam relies on knowing who has expertise for help on specific areas of kernel / userland. Doesn't apply with orts eg. gnome@ is an open mailing list and not a good place to discuss vulns. ports sec teams should have this knowledge.

ports sec team has authority to override maintainer where necessary for security reasons. It's about quick response and authority.

Need to find out who is on the team right now... and are still available to work on stuff. The list isn't dead but ports sec team is not used properly at the moment.

SO should answer to core. Ports Sec Team should answer to SO. Unfinished Charter is on website, needs review and ratification. How can people join the ports security team?

ports security team created by fission of secteam by Jacques around 2005.

Member of PST at EuroBSDCon should meet up and formalize things.

Package Set Policy - bapt

Just an idea now: want to make it real starting next month.

Want to create package sets quarterly, and maintain them properly.

• Q1 Q2 Q3 Q4 Weekly LatestQ

branches in SVN. Qx branches to be maintained for 1 year. Only portmgr and PST can commit to Qx branches.

Weekly is just a symlink to the latest package set pushed from build cluster each week. Security fixes only.

packages which have security problems we can fix -- package will be removed from site and port will be marked restricted. We need a mechanism to tell users they need to update to a new branch where no security update is available.

pkg news to be displayed after pkg update. Also add to Gnome etc. front-ends (Ref PC-BSD people for desktop support)

To start on 1st Nov (Q4) and next year Q4 will be in Oct. Need working Ports Sec Team, and working pkg build cluster. Can start with currently available pkg-test cluster.

Space usage on cluster with 5 package sets online at once. Will allow errata updates as well as security. Will have weekly builds for testing, so should allow fixing errors before the big Qx releases.

pkg-fallout: New hardware allows much more rigorous QA testing. BD has finished setting up machines so expect to receive way more mails real soon.

port maintainers' responsibility to contact portmgr or ports sec team if they have a security fix to push to Qx branches. (portmgr only for errata)

Test update from Qx -> Qx+1, Qx -> Qx+2, Qx+1 -> Qx+2 (all combinations of supported branches)

Huge need for base system people to inform portmgr of anything that will break ABI in enough time to build package sets for HEAD and test for regressions in the pkg build cluster. Really important to know about breakages that will prevent running a newer jail for HEAD.

Deadline for having working ports sec team?

pkg_install removal - bapt

We need to clean up pkg having dirty code to support pkg_install. Supporting two versions is becoming harder and harder. Can get rid of much dirty code in bsd.*.mk

Once we have two or three Qx release sets working for users then we can say we don't need to support pkg_install any more. portmaster works the same way with pkgng. Binary packages can't be updated with pkg_install: so delete all and rebuild using pkgng.

This is a big change late in the line of eg. 8.4-RELEASE.

Need update to the pkg.txz package on pkg.freebsd.org so that it installs a more recent version

Conversion is just

• /usr/sbin/pkg pkg2ng

and then continue maintaining you system as previously if you don't want to switch to binary package management.

We're blocked by pkg_install.

We need a really strong PR campaign that this is coming. 6 months warning required.

Bootstrapping pkgng when nothing else is installed still tells you to run pkg2ng Could just run a post install script.

List of *shiny* new features that can be implmented in a post-pkg_install world to be included in the PR campaign.

Could the ports tree support divergent feature sets for pkgng vs. pkg_install?

• Is very hard to maintain. Involves lots of duplication.

We don't have experience of a release which is pkgng-ified. (we should do by May when 10.?)

• Many large companies have already adopted pkgng. Config management tools like puppet pretty much imply pkgng.

Better to slip the pkg_install cut-off date if we discover a problem than delay pre-emptively.

Do 3rd party tools support pkgng yet: puppet, chef, ansible, salt, webmin, packagekit: support is either already in the ports or ready to add to ports.

Plans to fix huge securty hole which is pkgng bootstrapping? This will be fixed by checking signatures on pkgs. and repo catalogues. (Different certificates -- so you could bootstrap from an old CDRom, needs a cert less likely to be revoked)

Automatic message warning when using pkg_install "84 days left until pkg_install goes away..."

8.3 doesn't provide pkg bootstrap. Need to wait until April when 8.3-RELEASE expires.

8.3-RELEASE will be supported for it's lifetime.

8.4-RELEASE users will be told they have switch.

Open tasks

Package names: we're doing it badly for packages. Renaming packages based on options based on options is really bad.

If we want to be able to provide 3 versions of MySQL or php or whatever: different versions need different PKGNAMESUFFIX ports need distinct portnames. We should not need LATEST_LINK.

We should be able to get rid of all the 'portmaster -o ...' entries in UPDATING.

(Don't use PREFIX -- that's used for different language versions eg p5- py37-)

Distinct packages which happen to have the same name -- needs fixing. Change in policy, as this used to be OK if the ports were in different categories.

Most of the time the pkg name should reflect the directory name. Need flexibility to override in some cases though.

(This is the same issue as 'pkg install subversion' attempting to install 3 different versions of SVN).

Do we really need four different versions of perl?

Using metapackages eg. lang/python -- installs a mostly empty package which depends on the default version of python.

options -- should be mostly converted into sub packages. We shouldn't be aiming to support all possible different combinations of compilation options. 3 sorts of options.

• 1 -- adds extra files (just use a sub package) 2 -- depend on an alternative shlib (don't offer this
  • in general, or provide flavours where sensible)
  3 -- enable or disable different compiled in modules (convert
  • to using run-time modules)

... perhaps also

• 4 -- turn on or off compilation options or add an optional patch
  • (use separate ports -- this is v. similar to type 2)

We can't get rid of options entirely eg. bind

We want to get rid of the origin as the package identifier and use the pkgname instead.

Package names vary between linux distros. Same thing with FreeBSD. V. hard to coordinate even between just *BSD.

DFly is switching to ports. Will other BSDs follow suit?

Convert build options into run-time modules (ref: nginx)

CategoryHistorical