PackageKit backend for pkgng
Student: MattWindsor (mattbw@FreeBSD.org)
Mentor: JulienLaffaye (jlaffaye@FreeBSD.org)
Project description
This and following sections are written in the future tense because they were written before the project was carried out.
I intend to develop, test and document a packagekit backend for pkgng, ideally with the view of being able to use an existing packagekit frontend such as Apper to install, remove and upgrade packages on a FreeBSD system. This will be written in C against the packagekit and libpkg APIs and glib, and the deliverable will be a .so file that can be used by PackageKit as a backend as well as the source code to compile it.
This would make system administration using binary packages easier in these use cases:
- Situations where multiple systems running different operating systems need to be updated by the same people; for example, University Radio York runs both Debian and FreeBSD systems which could theoretically both be served by packagekit if a pkgng backend is created.
- Users who wish to install binary packages on FreeBSD, but would prefer to use a graphical user interface;
- Users who wish to install binary packages on FreeBSD and already know packagekit from an existing supported package manager.
There is an existing packagekit backend for FreeBSD Ports; ideally I would like to implement as much of the functionality already present in that backend as is applicable to pkgng. However, the core functionality (getdepends, getdetails, getupdates, installpackages, refreshcache, removepackages, searchgroup, searchname and updatepackages) would be the main focus of at least the first leg of the project.
Approach to solving the problem
I will be using the pkgng library, libpkg, to create a C-based packagekit backend (as opposed to, say, the ports backend, which uses Ruby). This allows for the backend to be a relatively natural gluing of the two systems.
I intend to make full use of FreeBSD's unique features (sbufs, stdlib extensions, etc) as well as GLib's featureset where applicable and appropriate to make the backend development easier.
Due to problems encountered with the latest version of packagekit, initial development efforts are being concentrated on the (obsolete) version of packagekit in FreeBSD ports (0.6.11) with the aim of porting the code upwards as far as possible after a decent feature set has been achieved.
Deliverables
Pre-midterm
- A packagekit backend that has tested support for basic package installing, removing and updating (the feature matrix features getdepends, getupdates, installpackages, refreshcache, removepackages, updatepackages);
- If possible, support or substantial progress towards support for searchgroup, searchname and getdetails (otherwise these will be implemented post-midterm);
- Constant releases on an at least weekly basis of experimental source code and instructions for building for use with the version of packagekit available for FreeBSD.
Post-midterm
- A fully tested packagekit backend that provides all functionality currently ascribed to the "ports" backend with the exception of any functionality not implemented or applicable for pkgng;
- Documentation from both user and developer perspectives explaining what packagekit is, how to obtain it, how to use it with pkgng (including with GUIs) and how to further develop it;
- A codebase (verified via static code analysis tools and peer review) that conforms to style(9) and best practices, and is ready for extension to support any features unimplemented by the end of the project.
Milestones
Week one (June 17): Start of coding, pk_backend boilerplate, mapping PackageKit package IDs to libpkg names, ideally some way of testing this (which probably entails implementing package description). By the end of week 1 the backend should be compiling and presenting a (very empty) feature set to packagekit, and this will be tested using pkcon. Done
Week two (June 24): package description implemented and contingency (as week 1 may be the first foray into actually programming against the APIs, I imagine it'll take longer to "get right") Done (plus GetFiles, GetRepoList)
Week three (July 1): local package installation Replaced with remote package installation, which was done instead
Week four (July 8): package removal Done
Week five (July 15): contacting external repositories, package database updating Done
Week six (July 22): package updating and remote package installation Remote package installation done, updating appears to be done but was never fully tested.
Week seven (July 29): Contingency and mid-term evaluation; I'll probably be going away on family holiday this week, but will try to do some off-Internet work (documentation and testing?)
Week eight (August 5): Remove query-based jobs; improve events system to provide correct errors on job failure; more in-depth checking for any issues remaining from the jobs change. Removal done, error reporting attempted but without much progress
Week nine (August 12): GetDepends, SearchDetails and GetRequires; at this stage with the exception of Cancel the backend should be at feature parity with ports. In-depth attempts to fix the install crash bug. Try implementing Cancel, RepoEnable (text file hackery probably), and WhatProvides; this can roll over into other weeks if needs be. GetDepends and GetRequires done; SearchDetails not.
- Week ten (August 19): Contingency for the above, and also preparation of test environment.
Week eleven (August 26): Tests for each idempotent action. Unit tests for most major utility/glue functions.
Week twelve (September 2): Tests, where possible, for non-idempotent actions, jail/VM if possible. Basic install/get-files/removal workflow test written.
Week thirteen (September 9): Create a port and upload to ports tree. Also, final call for updating PackageKit: if possible, updating; if not, contingency and cleanup. Port submitted for CFT. PackageKit not updated yet.
- September 16: End of coding (soft)
- September 23: End of coding (hard)
- Post-GSoC: I will ideally continue to maintain the backend and oversee the implementation of any functionality left out during GSoC.
Test Plan
Initial Ideas
To be discussed with my mentor. Test environment bring-up in week 10.
Current testing efforts are concentrated on informal checking of the output of pkcon as well as some experimental automated acceptance testing via running both pkcon and pkg's equivalent processes on a set of packages and seeing if the output of one can be "mocked up" by the output of the other.
As well as this, static analysis (clang static analyser as well as simple compiler warnings at WARNS level 7) is used.
Possible other forms of testing include:
unit testing Unit tests available (using ATF)
- finding test users
checking on multiple architectures and systems (URY systems may be useful here) Appears to work on a 9.1 32-bit system and 9.2 64-bit VM.
fuzz testing via specifically injecting invalid PackageIDs Not done yet
Tests Available
At time of writing there are:
- unit tests (using ATF, these work; use "make tests" in the backend directory, then "atf-run|atf-report" in the backend, backend/jobs and backend/query directories);
- functional tests in shell script comparing pkcon output to pkg (mostly broken, will be fixed shortly).
Running the Tests
To be updated as I go along. Ideally this should be reproducible elsewhere.
- Make a new FreeBSD jail.
- Bootstrap pkg.
- Get a copy of the ports tree; install subversion and kuya either as ports or packages.
svn clone https://socsvn.freebsd.org/socsvn/soc2013/mattbw/.
Either install PackageKit and make && make install in the "backend" directory, or test the port by cding to "backend/port" and running "make install". Note that the port will lag behind the svn repo.
For unit tests, run make tests then atf-run|atf-report in the "backend" and "query" directories.
For functional tests, check the "tests" subdirectory of "backend". Some of these will need the pkgng repo changing to the test repository (http://www-student.cs.york.ac.uk/~mbw500/testrepo).
The Code
The code (may not be in a compileable state at any given time) is at https://svnweb.freebsd.org/socsvn/soc2013/mattbw/. For cloning, replace "svnweb" with "socsvn".
I usually try to prefix any broken commits' messages with warnings (eg "(BROKEN)").
The main backend development effort is in the directory backend; other elements (such as test code, documentation and suchlike) lie outside of this directory.
All code is GPL2. (Sorry!) This is because PackageKit itself is GPL2. Any code that does not link against PackageKit (should any code exist) can probably be re-licenced as FreeBSD, but I'd rather not do so without legal assurances.
Compiling, Installing and Using
Remember to install ports-mgmt/packagekit and ports-mgmt/pkg before trying to compile.
Compiling
Unlike the PackageKit backends that live in PackageKit's tree, this backend uses a BSD-style Makefile; this means that you just need to run make and sudo make install. Compilation should work with both FreeBSD GCC and clang; any failure to build using the normal settings should be classed as a bug. (The port should build with clang's -Weverything -Werror, as long as system includes are disabled.)
Installing
make install installs to the normal PackageKit backends directory (/usr/local/lib/packagekit-backend).
You will need to edit the PackageKit configuration file (/usr/local/etc/PackageKit/PackageKit.conf), changing the default backend from ports to pkgng.
Using
The PackageKit CLI is called pkcon. Here are a few sample commands to try:
pkcon get-details PACKAGE-NAME - Retrieves a summary of the package.
pkcon search name NAME - Searches for a package by name.
pkcon search group NAME - Searches for a package by group (these are basically ports categories, but with different names).
pkcon get-files PACKAGE-NAME - Retrieves a list of files for that package (must be installed)
pkcon install PACKAGE-NAME - Installs a package.
pkcon remove PACKAGE-NAME - Removes a package.
pkcon update PACKAGE-NAME - Updates a package.
pkcon refresh - Updates the repository catalogues.
pkcon resolve PACKAGE-NAME - Mainly an internal command, this shows the mapping from a package name to a package.
The search and resolve commands can take filters; currently these are --filter installed and --filter "~installed" to limit the search/resolution to installed or non-installed packages respectively.
To specify a version for a package, use a package ID instead of a name. These are strings of the format name;version;arch;repository (where the last two can be omitted, the repository is installed or repo-REPOSITORYNAME, and the arch is for example freebsd:9:x86:32.
Note that you may need to run pkcon as root. Technically this should not be the case, but ConsoleKit/PolKit wouldn't work for me.
In addition to pkcon, you can use the GNOME PackageKit frontent for a graphical user interface...
Graphical user interface
By installing ports-mgmt/gnome-packagekit, you can use the GTK user interface (gpk-application). Again, you may need to run this as root (or solve the mysteries of PolKit and ConsoleKit...!).
The GTK interface is fairly self-explanatory, and is based around a menu of possible packages. You can select a package group to look through on the left hand side, or search using the search bar in the top-left. Other actions are available from the top-level menu.
There appear to be some minor glitches with the GTK interface, which may be due to problems in the backend. Hopefully these will be fixed eventually!
Further Work
Updating PackageKit to 0.7.x or 0.8.x (will need some changes to the backend);
- Implementing Cancel and any other features "left behind";
Implementing SearchDetails;
- Implementing some switches for existing features;
- Bug fixing.
Notes
- Check the README in the root of the branch for information about how to compile, bugs, etc. etc.
I'm generally most accessible via IRC (irc.freenode.net #pkgng and #PackageKit, EFnet #freebsd-soc). During the GSoC period I will be using the nick mattbw exclusively. Afterwards, I will be mattbw or hayashi.
Useful links
JustinMuniz's GSoC2013 project SummerOfCode2013/pkgQtGtk covers very similar ground, so there will likely be overlap and co-operation.
Porting guide for PackageKit 0.8.x, for use when porting 0.6.x code over.