pkg_improved
About
The current pkg-tools uses some ineffective ways of keeping track installed packages and it's information, so the goal is to make a self contained project replicating the original pkg_* programs with the use of the BerkeleyDB as a cache for making certain operations a lot faster. Adding additional fields for the package install date and a more useful size information (human readable) could also be needed.
Milestones
Determine what information should be indexed in the database; The things that are beeing indexed are files installed by each package, it's origin and installdate. By indexing each package the 'pkg_info -W' command executes much faster. By indexing it's origin, origin lookups work a lot faster, and this is used throughout the tools. And by indexing the installdate, the new features in pkg_{info/delete} comparison-search with installdate (e.g., pkg_info -M '*>2008-08-15') works fast.
- Implement database support functions (lib); Functions are beeing continually added/removed as needed to make the queries easy and the database-layer abstract so that switching databases if needed, could be made easy for the future.
- Modify existing tools to make use of the database cache; So far pkg_info has been improved by using the cache, like the -W option, this is a work in progress and I'm planning to make better size queries, and perhaps adding more fields to the database (like install date, this would have to be discussed).
- Write and add a converter(pkg_convert) that caches /var/db/pkg into a database file so that you can make use of the cache
- Update the database indexes correctly in pkg_add/pkg_delete and "make install" in /usr/ports, so that you won't have to run pkg_convert after each add/delete.
- Change the output formatting of some of pkg_info's options, especially size and pkg_add's -r option.
- Identifying bottlenecks with profiling tools like gprof.
If you have any suggestions or disagreements please contact me at <andenore AT SPAMFREE FreeBSD DOT org>
Changes
Converter:
- Added convert program that indexes information to PKG_DBCACHE_FILE
- Features to lookup keys and print them (or print whole db)
- Caches origin, installtime and which-files (files in package to pkgname)
Info:
- Improved performance for -W option using the dbcache
- Profiling showed that the default behavior for pkg_info was to read plist everytime even though not needed. Added check for this and speed improved significantly.
- Uses human readable output for -s (size option) I'm not sure if this breaks things, but it looks Ok. (The old output is available via the -b option)
- Print installation date with -n (human readable) or -N (seconds since epoch)
Added installtime comparison with the -M option, e.g, "pkg_info -M '*>2008-07-18 18:30:21'" will list all packages installed after the date YYYY-MM-DD hh:mm:ss.
Add:
- Indexes information to dbcache according to the add
- Added progress indication to the remote fetching option (-r), similar to that of fetch(1)
- Fixes dependencies when installing (scans installed ports, checks @pkgdep and adds matching dependency to the installing package's +REQUIRED_BY file)
Delete:
- Deindexes information according to the delete
Now supports range deletion with installdates, e.g., "pkg_delete -i -M '*>=2008-07-14<2008-08'" will delete all packages installed between the dates 2008-07-14 and 2008-08.
Lib:
- Changed methods in match.c to use cache if available (matchallbyorigin, ispkginstalled)
- Added global variable 'database' of type DB* (the database object)
- Added database helper functions
- Added function to cache a Packagelist (cache_plist())
- The environment variable PKG_DBCAHE_FILE can be set to make the file location different than the default /var/db/pkg/pkgcache.db
- pattern_match() will compare dates as well as version numbers.
Create:
- The -O option has been modified to cache information when installing ports from /usr/ports/
Packinglist:
"@comment DATE:YYYY-MM-DD hh:mm:ss" has been added to installed packages (affects add/create/info/lib)
Fixed bugs which exists in -CURRENT and possibly earlier versions:
- Fixed a bug with the oldstyle -W handling of empty @cwd's, which leads to a number of files who are installed by packages not beeing recognized.
- pkg_add -r of two packages (or more) where the first depends on one or more packages which is not installed causes a "fatal error during execution: getcwd" because it doesn't save previous working directories correctly.
Testing
If you would like to test the improved pkg_install tools here's what to do:
Download the latest patch for your release at http://home.no.net/andenore/patches/ into "/usr/src/usr.sbin/pkg_install/"
- cd to '/usr/src/usr.sbin/pkg_install'
Apply the patch with "patch -p1 < the_patch.diff"
Run "make && make install" to install the tools.
- Run pkg_convert(1) with no arguments to build the cache.
- After this you can use the programs like before.
If you notice any bugs your input will be greatly valued and please contact me.
NB. If you do get this:
duckjen# make ===> lib (all) "../Makefile.inc", line 3: Cannot open /usr/share/mk/bsd.own.mk "../Makefile.inc", line 17: Cannot open ../Makefile.inc make: fatal errors encountered -- cannot continue *** Error code 1
You are probably not compiling from /usr/src/usr.sbin/pkg_install/ and Makefile.inc will try to .include "../Makefile.inc" so comment out this line and specify BINDIR.
# Inherit BINDIR from one level up. #.include "../Makefile.inc" BINDIR?= /usr/sbin
Links
- Perforce depot: //depot/projects/soc2008/andenore_pkginstall/
Contact information here AndersNore