Historical data for the FreeBSD package building cluster

The idea is that the output generated by the package building cluster (cluster), build log files and contents of packages, will be used in to create a historical database of package information.

Current available systems

Cluster webpages (aka Pointyhat)

  • Display current package builds status and last attempt.
  • Mostly used by portmgr team to communicate with maintainers/commiters to investigate buildfailures.
  • Huge chunks of text, no easy searching.
  • Maintained by kris@, mostly static HTML pages.

Portsmon:

  • Display current package build failures, matches PRs related to packages.
  • Has some searching capabilities.
  • Maintained by linimon@, mod_python with mysql database.

FreshPorts:

  • Displays available ports, no packages.
  • Has searching capabilities.
  • Maintained by dvl@, PHP with postgresql database.

Current lacking issues

  • Build failures are too difficult to track, portsmon is a good first step but lacks the next steps (for me: Python isn't a language I know)
  • No historical build data available for end-users (but it's available on the cluster)
  • No package content information available (but it's in the packages built)

Requirements

  • Access to the build logs (they can be big)
  • Access to the packages (they can be big)
  • Alerts when a package is has been (attempted to be) build (Time + building type + OS Version + Architecture + ports path + package name + result)
  • Some backend database and a scripting language

Alerts generated by cluster

  • Prefered: SMTP email (reliable, fire and forget for cluster, forgable)
  • Workable: Tailing of a logfile (reliable, require file access to build cluster)
  • Advanced: XML post (backend must be available, has to be retried by cluster, authenticable by source address)

Workflow of backend

When an alert comes in, the backend...

  • grabs the buildlog, stores the metadata in the database.
  • extracts the +CONTENTS of the package and stores that data in the database.

Metadata

Buildlog: Datetime, port, OS version, architecture, build-type, packagename, result.

Contents: filename, path, size, package, package.

Features of the frontend

  • Search for package contents (which port / package installed that file)
  • Search for failed builds (per maintainer, Architecture, OS Version)
  • Browse historical data (this port has never been built successfully on 6.x, but always on 5.x)
  • Graphs! (overall build success rate, package availability, etc)

Other features

  • Improved CONFLICTS determination (because we know the contents)
  • Automatic alerting if the build of a package fails (for the first time)
  • Automatic alerting if the build of a package is successful again.
  • Grab my XML! (RSS feeds for maintainers, database contents for portsmon/freshports/others)

Things to think about

  • Can this be integrated on portsmon? (although I'm not capable of Python, and that machine doesn't have access to the packages, but it has interesting Gnats database information)
  • Can this be integrated on the cluster? (or a machine close to it so that we can grab the packages)
  • Make a Proof of Concept with build logs for a limited time.

PackageBuilderHistoricalData (last edited 2008-06-17 21:38:16 by localhost)