portmgr/home

This is still a work in progress. More details to be added for the builder section and REST API.

Port CI

Overview

The build application will be the central place for queueing, monitoring, and viewing exp-run results. It will have an HTML and JSON REST interface. It will also display production package build status. This application will be the "master" for exp-runs.

New builds will be queued into the master. They may or may not have patches. Patches can be for the ports tree or src tree. Builds can be configured to do a comparison once completed against a known good build to find new failures.

Client builders will check in to the master and ask for work. They will occasionally report their status as well.

Lost/crashed builds will be retried a few times before being marked failed.

Result analysis

Benefits

Problems with exp-run/build automation not solved here

  1. Poudriere stability. See Stability section at https://fossil.etoilebsd.net/poudriere/doc/trunk/doc/todo.wiki

  2. Patches against src/ports tree that the nightly reference is not using require running a specific reference for that patch. It could try to find the "closest" reference, but would deliver many false-positives. We don't want people looking at the list of failures and misjudging the results. We want confidence in the results so we do not introduce new failures into head.
  3. Similar to 2, need to automatically run a reference build if needed. Leaving out of initial design as it complicates things too much. In general the nightly reference build has worked well. Asking submitter to rebase patch against a known reference revision seems reasonable for version 1.

Build types

  1. Exp-run

    • Patch against the src or ports tree.
    • All ports will be built.
    • All existing packages will be deleted.
    • Completion will compare against a Reference build.
    • svn patch; bulk -ca

  2. Reference

    • No Patch.
    • All ports will be built.
    • All existing packages will be deleted.
    • Completion will compare against previous Reference build.
    • bulk -ca

  3. QAT

    • No patch.
    • All ports will be built.
    • All existing packages will be deleted.
    • Plist testing will be done.
    • Completion will compare against previous QAT build.
    • bulk -cat

  4. Port

    • Patch against the src or ports tree.
    • Only the 1 port will be built.
    • Existing packages for that 1 port will be deleted.
    • Plist testing will be done.
    • svn patch; testport -n -o

  5. Package

    • No patch.
    • All ports will be built.
    • Existing packages kept.
    • NO_PACKAGE packages not built, RESTRICTED packages cleaned up after bulk.
    • Completion will compare against previous build for new failures/skipped (not built as it will be different due to incremental).
    • bulk -a

Authentication

Queue system

Builder/Master

Builder registration

  1. Portmgr will login to master and generate a token. This token will only display once and never again. This is the shared secret for authenticating requests and must remain secure.
  2. Portmgr will setup new builder and specify the token and the master's SSL CA public key in its configuration.
  3. Builder starts up and generates an id from uuidgen.

  4. Builder connects to Master to register its id and hostname with the token.
    • Possibly uses DH1080 here to insert another shared secret that is never exchanged, for each to use in future communications.
  5. Master registers the builder and associates its id and hostname with the token.
    • If token is already registered, deny registration.

Builder requests

Master responses

Notifications

IRC

Gnats

Email

Queue/Build process

  1. Patch is uploaded by committer and configured for a specified build.
    • Only exp-run and port builds are available to non-portmgr.

    • Portmgr can queue any build type
  2. Build is marked unapproved until a portmgr approves it.

  3. Builders check in frequently for work.
    • Once builder takes a build, a new job is created and assigned to the builder.
    • The build will be marked running. It will provide a Build URL back to the master for it to update the job object.

    • A patch id will be given in the response, along with checksum.
  4. Builder will download associated patch from master if provided and compare checksum.
  5. Build starts.
  6. The builder will checkin every 5 minutes and on boot.
    • Missing 4 checkins will consider the job as lost and cause it to have the build's fail_cnt incremented and its status moved to retry to retry.

    • A crashed or stale build seen on startup will notify master of the failure and have the build's fail_cnt incremented and its status moved to approved to retry.

    • If a build crashes 3 times it is marked as failed and not retried again. It may be causing panics and should not continue to bring down builders.

    • A build can be cancelled at any time by a user. When the builder checks in, if the build the job is is running for is cancelled it will receive that notice and cancel its work. When completed it will report back to the master and it will mark the job and build as cancelled.

  7. When a job is completed the builder will notify the master and provide a URL to a tarball of its log files, along with checksum.
    • If the reported job is successful and has been requeued/reassigned then the new job should be aborted.
    • If the reported job is not finished, but is already considered lost, then the job will be aborted.
  8. Master will download the logfiles, compare checksum, and then extract locally for later display and analysis.

Exp-run/Port process

  1. When the build is completed, it will be marked as pending-analysis

  2. When the master has an adequate reference build available it will compare against pending-analysis builds, update their results, and then mark them as analysed.

  3. Results will be mailed to any associated PR, portmgr, and the person who queued it.

QAT process

  1. When the build is completed, it will be marked as pending-analysis

  2. When the master has an adequate reference build available it will compare against pending-analysis builds, update their results, and then mark them as analysed.

  3. New failures will be mailed to ports@, portmgr and potentially CC all committers on the hook for the commit range.

Master

Periodic checks

Database Tables

Token

Builder

Build

Job

Patch

REST API

User

Unauthenticated

List all builds

List all jobs

List all patches

List all builders

Retrieve patch

View build

View job

Authenticated: Committer

Create build

Authenticated: Portmgr,Clusteradm

Revoke builder

Halt new jobs

Resume new jobs

Authenticated: Portmgr

Generate builder token

Approve build

Abort build

Abort job

Builder

Authenticated

Register

Request work

Report status

Builder

portmgr/PortCI (last edited 2014-07-31 14:47:53 by AntoineBrodin)