PEP-517

Introduction

PEP-517 introduces a new build and install paradigm for Python packages that the ports framework, as of Q2 2022, does not support. This page explains why such support is needed and discusses how to get there.

In a nutshell, PEP-517 is a minimal format/interface/standard for Python package build systems to generate wheels from source trees or distributions, which can then be installed into one's Python distribution (site-packages).

Why support PEP-517

Packages from CHEESESHOP are increasingly not shipping with setup.py that USE_PYTHON=distutils executes to build and install the package in staging. Instead, they follow PEP-517 with a pyproject.toml, similar to Rust/Cargo packages' Cargo.toml; any additional files related to building and installing are specific to the chosen build systems in pyproject.toml. setuptools no longer plays the central distutils role of building and installing; it is now one of many build backends packages can specify.

Additionally, distutils is deprecated with removal planned for Python 3.12. We need to start future-proofing the framework now, not least to avoid surprises later, but to gradually allow existing ports to transition and add new ports following PEP-517.

Getting there

The framework will almost entirely use devel/py-build as the build frontend for do-build and devel/py-installer as the install frontend for do-install so it is important that these and their RUN_DEPENDS bootstrap properly, without distutils or devel/py-setuptools in the environment. The ports framework serves as the integration frontend.

Bootstrapping

Especially since none of the tooling needed for PEP-517 support exists in the base Python distribution, unlike distutils, we have to bootstrap some of the tooling. Leveraging the framework comes after successful bootstrapping.

We will follow the process for bootstrapping devel/py-flit-core since it does not depend on anything or introduce circular dependencies. Other packages during this step will use it to build themselves.

  1. devel/py-flit-core https://reviews.freebsd.org/D34786

  2. devel/py-installer https://reviews.freebsd.org/D34789

  3. devel/py-build committed but not in a release

    1. devel/py-packaging committed but not in a release

    2. devel/py-pep517 https://reviews.freebsd.org/D34875

    3. textproc/py-tomli https://reviews.freebsd.org/D34876

Framework

We will introduce a new USE_PYTHON=pep517 to streamline do-build and do-install's common mechanics. It will be the porter's responsibility to parse pyproject.toml for the correct build backend and add it into BUILD_DEPENDS. Example:

PORTNAME=       sample
DISTVERSION=    1.2.3
CATEGORIES=     devel

MAINTAINER=     freebsd@example.org
COMMENT=        Python sample module

BUILD_DEPENDS=  ${PYTHON_PKGNAMEPREFIX}flit>0:devel/py-flit@${PY_FLAVOR}
RUN_DEPENDS=    ${PYTHON_PKGNAMEPREFIX}six>0:devel/py-six@${PY_FLAVOR}

USES=           python
USE_PYTHON=     autoplist pep517

.include <bsd.port.mk>

In this configuration, during do-build, devel/py-build parses pyproject.toml, checks that requirements are met and executes the specified build backend, in this case devel/py-flit, which generates a wheel. During do-install, devel/py-installer installs the wheel into STAGEDIR.

USE_PYTHON=autoplist will read the installed/staged wheel's RECORD (CSV) file, prepend ${PYTHONPREFIX_SITELIBDIR} without ${PREFIX} to each line and strip the other columns to generate the plist. Since compiled bytecode is not included in RECORD, plus to accommodate future pkg triggers that compile bytecode at install time rather than stage time, we will disable compiling bytecode.

"Test suite"

The above mentioned bootstrap ports/packages will serve as the "test suite". It is a small quantity yet diverse enough collection to exercise the framework, including overridable variables.

Design goals

To start, provide the same features available in both USE_PYTHON=distutils and PEP-517 using the same syntax as practicable. Leave room for accommodate future Python standards or accepted practices in the wild.

Supported feature comparison

Feature

distutils

pep517

concurrent

(./)

(./)

cython

(./)

(./)

cython_run

(./)

(./)

flavors

(./) (implied)

(./) (implied)

noflavors

{X}

{X}

allflavors

(./)

(./)

optsuffix

(./)

(./)

distutils

(./)

{X}

autoplist

(./)

(./)

py3kplist

{X}

{X}

pythonprefix

(./)

(./)

noegginfo

(./)

{X}

nose

nose2

pytest

pytest4

unittest

unittest2


CategoryPorts

Python/PEP-517 (last edited 2022-11-17T06:50:48+0000 by CharlieLi)