PEP-517

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255722 https://reviews.freebsd.org/D36290

Introduction

PEP-517 introduces a new build and install paradigm for Python packages that the ports framework, as of Q2 2022, does not support. This page explains why such support is needed and discusses how to get there.

In a nutshell, PEP-517 is a minimal format/interface/standard for Python package build systems to generate wheels from source trees or distributions, which can then be installed into one's Python distribution (site-packages).

Why support PEP-517

Packages from CHEESESHOP are increasingly not shipping with setup.py that USE_PYTHON=distutils executes to build and install the package in staging. Instead, they follow PEP-517 with a pyproject.toml, similar to Rust/Cargo packages' Cargo.toml; any additional files related to building and installing are specific to the chosen build systems in pyproject.toml. setuptools no longer plays the central distutils role of building and installing; it is now one of many build backends packages can specify.

Additionally, distutils is deprecated with removal planned for Python 3.12. We need to start future-proofing the framework now, not least to avoid surprises later, but to gradually allow existing ports to transition and add new ports following PEP-517.

Usage

USE_PYTHON=pep517 exists to streamline do-build and do-install's common mechanics. Operation is otherwise similar to USE_PYTHON=distutils. It is the porter's responsibility to parse pyproject.toml for the correct build backend and add it into BUILD_DEPENDS. Example:

PORTNAME=       sample
DISTVERSION=    1.2.3
CATEGORIES=     devel

MAINTAINER=     freebsd@example.org
COMMENT=        Python sample module

BUILD_DEPENDS=  ${PYTHON_PKGNAMEPREFIX}flit>0:devel/py-flit@${PY_FLAVOR}
RUN_DEPENDS=    ${PYTHON_PKGNAMEPREFIX}six>0:devel/py-six@${PY_FLAVOR}

USES=           python
USE_PYTHON=     autoplist pep517

.include <bsd.port.mk>

In this configuration, during do-build, devel/py-build parses pyproject.toml, checks that requirements are met and executes the specified build backend, in this case devel/py-flit, which generates a wheel. During do-install, devel/py-installer installs the wheel into STAGEDIR.

USE_PYTHON=autoplist will read the installed/staged wheel's RECORD (CSV) file, prepend ${PYTHONPREFIX_SITELIBDIR} to each line and strip the other columns to generate the temporary plist. If USE_PYTHON=concurrent is specified, files installed into bin/ are moved to their proper names. ${PREFIX} is stripped to generate the final plist. Compiled bytecode is not included in RECORD, so it is not compiled unlike USE_PYTHON=distutils. This also accommodates future pkg triggers that compile bytecode at install time rather than stage time.

Build backend

The build backend is usually specified in the [build-system] table. If this table is not defined, per PEP-518, the default is:

[build-system]
# Minimum requirements for the build system to execute.
requires = ["setuptools", "wheel"]  # PEP 508 specifications.

thus both of these need to be specified in BUILD_DEPENDS.

For those using flit, BUILD_DEPENDS shall include devel/py-flit and not devel/py-flit-core. The latter is for bootstrapping the Python packaging ecosystem only (see below).

Feature notes

Supported feature comparison

Feature	`distutils`	`pep517`
`concurrent`
`cython`
`cython_run`
`flavors`	(implied)	(implied)
`noflavors`
`allflavors`
`optsuffix`
`distutils`
`autoplist`
`py3kplist`
`pythonprefix`
`noegginfo`
`nose`
`nose2`
`pytest`
`pytest4`
`unittest`
`unittest2`

autoplist

This method of parsing RECORD only supports entry_points installed to bin/ and not other data_files installed elsewhere in hier(7). At least setuptools, through PyPA guidance, have deprecated this practice, and general consensus amongst language package managers regard this as out of their scope, left as exercises for downstream packagers. Install any such data_files ourselves, using the appropriate INSTALL variables, through post-install and append them to PLIST_FILES, rather than manually constructing pkg-plist. Please work with your package upstreams to comply with Python packaging standards.

Implementation

The framework will almost entirely use devel/py-build as the build frontend for do-build and devel/py-installer as the install frontend for do-install so it is important that these and their RUN_DEPENDS bootstrap properly, without distutils or devel/py-setuptools in the environment. The ports framework serves as the integration frontend.

Design goals

To start, provide the same features available in both USE_PYTHON=distutils and PEP-517 using the same syntax as practicable. Leave room for accommodate future Python standards or accepted practices in the wild.

Bootstrapping

Especially since none of the tooling needed for PEP-517 support exists in the base Python distribution, unlike distutils, we have to bootstrap some of the tooling. Leveraging the framework comes after successful bootstrapping.

We will follow the process for bootstrapping devel/py-flit-core since it does not depend on anything or introduce circular dependencies. Other packages during this step will use it to build themselves. The Python Team (python@) shall maintain these ports:

devel/py-flit-core https://reviews.freebsd.org/D34786
devel/py-installer https://reviews.freebsd.org/D34789
devel/py-build https://reviews.freebsd.org/D37921
1. devel/py-packaging https://reviews.freebsd.org/D37920
2. devel/py-pyproject_hooks
  1. formerly devel/py-pep517 https://reviews.freebsd.org/D34875
3. textproc/py-tomli https://reviews.freebsd.org/D34876

"Test suite"

The above mentioned bootstrap ports/packages will serve as the "test suite". It is a small quantity yet diverse enough collection to exercise the framework, including overridable variables.

CategoryPorts