• Python

Python

FreeBSD supports multiple, concurrently installable Python versions and environments for Python users and developers to work with. Every Python available in the FreeBSD ports tree can be built for any (and multiple) Python version using the FLAVORS and DEFAULT_VERSIONS framework mechanisms.

Binary packages available from the official FreeBSD Package repositories are built using a DEFAULT_VERSION of Python, which evolves over time.

Our Python Team works closely with upstream, aiming to resolve issues with an upstream-first approach and stays close-to-upstream with respect to documentation, guidance and recommendations.

FreeBSD is officially supported by, and provides multiple FreeBSD continuous integration workers for the Python project to ensure future versions of FreeBSD are supported, and so bugs and regressions are identified and resolved prior to release.

Contact

• #freebsd-python IRC channel (Libera Chat)

• freebsd-python mailing list

Goals

∞ - Ongoing

• Resolve Open Python Issues

• Keep Python Ports up-to-date: Portscout and Repology

• Fullfill the Python Port Wishlist

• Improve Python Ports Policy

↻ - In Progress

☐ - Open

• ☐ Files that fail to compile (compileall) are still added to distutils (therefore setuptools, therefore pip) --record output, breaking plists

  • Python issue 20397 Needs patch

• ☐ (Un)Official poudriere repository with Python 3.x default (If you want to do this talk to koobs@)

• ☐ Support USES=python:x.x,x.y,x.z (or x.x-x.z,!x.y) version specification
• ☐ Productivity: Tool to create Python port from a Python (PyPI) sdist (and its setup.py file, etc)
• ☐ Add a stage to Python framework to find and strip shared libraries in a consistent manner
  • Right now, either each port does it itself, or not at all (BLANKET)
  • Once done, remove blocks from ports that do this themselves

• ☐ Idea: Extend USES=python[:features] or USE_PYTHON=<feature>

  • ☐ :tests - Adds TEST_DEPENDS to BUILD_DEPENDS, adds regression-test: ${PYTHON_CMD} ${PYDISTUTILS_SETUP} test

  • ☐ :<nosetests|pytest|tox|etc> - Adds the right things to TEST_DEPENDS, then adds/runs regression-test: <correct invocation>

  • Q: How can we easily switch tests on/off for upstream packages that fix or break the test suite over time
  • Q: Can we unconditionally turn tests on, but not have packaging fail if the test fails? Fiddle with stderr? Test suite args?
• ☐ Remove Python 3 specific port duplicates, once pkg_install support is retired. Ports with the same ORIGIN can be installed for different Python versions in parallel.
  • ☐ Remove devel/py-setuptoolsXX
  • ☐ Remove devel/py3-dbus, devel/py3-gobject3, devel/py3-xdg, audio/py3-pylast, textproc/py3-libxml2, graphics/py3-cairo, multimedia/py3-gstreamer1

• ☐ Resolve Distutils install_data command contains directories in --record <RECORDFILE>

  • Observable with PYDISTUTILS_AUTOPLIST when data_files contains a directory-only entry

  • Original commit and issue #444589

  • Example: devel/buildbot-slave (setup.py)

  • Example: databases/pyspatialite (setup.py)

  • Example: graphics/py-pygraphviz (setup.py)

• ☐ Don't package (plist) or install .pyc and .pyo files. Details: /CompiledPackages

• ☐ Support for granular dependencies (eg: >=1.0,<2.0)

  • ☐ Look at what adding pip support might look like (supports these version checks)
• ☐ bsd.python.mk is PREFIX-safe

  • What specific things are currently not PREFIX-safe ?

• ☐ Consistent Python ports prefix policy and remove PYTHON_PKGNAMESUFFIX (see this blog post for details)

• ☐ All Python ports are local-patch free (upstream or remove all local patches)

  • ☑ Issue #18178: Redefinition of malloc(3) family of functions at build time (Status: Upstreamed)

  • ☐ Disable ncurses detection patch (Status: needs upstreamable patch)

    • ports/171246: Make python curses module work with unicode

  • ☐ Posix P1003.1b semaphore support patch (Status: needs issue created)

    • ports/178301: lang/python2[67]: SEM option is inconsistent

• ☐ Fix gtk20's /usr/local/bin/gtk-builder-convert dependency to Python by splitting it up into an own slave port?
• ☐ Integrate lang/pypy into ports framework (knobs etc)

  • Mk/Uses/python.mk: add support for PyPy patch review

• ☐ Python 3.x is the default Python language version
• ☐ Python version agnostic DOCSDIR and EXAMPLESDIR; they currently need to be overridden manually - may conflict with the PYTHON_CONCURRENT_INSTALL efforts
• ☐ Make it possible to add extra arguments to PYDISTUTILS_INSTALLARGS and friends without having to include bsd.port.pre.mk.
• ☐ create guideline for which ports python@ will accept to maintain. In particular, address 'leaf ports' and 'non-infrastructure'

☑ - Done

• ☑ Python ports framework complies with all relevant PEP's

  • ☑ PEP-394: The "python" Command on Unix-Like Systems

  • ☑ PEP-449: Removal of the PyPI Mirror Auto Discovery and Naming Scheme - r400105

  • ☑ PEP-3147: PYC Repository Directories (we use different site-packages/ and support concurrent installation (which creates distinct directories)

• ☑ Deprecate INSTALLS_EGGINFO: Use --record output to determine if a Python package installs .egg-info metadata files or directories.

  • 20141002: Commit r369800

• ☑ --record <RECORD> can contain paths that include ../ or ./

  • 20140925: Commit r369296

• ☑ Split bsd.python.mk into a USES for consumer ports and an port-tree specific .mk file

  • 20140809: Commit r364450

• ☑ Reduce shared/static build complexity in lang/pythonXX ports using lang/python34 as template

  • 20140802: Commit r363790 Commit r363730 Commit r363788

• ☑ Remove easy_install specific bits in bsd.python.mk

  • 20140708: Commit r361263

• ☑ PYDISTUTILS_SETUP does not strip arguments properly, which causes setuptools and distutils to jump to wrong conclusions. This is a nuisance right now (manifest_maker warnings), but can become a serious issue.

  • 20140628: Commit r369663

• ☑ DATADIR, EXAMPLESDIR, binaries, scripts and files in ETCDIR need to be python prefix safe

  • 20140607: Commit r356921

• ☑ Use python setup.py install --root=<path> instead of setup.py easy_install in Mk/bsd.python.mk

  • 20131215: Commit r336553

• ☑ Ports limited to certain Python versions must not rely on ${PREFIX}/bin/python, but the proper python versions

  • 20131218: Commit r336850 - lang/python has been removed as implicit build dependency

• ☑ Chase the distribute merge back into setuptools

  • 20131102: [RFC] devel/py-setuptools

  • 20131127: Commit r335043 (Credit: wg)

• ☑ STAGE support for setuptools-based ports (USE_PYDISTUTILS=easy_install)

  • Mk/bsd.python.mk currently uses the easy_install setup.py target to install setuptools-based modules.

  • setuptools and distutils build and install targets support a --root option. The easy_install target and CLI command do not.

  • ☑ 20131110: Workaround easy_install by installing in stage dir and recompiling r333286 (Credit: nivit, wg)

• ☑ 20131029: freebsd-python: List of ports with "python 3" errors in poudriere

  • lang/python installation conflict

• ☑ Retired devel/py-psyco (12 March 2012: Psyco is unmaintained and dead)

  • 20131015: Commit r330399

• ☑ Automate pkg-plist generation (r328403: PYDISTUTILS_AUTOPLIST)

• ☑ Zope

  • remove zope27

  • remove 28 29 210

  • update 211 (it's on it latest version already)

  • import 212 (do we really need this?)

  • import 213

• ☑ (plone (import plone4))
• ☑ (turbogears2 (update to 2.1))

• ☑ [EXP-RUN] [patch] lang/python: speed up upgrade-site-packages) ports/155970 r304961

Other Interesting Things

• PEP-446 -- Make newly created file descriptors non-inheritable (Python 3.4, FreeBSD *_CLOEXEC Support)

  • Supercedes PEP-433

• PEP-427 -- The Wheel Binary Package Format 1.0

Tooling Challenges

The upstream Python packaging team has expressed interest in understanding the challenges faced by OS packagers with Python packaging tools. This information will be used to inform ongoing improvements and developments. Please document your experiences below:

• StageDir with setuptools-based (easy_install) modules -- Doesnt support --destdir or equivalent

Steps to switch the Python default version

1. A successful exp-run is mandatory
2. Set the new default version in
   • bsd.ports.default.mk
     • PYTHON3_DEFAULT
   • Uses/python.mk
     • _PYTHON_VERSIONS (retain order of preferred versions)
     • _PYTHON_PORTBRANCH
3. Bump the PORTREVISION of following ports
   • devel/py-setuptools
   • lang/cython
   • math/py-numpy
4. Add UPDATING instructions

Legend:

• pyMM, pythonMM, pythonM.M or MM = Old Python version, e.g. py37, python37, python3.7 or 37
  pyNN, pythonNN, pythonN.N or NN = New Python version, e.g. py38, python38, python3.8 or 38

Current instructions are:

• The default version of python3 and python was switched to N.N.
  
  For ports users wanting to keep version M.M as default,
  add DEFAULT_VERSIONS+= python=M.M python3=M.M to make.conf
  
  Following procedures may ease the upgrade:
  
  # sh
  # for i in $(pkg query -g %n 'pyMM-*'); do pkg set -yn ${i}:pyNN-${i#pyMM-}; done
  # pkg upgrade
  
  For portmaster users:
  # sh
  # portmaster -o lang/pythonNN pythonMM
  # REINSTALL="$(pkg info -qo "*pyMM*")"
  # pkg delete -f "*pyMM*"
  # portmaster $REINSTALL
  # REBUILD=$(pkg query "%n:%dn" | grep py3 | grep -v pyNN | cut -d : -f 1 | sort -u)
  # portmaster $REBUILD
  # REBUILD2=$(pkg list | grep python-MM | xargs pkg which | awk '{print $6}' | sort -u)
  # portmaster $REBUILD2

  Please note that the instructions from above for portmaster are not perfect for more complex scenarios of installed ports (see TODO below).

TODO for the instructions in UPDATING

Following parts should be added with the next Python switchover:

Refine instructions for Portmaster

With more complex scenarios of installed ports the upgrade might not go smoothly. There have been reported issues if following ports were installed:

• databases/tdb
• devel/gobject-introspection
• devel/meson
• devel/tevent

There are improved instructions that still needs to be tested thoroughly:

Improved UPDATING instructions for Portmaster

Add clean up instructions

The following is a draft based on a suggestion from ThierryThomas (thierry) to improve the user experience.

• Final steps (for pre-built packages & portmaster):
  
  If no longer required, Python M.M can be removed via
  "pkg remove pythonMM" and the directory /usr/local/lib/pythonM.M can
  then be deleted afterwards, if not empty.

Python Team

• You! Come and say hi!

• BabakFarrokhi (farrokhi@)

• CharlieLi (vishwin@)

• DaniloBaio (dbaio@)

• FukangChen (loader@)

• KaiKnoblich (kai@)

• LiWenHsu (lwhsu@)

• MuhammadMoinurRahman (bofh@)

• NicolaVitale (nivit@)

• PoChuanHsieh (sunpoet@)

• RuslanMakhmatkhanov (rm@)

• SofianBrabez (sbz@)

• StevenKreuzer (skreuzer@)

• WenHeping (wen@)

CategoryPorts CategoryTeam CategoryStale