Workflow for Python packages

[slsa-framework]

Workflow for Python packages

[ianlewis]

Here is a summary of the relevant file formats and tools to consider when building python packages. The Python Packaging User Guide provides a good summary of the available tools and methods.

Project definition files:

• pyproject.toml- this is the de-facto project definition and is defined in PEP-518. It specifies some common metadata in the build-system section but most information seems to be tool specific under the tool section.
• setup.cfg/setup.py - pre-PEP-518 configuration for project used by setuptools.

Tools to build packages

These tools and packages support building packages based on either pyproject.toml or setup.cfg. Whether projects use pyproject.toml or setup.cfg there is enough tool specific configuration that we will likely need to detect the build tool used.

For example, many projects specify dependencies of only requires = [ "poetry-core>=1.0.0",] in the tool-agnostic build-system section and specify the real dependencies under tool.poetry.dependencies.

build seems to be the de-facto "blessed" build tool, however actual usage seems heavily divided among many build tools and there is no one clear leader.

• build was built as a result of PEP-517 and PEP-518 which aimed to replace distutils and setuptools. It seems that if the user specifies build-backend and/or backend-path then build can defer to that package manager. If the package manager required is specified in requires then the package could be self-contained. It's unclear how many actual pypackage.toml files are written in such a way that this actually works however.
• Poetry is a fairly popular project for Python packages.
  • It has many pyproject.yaml extensions. A naive search yields significant usage of these extensions.
• hatch is a python package build tool.
  • Unsure of how widely adopted it is.
  • Need to investigate plugins and extensions.
• flit is a python package build tool.
  • Unsure of how widely adopted it is.
  • Need to investigate plugins and extensions.
• bento is a python package build tool.
  • Unsure of how widely adopted it is.
  • Need to investigate plugins and extensions.
• setuptools uses setup.cfg. Its use is discouraged in favor if PEP-518 compatible tools but IIUC still has significant adoption.
• bazel has python rules but they don't seem to be geared towards creating reusable packages for pypi
• others - There seem to be a myriad of tools that are out there. We should probably focus on the most widely adopted tools.

Tools to publish packages

Support for publishing provenance could maybe be added to these tools after pypi has support?

• twine seems to be the "blessed" de-facto tool to upload a pre-built package to pypi.
• hatch supports publishing to pypi.
• flit supports publishing to pypi.

Tools used to install packages

• pip is the de-facto python package installer. It's very widely used and is ubiquitous enough that we can probably ignore other installers of pypi packages.

Scientific community

The scientific community seems to use other pypi/pip incompatible package managers fairly frequently due to needing access to specialized libraries and tools.

• conda - mostly used in the scientific community. AFAICT the packages are incompatible with pip packages and are served on a totally different infrastructure from pypi.
• spack - used mostly by the scientific community(?)
• hashdist - used mostly by the scientific community(?)

[ianlewis]

@di I summarized some info on python package builders above with the goal of better understanding we could put together a builder that will be able to build python packages and generate SLSA provenance for it.

As I understand it, if folks wrote pypackage.yaml in such a way as to set requires, build-backend, and build-path properly then theoretically any package build using pypackage.yaml could be bootstrapped by the build tool.

However, I don't think most packages are set up this way and, without modification, will require us to know the build tool up front in order to build the package properly. That or we force them to update their package to allow building with the build tool. And many folks still use setuptools.

Any thoughts or comments? Anything I got wrong or am misunderstanding?

[di]

Any thoughts or comments? Anything I got wrong or am misunderstanding?

I think this is fairly accurate. I'd say you should think of build as the canonical generic build tool, and everything else (flit, setuptools, etc) as PEP-517 build backends (even though some can also be used as build tools).

As I understand it, if folks wrote pypackage.yaml in such a way as to set requires, build-backend, and build-path properly then theoretically any package build using pypackage.yaml could be bootstrapped by the build tool.

This is correct (although it's pypackage.toml 😉).

However, I don't think most packages are set up this way and, without modification, will require us to know the build tool up front in order to build the package properly. That or we force them to update their package to allow building with the build tool. And many folks still use setuptools.

I think it's really our only option. We can do some things to guess the build backend that needs to be present, but it's not guaranteed to be correct, or capture all bulid-time dependencies. Instead, pyproject.toml is the standardized way to do this. I think it would be reasonable to say we only support building projects that specify pyproject.toml.

From my perspective, we're seeing a lot of projects moving to this (and a lot of tools moving their configuration into this file).

[ianlewis]

@di Thanks. I think build is definitely the thing we should do first. I think you're right that a decent number of folks use pyproject.toml but I didn't see many in my searches that set the backends properly. They looked like they expected you to install and/or run the build backend directly.

That said, it should be pretty easy for folks to update their pyproject.toml to allow us to just run python -m build on their project and have it work, so maybe that will cover ~80% of projects. Folks still using setuptools only would have issues though but maybe we can support that at some point if it's as issue.

Scientific community projects also worry me because lots use conda, but I'm ok with not supporting them, at least initially.

[di]

I think build is definitely the thing we should do first. I think you're right that a decent number of folks use pyproject.toml but I didn't see many in my searches that set the backends properly. They looked like they expected you to install and/or run the build backend directly.

Note that build uses a fallback backend if none is specified, so that might be the case: https://python-build.readthedocs.io/en/stable/#fallback-backend

[ianlewis]

Note that build uses a fallback backend if none is specified, so that might be the case: https://python-build.readthedocs.io/en/stable/#fallback-backend

Ah, ok. Noted. Though I did see a lot of pyproject.toml files with [tool.foo] sections but without any requires or backend set so I expect those not to work.

[di]

Yeah, these would be the projects that build would use the fallback backend for. The entire [build-system] section would be missing.

[ianlewis]

Yeah, these would be the projects that build would use the fallback backend for. The entire [build-system] section would be missing.

If, for example, it's exclusively [tool.poetry.x] sections in pypackage.toml, I wouldn't expect python -m build to work because build would fallback to setuptools. For example: https://github.com/razy69/poetry-tox-template/blob/208cbc3f7d972a0c7862c2f54c9dfe3fe5b74e54/pyproject.toml

If there was a separate and analogous setup.cfg or setup.py it would work though I guess.

Am I understanding it right? Or would build figure it out somehow?

[ianlewis]

We may also want to consider not having a builder in this repo but one that is better endorsed by the python commmunity. e.g. via https://github.com/pypa

[di]

That example specifies:

[build-system]
requires = ["poetry-core>=1.1.0a6"]
build-backend = "poetry.core.masonry.api"

So in theory, the build should work, but looks like it doesn't due to some issue with that project, not with Poetry or build:

$ git clone [email protected]:razy69/poetry-tox-template.git
Cloning into 'poetry-tox-template'...
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 14 (delta 2), reused 14 (delta 2), pack-reused 0
Receiving objects: 100% (14/14), 22.00 KiB | 7.33 MiB/s, done.
Resolving deltas: 100% (2/2), done.

$ cd poetry-tox-template/

$ python -m build
* Creating venv isolated environment...
* Installing packages in isolated environment... (poetry-core>=1.1.0a6)
* Getting dependencies for sdist...
* Building sdist...
Traceback (most recent call last):
  File "/home/di/.local/lib/python3.9/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
    main()
  File "/home/di/.local/lib/python3.9/site-packages/pep517/in_process/_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/di/.local/lib/python3.9/site-packages/pep517/in_process/_in_process.py", line 314, in build_sdist
    return backend.build_sdist(sdist_directory, config_settings)
  File "/tmp/build-env-oxrpsqpl/lib/python3.9/site-packages/poetry/core/masonry/api.py", line 77, in build_sdist
    path = SdistBuilder(poetry).build(Path(sdist_directory))
  File "/tmp/build-env-oxrpsqpl/lib/python3.9/site-packages/poetry/core/masonry/builders/builder.py", line 86, in __init__
    self._module = Module(
  File "/tmp/build-env-oxrpsqpl/lib/python3.9/site-packages/poetry/core/masonry/utils/module.py", line 71, in __init__
    raise ModuleOrPackageNotFound(
poetry.core.masonry.utils.module.ModuleOrPackageNotFound: No file/folder found for package template

ERROR Backend subproccess exited when trying to invoke build_sdist