RFC: Adopt new security model for publishing

[sethmlarson]

RFC: Adopt new security model for publishing

[ncoghlan]

The proposal makes sense to me.

I'd be happy to offer contextlib2 as a guinea pig (there are a couple of languishing requests to sync with a newer base version of contextlib, so it's definitely due to release a new version)

(Edit: I've had a productive couple of days, so the pending changes are ready to go. It's been 3 years since the last release though, so there's no problem waiting a few weeks to actually make the release. If the timeline is likely to be longer than that then I should probably just go ahead and release it the old way)

[hugovk]

I think this is an excellent idea!

For consistency and to facilitate future upgrades to the process I recommend standardizing on the names of release workflows definitions, GitHub teams and GitHub Environments (suggesting publish.yml (project)-team-leads, and publish are my bikeshed suggestions).

My empirically-backed bikeshed is release.yml:

• https://mastodon.social/@hugovk/112054931871821050

[hugovk]

maybe have a stage.yml or similar to push to the staging pypi (for the times we're not yet confident) ?

Yes, I usually push to TestPyPI for merges to main, and it works well in the same workflow, especially when using https://github.com/hynek/build-and-inspect-python-package.

For example:

• https://github.com/hugovk/norwegianblue/blob/main/.github/workflows/deploy.yml
• https://github.com/hugovk/norwegianblue/actions/runs/9089498511

[sethmlarson]

@hugovk Always good to have data, I've updated my proposal to match.

[blaisep]

(Edit: I've had a productive couple of days, so the pending changes are ready to go. It's been 3 years since the last release though, so there's no problem waiting a few weeks to actually make the release. If the timeline is likely to be longer than that then I should probably just go ahead and release it the old way)

Hi @ncoghlan , could we pair up? I'm interested in writing down the usual workflows to make it easier to onboard new roadies #366

btw, @sethmlarson , there is a lot of good work in here:
https://github.com/hynek/build-and-inspect-python-package/tree/main/.github/workflows

We could take it further and include some services to could give actionable insight into the condition and progress of a project. Tools like these have useful reporting options that can build confidence in a project with little toil:

• Allure HTML
• Bandit security scanning
• Wily code complexity
• Whatever the Python counterpart of the Oranda landing page generator is, although I hear from axodotdev that it will work with Python projects.

[Mogost]

I fully support this proposal. The reimagined security model using GitHub Actions and especially CODEOWNERS is a significant improvement that addresses current security concerns.

[blaisep]

I'm dropping this link here: https://github.com/hynek/build-and-inspect-python-package
because it aligns with our direction and I'll eventually weave it into #366 or a derivation.

[blaisep]

@hugovk

My empirically-backed bikeshed is release.yml:

* https://mastodon.social/@hugovk/112054931871821050

maybe have a stage.yml or similar to push to the staging pypi (for the times we're not yet confident) ?

[blaisep]

Another step which might please @sethmlarson is using hardened build images, like the wolfi-act
eg. https://github.com/wolfi-dev/wolfi-act/tree/main/.github/workflows

[webknjaz]

My empirically-backed bikeshed is release.yml:

FWIW, I'm in the camp of ci-cd.yml because I believe in testing the artifacts before upload, not merely rebuilding them with no testing in between.

There's also a problem with GitHub, suggesting a poorly made starter workflow that @woodruffw is trying to fix (including in the GitHub's Docs, which is slightly better).

Though, my guide https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/ suggests a simplified version because I didn't want to introduce too many things. Still, I use the build->test->(pause)->publish->update-git structure wherever I can.

What @jezdez set up in this org already requires reviewing the dists by forcing uploads to Jazzband's own intermediate server, not to PyPI: https://jazzband.co/about/releases. That server is what gates the process and pushes the approved dists to PyPI post-review.

[ncoghlan]

Hi @ncoghlan , could we pair up? I'm interested in writing down the usual workflows to make it easier to onboard new roadies

@blaisep As far as I'm aware, https://jazzband.co/about/releases#continuous-integration covers the existing release process, so I don't think there would be a lot to see in a pairing session (final review to check repo contents are ready for the release, make tag locally, push tag to Github). The private repo sends me a link once the staged package is ready for review, and that's currently a manual task to actually pull it down and test it (hence this issue).

@sethmlarson If the release process intrinsically grabbed the new sdist and wheels and used https://diffoscope.org/ to compare them against the previous sdist and wheels, that would be pretty helpful. I'd started experimenting with that idea in a previous role, and found it could be incredibly helpful with ensuring unexpected changes didn't creep into maintenance releases.