CI wrongly considers `noxfile.py` is designed for all sessions to pass

[pypa]

CI wrongly considers `noxfile.py` is designed for all sessions to pass

[abravalheri]

Umm, I don't know if I understand the problem.

Let's suppose that in the CI, we use setup-python to install in the same CI job worker, Python versions: 3.8, 3.10 and 3.12. To ensure noxfile.py works well, then we have to set the environment variable the environment variable NOXFILE_PYTHON_VERSIONS="3.8 3.10 3.12" before running nox. This way it should not matter (🤞) which Python we use to install nox, for each different section nox should take care of spawning the correct virtual environments and combining then consistently when reporting.

Now if contributors want to update the table.md file, in their personal computer they will also need to install the same Python versions used in the CI and set the environment variable NOXFILE_PYTHON_VERSIONS="3.8 3.10 3.12" before using the procedure described in the README.md.

This is how I suppose it would work, but the best would be actually check that against the github CI. Maybe other unforeseen errors will appear...

If that is too complicated, we can also drop all togheter this idea of checking if table.md is updated accordingly... Just pick the combinations of Python versions and nox sessions that we know should work and only run those...

[chrysle]

Sounds good! I'll take a look at that.

[abravalheri]

This might be useful in the context of running the whole nox matrix to generate the complete compatibility report:

https://github.com/actions/setup-python/releases/tag/v4.4.0

[chrysle]

fail if there is any uncommitted alteration in that file specifically

One concern I have is that there might be different results across different Python versions (that's what this repository is intended to determine, right), for which the user can't possibly determine the compatibility, thus running the whole workflow with different Python versions on a pull request will most probably result in a CI failure?

[abravalheri]

Hi @chrysle, I believe the best is to run all the Python versions from a since CI job.
Probably https://github.com/actions/setup-python/releases/tag/v4.4.0 (see release notes) can be used for that.

Otherwise, it would be necessary to find out a way of integrating/merging the report from different jobs, before converting them to a table and deciding if whole CI workflow passes or not. Individually running nox for a single Python version is not really useful.

This happens because the noxfile.py was designed to run all the sessions (with the different Python versions) at once and only at the end of all sessions, generate a report file (which can then be processed by the script that creates the table).

I don't think this fits in the idea of mapping each Python version to a different CI job (running in a different container/VM).

[chrysle]

I believe the best is to run all the Python versions from a since CI job

I agree with that. But if I've understood correctly, the different table reports for the Python versions will be combined into one compatibility matrix. And if running the nox sessions results with, for example, 3.10, in a different table than with 3.8,
which the user employed to update the table, won't the CI process on a PR fail?

[chrysle]

Thanks for your patience, I suppose I had a little short-circuit here. I'll look into implementing that soon.

[abravalheri]

Don't worry, me too to be honest. The way the noxfile.py is written and how the report is generated is a bit cumbersome to translate to Github CI. I scratched my head many times already about this 😆.

[abravalheri]

@chrysle this is a quick test I did to see if a single job can run multiple nox sessions:

https://github.com/abravalheri/experiment-setup-python-multiple-versions/actions/runs/7891544198/job/21536102814#step:6:32

I used a .python-version to maintain a single source of truth for both Github Actions and nox. (.python-version was just an arbitrary name that can be chosen differently).

(Although usually GHA can read from .python-version automatically, it seems that you need to manually set that when you are attempting to use multiple python versions -- see actions/setup-python#734).

[chrysle]

@chrysle this is a quick test I did to see if a single job can run multiple nox sessions:

Thank you @abravalheri, that is really useful!

I used a .python-version to maintain a single source of truth for both Github Actions and nox. (.python-version was just an arbitrary name that can be chosen differently).

Good idea! Actually, we might rename the file since it would interfere with customization by pyenv users?

[abravalheri]

Yeap, there should be no problem with that.