Support latest Python version

[argoproj-labs]

Support latest Python version

[harelwa]

Hi,

pitching in with a bit of a different suggestion.

For CICD purposes I find Github Actions matrices more comfortable than tox.

Why ?

• You have control over build, not relying on a wrapped build ran by tox. I find this control is necessary, for when things fall behind in wrappers. e.g. -- you'll see deprecation warnings thrown by tox using long deprecated setuptools deprecation warning. considering the complexity of building wheels, i find the wrapper an overhead.
• You have an action per target build, which is comfortable for humans [ more below related to matrices ]
• this will play in the same manner for the cross of platforms ( linux, mac-os, win )

A good enough reference for that you can find in pydantic --

https://github.com/samuelcolvin/pydantic/blob/master/.github/workflows/ci.yml
https://github.com/samuelcolvin/pydantic/runs/4846228861?check_suite_focus=true

This approach actually also makes sense to me for local development --

• Having control of the build, install and test flow, separately from tox.
• Have that flow for python, agnostic to version and where it's py38, py38 will be used etc.

I'd love to hear what you think.

And either way, be happy to contribute.

Thanks,

Harel

[bobh66]

I like the idea of using both. I use tox extensively during development because I can run the full suite of tests locally against all of the supported python versions by just running one command. The bigger issue might be with Pipenv only wanting to support one python release, so Pipenv.lock can't be stored in git. I like the principles behind pipenv but the single-python-release requirement seems a bit unrealistic for most projects.

[flaviuvadan]

Hey @hirosassa! Thanks for the issue submission 🙂 Sure, this sounds great! I am curious, however, whether you foresee any potential future issue submissions that have to do with some potential backward incompatible changes? I have not looked at the release notes of 3.8 and I'd love to know whether you can share some potential drawbacks of this migration! My main concern is, to be perfectly honest, any syntax changes Hera might introduce, which can break existing application usage.

[bobh66]

@flaviuvadan - have you considered using tox to provide test environment support for multiple python releases? That would allow the test suite to run on every supported python release for every build and hopefully find any compatibility issues in the pipeline. I can push a PR if you're interested.

[hirosassa]

@flaviuvadan @bobh66 Thanks for the comments!

In my experiences, there's no significant changes (including syntax changes) in migration from Python 3.7 to 3.8+.
Only concern is dependent packages, but in Hera, dependencies are simple and all the dependencies supports Python 3.7+.
So this migration has no (or very small) concerns, IMO.

https://github.com/argoproj-labs/hera-workflows/blob/main/Pipfile
https://github.com/argoproj-labs/hera-workflows/blob/main/setup.py#L37-L44

[hirosassa]

As @bobh66 says, I also think using tox is the best way to support multiple versions of Python.

examples:

• https://github.com/spotify/luigi/blob/master/tox.ini (complex)
• https://github.com/m3dev/gokart/blob/master/tox.ini (simple)

[bobh66]

Also note tox-pipenv: https://github.com/tox-dev/tox-pipenv to allow for tox and pipenv to work together cleanly

[harelwa]

now i see that luigi is actually an example case where both of these approaches are used together !

https://github.com/spotify/luigi/blob/master/.github/workflows/pythonbuild.yml#L13

[hirosassa]

I agree with the idea of using both, too. The idea covers both running tests locally and CI environments.

The bigger issue might be with Pipenv only wanting to support one python release, so Pipenv.lock can't be stored in git.

Yeah, this is true. I'm currently using poetry for package management in my personal project, it supports multiple versions of Python releases.

[flaviuvadan]

This is a very valuable discussion 🙂 In light of the preferences outlined here, it seems like multiple local Python versions for Hera development is preferred, along with tox (to run tests on local using multiple environments), with the potential to transition Hera to poetry for package and version management?

[hirosassa]

@flaviuvadan I think it is right, but applying all of the solutions is not preferable for the developers and such a big change will cause bugs.
So my recommendation is supporting for multiple Python version using tox as the first step.

[bobh66]

I agree with @hirosassa - I guess the next question is do we want to continue using make/Makefile for now and just have the tox environments invoke the make commands? That would probably be the easiest first step forward. There is also the issue of the Pipenv.lock file being stored in git. I started to look at tox-pipenv but got distracted - I'll try to pick it up again and see how it works.