oscar-system/oscardevtools.jl

Tools for developing Oscar and for continuous integration

oscardevtools.jl's Issues

Pass arguments to the tests

julia >= 1.3 can pass arguments to the test process by specifying the test_args::Union{Cmd, Vector{String}} keyword arguments for Pkg.test.

It would be great if the configuration toml would have a way to specify the arguments for the tests of an individual package. E.g.

[testoptions]
Hecke = ["short"]

The reason is that the Hecke tests are way too long to be useful for the upstream package testing. So I would like to limit them.

I can have a stab at it if we agree that this is a useful feature.

If the current package is incompatible with other packages last release, don't treat this as faillure

[ We've discussed this before, elsewhere, I just wanted to note it down somewhere for easier reference -- so in particular I do not mean to say "why haven't you done this already" ! ]

Consider this scenario:

• The latest Oscar.jl release has compat 0.6 for GAP.jl
• in the meantime, the latest release of GAP.jl is 0.7, so it can't be loaded in Oscar
• now we run CI tests using OscarDevTools.jl for GAP.jl, which test its master branch against the latest Oscar.jl release
• of course these can't pass, as the latest Oscar.jl is incompatible with GAP.jl master (in fact, something even worse happens: an ancient Oscar.jl version 0.2.0 is installed by Julia.

Ideally, we'd detect this situation: when doing tests for package A against release versions of packages B, C, ..., if the current version of A is not compatible with the latest versions of B, C, ... then don't run any tests and just mark the job as success (after all, there is no incompatibility).

I think the somewhat hard part may be to prevent Julia from downgrading packages B, C, ... to make things work.

Also run doctests

We should also run doctests of the involved packages: we just had a case where the regular tests worked, but a PR broke doctests of an upstream project.

One thing to remember for that is that usually our doctests require a specific Julia version; but it should be possible to deal with that (and I think basically Julia 1.6 is currently used for this by all our packages?)

Run the tests separately

In AA, we test four packages (https://github.com/Nemocas/AbstractAlgebra.jl/blob/master/OscarCI.toml). Would it be possible to have the four tests in separate jobs? The problem we have at the moment is that the Oscar tests run first, but they might (and often do) fail for unrelated reasons. Then all the tests of the other packages are not run.

I am not sure we want to have separate jobs or handle it inside the julia command here: https://github.com/oscar-system/OscarDevTools.jl/blob/master/src/OscarCI.jl#L283

Thoughts?

Edit: Sorry, I just checked. The Oscar tests run last. But my problem is that I have to scroll through everything to spot which tests passed, which did not, and to find out which did not even run. I hope you get my point.

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Create a GitHub Action (or several) to simplify using this

Right now to use this we copy & past the oscar.yml file from here to each of our repositories and edit it as needed. Drawback is that tweaking anything in there now required updating several repositories. It can also be easy to miss difference between the different copies. And someone trying to understand the oscar.yml has a lot on their plate.

We could create a GitHub Action (or multiple) to factor out some or even all of the duplicate code. E.g. right now, I think this sequence is repeated twice in our actions:

      - name: re-using OscarDevTools checkout
         if: github.repository == 'oscar-system/OscarDevTools.jl'
         run: |
           julia --project=oscar-dev -e "using Pkg;
                                         Pkg.develop(PackageSpec(path=\".\"));
                                         Pkg.instantiate();"
       - name: fetch OscarDevTools
         if: github.repository != 'oscar-system/OscarDevTools.jl'
         run: |
           julia --project=oscar-dev -e "using Pkg;
                                         Pkg.add(PackageSpec(name=\"OscarDevTools\",version=\"0.2\"));
                                         Pkg.instantiate();"

Well, this could be put into an action oscar-system/install-devtools and then can be replaced by something like this:

       - uses: oscar-system/[email protected]

Writing a simply GitHub Action (which basically acts like a "macro" that contains normal GH Actions YML, with a few parameters), is actually relatively simple. We did it for GAP, see e.g. here: https://github.com/gap-actions/setup-gap-for-packages -- the only thing that really matters there is the action.yml which is mostly self-explanatory, I think. Of course adding a README, LICENSE, tests for the action we nice but not required. Nor is it required to put this into the marketplace.

Anyway, this is not urgent, I simply wanted to record this thought.

Cut down number of jobs in the presence of feature branches

Nemocas/AbstractAlgebra.jl#805 created 3^4 = 81 jobs, because there are four packages, each with master, latest and we also had a custom branch with the same name in every repository (on purpose).

I know there are different ways to handle this, but one thing that came to mind was that if I am working on package A on branch #bla, and dependency B has also a branch #bla, then restrict the version for testing B only to B#bla (or maybe B#bla and B#stable) and not to all three B#latest, B#master and #Bla.

This is just one suggestion. Happy to hear what you think about this.