gap-actions/run-pkg-tests

run-pkg-tests's Issues

Add input that passes options to GAP

It's merely an idea/suggestion, but I think it would be useful to have the ability to pass options to GAP. In particular, the -A flag would (imho) be useful for testing if a package still works properly when loaded with the OnlyNeeded option. This would require a small change in pkg-ci-scripts/run_tests.sh as well, of course.

Test the `only-needed` input to this workflow in the CI

Currently only the NO_COVERAGE and GAP_TESTFILE inputs to this workflow are tested in the CI.

This action no longer works with GAP master branch and cygwin

See:

https://github.com/semigroups/Semigroups/actions/runs/7420584539/job/20192252720?pr=969

It looks like the branch cygwin-v2 hasn't be synced with v2, the diff contains (for example):

  ln -f -s $PWD /tmp/gaproot/pkg/

          # start GAP with custom GAP root, to ensure correct package version is loaded
-         GAP="$GAPROOT/gap -l /tmp/gaproot; --quitonbreak"
-
-         if ${{ inputs.only-needed }} = 'true' ; then
-           GAP="$GAP -A"
-         fi
+         GAP="$GAPROOT/bin/gap.sh -l /tmp/gaproot; --quitonbreak"

Document more precisely what `NO_COVERAGE` does

Currently:

NO_COVERAGE: Set to a non-empty string to suppress gathering coverage.

For people who already know what this means, I think this is probably clear enough. But for people who don't... such as me, I think this could be clearer.

Does it stop the profiling package from collecting coverage data on the GAP side of the package? Does it also stop the package's kernel module from having code coverage collected? If this variable is set, does gap-actions/process-coverage stop working, or at least redundant?

This issue was inspired by my confusion in gap-packages/profiling#94

Implement: Honour `TestFile` from `PackageInfo` record

As mentioned in the code:

run-pkg-tests/action.yml

Lines 43 to 45 in 2f2aa4f

	# TODO: honor TestFile from PackageInfo record, but make sure that it
	# is for the package in the current directory
	$GAP ${GAP_TESTFILE:-tst/testall.g}

Split coverage uploading (and gathering?) into separate action

People may want to use additional or alternative coverage tracking services, like coveralls, or simply not track coverage at all. So I think that uploading (and possibly gathering) of coverage should be in separate actions. Also, codecov has its own GitHub action for uploading coverage data, which one might want to use?

Of course we can't just remove them here, packages will need to adjust their CI.yml files -- but that's why actions can be versioned. So in v2 (or whatever) of this action, these steps can be removed.

The action should fail if the test file doesn't exist

Currently that's not happening, see e.g. https://github.com/gap-packages/anupq/runs/2315066518?check_suite_focus=true where the tests are 'passing' even though the test file can't be found.

Run tests three times: with default packages (as it does now), with all, with minimal set

.... to more closely mirror what ends up happening in the package distro.

For slow test suites, it should be possible to turn this off via a setting.

For the "test with all packages", it really would be important to use pre-made containers, otherwise not all packages will be available (namely those which involve compiled components)

Recursive folder structure may "spam" codecov action

In the action.yml, the following recursive folder structure is created:

mkdir -p gaproot/pkg/
ln -f -s $PWD gaproot/pkg/

However, v2 of codecov-action also looks for coverage reports in subfolders, causing it to find a lot of them (although it seems to automatically stop at 41). The most recent example in gap-packages at the time of writing:

[2022-03-18T13:22:23.874Z] ['info'] => Found 41 possible coverage files:
  gap-coverage.json
  gaproot/pkg/ClassicalMaximals/gap-coverage.json
  gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gap-coverage.json
  gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gap-coverage.json
[...]
  gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gap-coverage.json
[2022-03-18T13:22:23.875Z] ['info'] Processing /home/runner/work/ClassicalMaximals/ClassicalMaximals/gap-coverage.json...
[2022-03-18T13:22:23.880Z] ['info'] Processing /home/runner/work/ClassicalMaximals/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gap-coverage.json...
[2022-03-18T13:22:23.880Z] ['info'] Processing /home/runner/work/ClassicalMaximals/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gap-coverage.json...
[2022-03-18T13:22:23.881Z] ['info'] Processing /home/runner/work/ClassicalMaximals/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gap-coverage.json...
[...]
[2022-03-18T13:22:23.898Z] ['info'] Processing /home/runner/work/ClassicalMaximals/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gaproot/pkg/ClassicalMaximals/gap-coverage.json...

It doesn't seem to cause any actual problems, and the coverage report works fine as well, but (to me, at least) it doesn't seem like this is intended behaviour.

This can probably be "fixed" using the "files" or "directory" flag for codecov-action, but that would imply updating every package's CI.yml, and that's treating the symptoms rather than the cause.