Allow specific tests or groups of tests to be run selectively

[arduino-ci]

Allow specific tests or groups of tests to be run selectively

[ianfixes]

• all tests in a file
  bundle exec arduino_ci_remote.rb -file interesting-tests-file
• multiples and combinations of each
  bundle exec arduino_ci_remote.rb -file test-file -test some-test -test some-other-test

This is implemented in #84 as:

• whitelisting all tests in a file
  • bundle exec arduino_ci_remote.rb --testfile-select=interesting-tests-file
• blacklisting all tests in a file
  • bundle exec arduino_ci_remote.rb --testfile-reject=boring-tests-file
• multiples and combinations of each (as globs)
  • bundle exec arduino_ci_remote.rb --testfile-select=tests-interesting*.cpp --testfile-reject=tests-interesting-but-not-that-interesting.cpp

[ianfixes]

Fix is live in 0.1.17

[scls19fr]

What about making a Ruby script directly inspired by py.test or nosetests which is able to automatically discover unit tests (ie files whith name starting with test_...
After installing this Ruby package, this script could be available from any directory.

[scls19fr]

https://docs.pytest.org/en/latest/usage.html

[ianfixes]

I enable targeting (somewhat) with the configuration:

unittest:
  testfiles:
    select:
      - "*-*.*"
    reject:
      - "sam-squamsh.*"

So what's involved here is just to let the file matching come from the command line as well.

Unfortunately, I'm not sure about selecting individual tests. Those come from macros within C++, so ruby has no knowledge of what they are and no ability to control which tests run when the executable runs.

So this fits into the larger idea of "have a way for the C++ test code to communicate (hopefully bidirectionally) with the Ruby runner", and I'd like to have a bigger, more future-proof plan in mind before I go down that path. (For example, I could push up all the reporting into Ruby instead of trying to make all the console logs look pretty even though they are coming from a mix of different binaries)

TL;DR in the short term I'd make use of an overriding .arduino-ci.yml file in your test/ project directory to target just the file you're working on.

unittest:
  testfiles:
    select:
      - "file-im-working-on-right-now.cpp"

[ianfixes]

@scls19fr did that workaround serve your needs? I'm trying to figure out what areas of this project to focus on during 2019.

[scls19fr]

I think this idea could fit most of usecases but we can't really be sure of it until trying with many libraries...

Maybe a default .arduino-ci.yml like

unittest:
  testfiles:
    select:
      - "test_*.cpp"

could be used

I think we should have a look in http://downloads.arduino.cc/libraries/library_index.json for libraries that could be tested and advocate in arduino/Arduino#7567 for better testing of Arduino libraries.

Maybe people like @per1234 @mmurdoch @toddtreece @ladyada @deanm1278 @cmaglie could bring their opinion about such a global effort to insure quality of Arduino libraries.

I really think this can only come with a collective effort... maybe a new year resolution?

Best wishes!

[scls19fr]

Being able to use command line parameters to overhide config file could help a lot

[ianfixes]

My apologies, #82 has been going on all this time.

In the short term I can provide command-line overrides of files, but overriding tests will require some additional effort (since Ruby can't know the test names until they get executed). It will require some as-yet-unimplemented glue between the Ruby and compiled C++ environments, and I've opened #83 to track that.

If an inability to specify unit tests by name (within a given test.cpp file) is still a problem, please open a separate issue for that.

[scls19fr]

Thanks @ianfixes. I can perfectly use one file per unit test as a temporary workaround. Being able to run just one test file was the most important feature I was looking for because it should drastically speedup my development (not having to wait all project tests are running fine).