e-j/qt-multiple-tests

Simple extension for the Qt Test library that allow running multiple tests cases in one application

qt-multiple-tests's Issues

Mutliple case selection

The -case allow to run one single test case instead of the all suite.
It can be more flexible to allow to repeat that options.
For example two cases can be run over 3 by command <testbinary> -case firstCase -case secondCase
Note that for clarity, both cases must share the same options. That mean <testbinary> -case firstCase -case secondCase -xml must output XML for the both cases.

Better support for XML options

The XML options of Qt Test library is a major feature for integration with external tools.

It can be used for produce test log that will be then be parse by IDE ( Eclipse Unit Tests runner) or continuous integration ( Jenkins XUnit)

As is with the actual MultiTest, those integration need an extra scripting phase for :

1. Create a single consistent XML output.
2. Save to different files each cases result

Note that other options provides by Qt Test Lib ( lightxml and xunitxml) should be concerned too.

Consistent single file

The purpose will be to have a single consistent XML file that cover all test suite.
Each case have a section.

I've experiment that it's possible to produce a similar file in few steps from the output of current MultiTest runner with the -xml option

1. Create a top level <TestCase name=testSuite> XML element
2. For all cases remove XML header and output them are children of the test suite element
3. (Optional for Eclipse) : BenchmarkResult need a rename of WalltimeMilliseconds to walltime
4. Append after the cases results a list of all messages
5. Close the top level </TestCase>

File splitting

The purpose is to be able to save into one directory a XML file for each of the test case.

The XML header is an easy tag to detect the start of a new case.
Name of outputs files can just be incremented from one case to the next one ( result_1.xml, result_2.xml, etc.)

Test for the project

Create a test suite that ensure that the features of MultiTests are still going well.
Use travis-ci for run that suite.

We can run the suite on several Qt configuration (mainly qt5 and qt4.8) as explained here or here

Documentation

A more precise documentation must be written, mainly on :

• How to use the Multiple tests tool (if the README is not sufficient)
• What options are available on the runner
• How to contribute to the project. A section must explain the custom test tool.

We can or use the wiki or a dedicated directory of the project, containing markdown files