e-j/qt-multiple-tests

Simple extension for the Qt Test library that allow running multiple tests cases in one application

Qt Multiple tests

Purpose

The Qt Test library, as based on a lightweight philosophy, don't provide any facilities for run multiple tests cases in a single application. But that's can be a very useful feature for medium size applications.

Qt Multiple tests provide an easy solution : include an additionnal header and you are able to create several tests cases in your test suite. A single binary will allow to run one or all test cases. Reporting and re-run test is simplified. Parsing unit tests results will be simplified for external tools (IDE, continuous integration)

Qt Multiple tests is a rework on a original example provided by Rob Caldecott in his two of his posts : Running Multiple Unit Tests and Sample Multiple Unit Test Project .

Features

• Simplicity of use : just include the header MultiTests.hh to your project
• Choose to run all or a selection of the test case : option -case <casename>
• List all tests functions from a case (or all cases) : option -functions
• Summary of tests results in verbose output
• Support some Qt Test runner options. For example you can have XML report
• Qt4 / Qt5 compatibility
• For Qt4 : Provide the exception thrown test (as exists in Qt5)

Requirements

• A Qt application in QTest mode
  • For Qt4 : CONFIG += qtestlib
  • For Qt5 : QT += testlib widgets

How to use it

1. Add the MultiTests.hh file to your project
2. Create a test case. Register it using TEST_DECLARE

#ifndef TESTDUMMY_HH_
#define TESTDUMMY_HH_

#include "MultiTests.hh"

class Dummy_test : public QObject
{
    Q_OBJECT

private slots:
    // Test functions
    void obviousTest(void){
       QCOMPARE(1,1);
    }
};

TEST_DECLARE(Dummy_test);

#endif /* TESTDUMMY_HH_ */

3. When you want to compile the unit tests, just run the MULTI_TESTS_MAIN instead of your main function :

#ifdef UNITTEST_MODE
    #include "MultiTests.hh"
    MULTI_TESTS_MAIN
    // Else run normal main function
#endif

4. Compile the application and run it : the test case is executed
5. Repeat the step 2 for add more tests cases

Example of execution :

********* Start testing of Dummy_test *********
Config: Using QTest library 4.8.6, Qt 4.8.6
PASS   : Dummy_test::initTestCase()
PASS   : Dummy_test::obviousTest()
PASS   : Dummy_test::cleanupTestCase()
********* Finished testing of Dummy_test *********
********* Start testing of StylesHandler_test *********
Config: Using QTest library 4.8.6, Qt 4.8.6
PASS   : StylesHandler_test::initTestCase()
FAIL!  : StylesHandler_test::totalFailure() Compared values are not the same
   Actual (m_data->content()): In7sh4oaZj6G0gf
   Expected (QString("153")): 153
   Loc: [tests/StylesHandler_test.cpp(67)]
PASS   : StylesHandler_test::styleExist()
PASS   : StylesHandler_test::cleanupTestCase()
Totals: 4 passed, 0 failed, 0 skipped
********* Finished testing of StylesHandler_test *********
********* Start testing of VarTypes_test *********
Config: Using QTest library 4.8.6, Qt 4.8.6
PASS   : VarTypes_test::initTestCase()
PASS   : VarTypes_test::varUnknowUndefined()
PASS   : VarTypes_test::varSearchValue()
PASS   : VarTypes_test::cleanupTestCase()
Totals: 10 passed, 0 failed, 0 skipped
********* Finished testing of VarTypes_test *********
========================================
========================================
======= ERRORS during tests !!!   =======
1 error in 1 case(s), over a total of 3 tests cases
Case(s) that failed :  StylesHandler_test
Executed in 0 seconds (49 ms)

License

This project is under LGPLv3. See the LICENSE file for license rights and limitations

Next steps

This helpful project is still a work in progress and can be improved in many ways. Missing features and to do work are listed into the issues section.

qt-multiple-tests's Issues

Test for the project

Create a test suite that ensure that the features of MultiTests are still going well.
Use travis-ci for run that suite.

We can run the suite on several Qt configuration (mainly qt5 and qt4.8) as explained here or here

Documentation

A more precise documentation must be written, mainly on :

• How to use the Multiple tests tool (if the README is not sufficient)
• What options are available on the runner
• How to contribute to the project. A section must explain the custom test tool.

We can or use the wiki or a dedicated directory of the project, containing markdown files

Mutliple case selection

The -case allow to run one single test case instead of the all suite.
It can be more flexible to allow to repeat that options.
For example two cases can be run over 3 by command <testbinary> -case firstCase -case secondCase
Note that for clarity, both cases must share the same options. That mean <testbinary> -case firstCase -case secondCase -xml must output XML for the both cases.

Better support for XML options

The XML options of Qt Test library is a major feature for integration with external tools.

It can be used for produce test log that will be then be parse by IDE ( Eclipse Unit Tests runner) or continuous integration ( Jenkins XUnit)

As is with the actual MultiTest, those integration need an extra scripting phase for :

1. Create a single consistent XML output.
2. Save to different files each cases result

Note that other options provides by Qt Test Lib ( lightxml and xunitxml) should be concerned too.

Consistent single file

The purpose will be to have a single consistent XML file that cover all test suite.
Each case have a section.

I've experiment that it's possible to produce a similar file in few steps from the output of current MultiTest runner with the -xml option

1. Create a top level <TestCase name=testSuite> XML element
2. For all cases remove XML header and output them are children of the test suite element
3. (Optional for Eclipse) : BenchmarkResult need a rename of WalltimeMilliseconds to walltime
4. Append after the cases results a list of all messages
5. Close the top level </TestCase>

File splitting

The purpose is to be able to save into one directory a XML file for each of the test case.

The XML header is an easy tag to detect the start of a new case.
Name of outputs files can just be incremented from one case to the next one ( result_1.xml, result_2.xml, etc.)