damorim/testsuite-parallelization

testsuite-parallelization's Issues

Ensure "cpuness" statistics is consistent with io-bound subjects

reminder...

Comments on .tex file

It would be good if we could generate the .pdf file with commentaries to share ideas, opinions, suggestions, and so on.

Search surefire reports recursively

During the execution of #7, I noticed some subjects does not necessarily output the XML reports in {BASE-DIR}/target/... when the project is composed by multiple Maven modules.

The fix should use the tool find to list all XML files and reuse the existing function over each path.

Validate experiment pipeline

reminder... check that everything is okay by running our scripts with a subset from our subjects: we don't want to waste time to spot problems and take action with our experiment.

Add support to timeouts in case of deadlocks

In the script find_parallel_failyres.py, I invoke test in parallel mode at line 27.

Some parallel configurations may lead to deadlocks on certain subjects. For instance, when using threads on GraphHopper, the executions hangs indefinitely while it finishes when using only the fork option.

There should be some mechanism to spawn this test invoke in a separated thread and wait for it until it finishes of until a given timeout occurs. In case of timeout, it would be possible to abort the current subject and let the execution flows to the following subjects.

Additionally, it is extremely important to have a deep understanding about what circumstances lead to these outcomes.

Create subset sample of relevant test for nightly execution

Create a CSV file with relevant subjects that can be executed while I back home.
It is important to keep the results from this subset to check if the output data is consistent over several executions.

Add more interesting subjects to the motivation experiment

Reminder:

• BerkeleyDB (..as mentioned by Marcelo)
• Vertx3-core: According to their CI server, it takes about 8min to run tests and it has a considerable number of tests. Plus: Maven build system
• Eclipse Collections. Should be an interesting subject as well

...Keep searching! 🏃

Plot data from RQ1, RQ2, and RQ3

Once #6 and #7 are solved, we are good to run our experiment to generate the raw data.
Once the raw data is available, use the existing R scripts to plot data.

Check inconsistent subjects pointed by the Sanity script

Some executions seems to be aborted and others seems to be flaky.
Group them and investigate. In the worst-case scenario, we will have to eliminate these noises.

Use a better approach to find parallel flakiness

In the motivation-experiment, I'm currently using a magic number to re-execute a subject's test suite to find tests that fail in some parallel execution. A smarter approach would be halting the re-execution after N executions without any new failing test.