testsuite-parallelization's People
testsuite-parallelization's Issues
Ensure "cpuness" statistics is consistent with io-bound subjects
reminder...
Comments on .tex file
It would be good if we could generate the .pdf
file with commentaries to share ideas, opinions, suggestions, and so on.
Search surefire reports recursively
During the execution of #7, I noticed some subjects does not necessarily output the XML reports in {BASE-DIR}/target/...
when the project is composed by multiple Maven modules.
The fix should use the tool find
to list all XML files and reuse the existing function over each path.
Validate experiment pipeline
reminder... check that everything is okay by running our scripts with a subset from our subjects: we don't want to waste time to spot problems and take action with our experiment.
Add support to timeouts in case of deadlocks
In the script find_parallel_failyres.py, I invoke test in parallel mode at line 27.
Some parallel configurations may lead to deadlocks on certain subjects. For instance, when using threads on GraphHopper, the executions hangs indefinitely while it finishes when using only the fork option.
There should be some mechanism to spawn this test invoke in a separated thread and wait for it until it finishes of until a given timeout occurs. In case of timeout, it would be possible to abort the current subject and let the execution flows to the following subjects.
Additionally, it is extremely important to have a deep understanding about what circumstances lead to these outcomes.
Create subset sample of relevant test for nightly execution
Create a CSV file with relevant subjects that can be executed while I back home.
It is important to keep the results from this subset to check if the output data is consistent over several executions.
Add more interesting subjects to the motivation experiment
Reminder:
- BerkeleyDB (..as mentioned by Marcelo)
- Vertx3-core: According to their CI server, it takes about 8min to run tests and it has a considerable number of tests. Plus: Maven build system
- Eclipse Collections. Should be an interesting subject as well
...Keep searching! ๐
Plot data from RQ1, RQ2, and RQ3
Check inconsistent subjects pointed by the Sanity script
Some executions seems to be aborted and others seems to be flaky.
Group them and investigate. In the worst-case scenario, we will have to eliminate these noises.
Use a better approach to find parallel flakiness
In the motivation-experiment
, I'm currently using a magic number to re-execute a subject's test suite to find tests that fail in some parallel execution. A smarter approach would be halting the re-execution after N
executions without any new failing test.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.