Visualise tests whose state changes most often. During software development, it is often common that some tests start to randomly fail,
but finding those tests is a tedious and time consuming problem. Flaky tests detection solves that problem by processing historical xunit test
results and cheks which tests changes state most often. Flaky tests detection is available as Github Action plugin and
Python package. For usage, see example
at the actions
page.
Implementation is based on "Modeling and ranking flaky tests at Apple" by Kowalczyk, Emily & Nair, Karan & Gao, Zebao & Silberstein, Leo & Long, Teng & Memon, Atif.
- Prints out top test names and their latest calculation window scores (normal fliprate and exponentially weighted moving average fliprate that take previous calculation windows into account).
- Calculation grouping options:
n
days.n
runs.
- Heatmap visualization of the scores and history.
--test-history-csv
- Give a path to a test history csv file which includes three fields:
timestamp
,test_identifier
andtest_status
.
- Give a path to a test history csv file which includes three fields:
--junit-files
- Give a path to a folder with
JUnit
test results.
- Give a path to a folder with
-
--grouping-option
days
to usen
days for fliprate calculation windows.runs
to usen
runs for fliprate calculation windows.
-
--window-size
- Fliprate calculation window size
n
.
- Fliprate calculation window size
-
--window-count
- History size for exponentially weighted moving average calculations.
-
--top-n
- How many top highest scoring tests to print out.
--heatmap
- Turn heatmap generation on.
- Two pictures generated: normal fliprate and exponentially weighted moving average fliprate score.
- Same parameters used as with the printed statistics.
- Precomputed
test_history.csv
with daily calulations. 1 day windows, 7 day history and 5 tests printed out.--test-history-csv=example_history/test_history.csv --grouping-option=days --window-size=1 --window-count=7 --top-n=5
JUnit
files with calculations per 5 runs. 15 runs history and 5 tests printed out.--junit-files=example_history/junit_files --grouping-option=runs --window-size=5 --window-count=3 --top-n=5
- Precomputed
test_history.csv
with daily calculations and heatmap generation. 1 day windows, 7 day history and 50 tests printed and generated to heatmaps.--test-history-csv=example_history/test_history.csv --grouping-option=days --window-size=1 --window-count=7 --top-n=50 --heatmap
make install
make install_dev
make run_test
The package was developed by F-Secure Corporation and University of Helsinki in the scope of IVVES project. This work was labelled by ITEA3 and funded by local authorities under grant agreement “ITEA-2019-18022-IVVES”