lms24/jsr

JSR - The Java Test Suite Reduction Framework

jsr's Issues

Implement additional coverage metrics

• Statement Coverage
• Method Coverage
• Branch Coverage
• Path Coverage?
• Def-Use Coverage??

Create CLI Prototype

Just the basic functionality:

• a complete TSR run from start to finish
• TSR run with given coverage report

Generate XML based Checked Coverage report

Very handy to check and read checked coverage results

Clean up slicing files after a successful slicing round

To avoid endlessly growing output files

Core Library Implementation

Everything related to logic, behaviour, interactin w/ tools etc.

Improve Readme

Upgrade Project to Java 17

Afterr the involuntary downgrade to JDK11, I really want to get this project up to date to JDK17. Main Problem: SL4J compatibility.

• Figure out what exactly causes the SL4J compilation errors when using a JDK >11
• Fix it (submit PR to SL4J if successful)
• Upgrade JSR Project to JDK17
• Use new language features (String templates!!, pattern matching, etc)

Figure out how to universally create AG-jars w/ test classes

Create a JUnit 5 runner and let it be user-selectable which runner to execute

Implement Genetic TSR Algorithm

Refactor SFL Matrix Export

After the last meeting, a request was made to slightly change the FSL export behaviour:

• If a TC passes, the SFL A-matrix row should include the checked coverage records
• If a TC fails, the SFL A-matrix row should include the line coverage records

Tasks:

• Adapt this behaviour
• Remove SFL export during TSR run
• Integreate SFL Export into the CLI (makes more sense than bloating the IDE Plugin with it)

Add Delayed Greedy Algorithm

As discussed, another greedy algorithm should be added for better comparability, since the genetic algorithm isn't working particularily well. The decision was made to implement the delayed greedy algorithm heuristic as described by Tallaman and Gupta 2005.

• Implement a new Reduction Strategy, based on the paper
• Test correctness w/ example from paper and integration tests
• Adapt IDE Plugin to support the new strategy
• Adapt CLI to support the new strategy

CLI Implementation

Simple CLI to interact w/ the core library

Dynamic Slicing

For computing checked coverage, we have to create a dynamic slice using the tool Slicer4J.
After hours of figuring out how to work with this tool, we finally figured out how to compute the dynamic slice of a test case's statement.

We used the python script for it, as it provides a nice facade to hide the slicer's complexity. Especially for TC dynamic slicing this is extremely helpful.

Prerequisites

• The project for which the slice is calculated has to be compiled and a fat jar, containing all main and test classes plus additional libraries has to be created before calling the slicer. Since fat jars usually only contain main classes and their libraries, I shall therefore call this type of jar "extra fat jar" or for shits and giggles "Alfred Gusenbauer Jar" (AG-Jar).

To create such a jar, we used gradle for our projects. However, this is a cumbersome task, considering that gradle syntax seems to change weekly and almost no Stackoverflow et al. answers are usable without modifications. Therefore, we find our first challenge:

• #2

Executing the Dynamic Slicer

Now that we have our AG-jar, we need to call the slicer. Slicer4J works in three steps:

1. Instrument the Jar.
2. Execute the Jar (either via a TC/TM specification or via a main method spec) and collect the execution trace.
3. Create a backward slice from the execution trace based on supplied slicing criterion

As already stated, it's best to use the python script. It is called in the following way:

python3 slicer4j.py
-j <path-to-AG-jar>
-o <output-directory>
-b <slicing-criterion> 
-tc <full test class name>
-tm <test method name>

An example call from the Slicer4J/scripts directory:

python3 slicer4j.py
-j ../../../../master-utils/build/libs/testJar.jar
-o utils-out/
-b "at.tugraz.ist.stracke.jsr.CalculatorTest:39"
-tc "at.tugraz.ist.stracke.jsr.CalculatorTest"
-tm "divideZeroByNotZero"

Problems:

During compilation we observed some JDK level problems:

• The slicer and the slicing core have to be compiled with JDK-8 (at least on my system, else I get JAVA_HOME not set errors)

• The provided JUnit runner's and the test classes' JDK levels have to match and the SUT's syntax has to be <=JDK-9.

• Challenge 2: Ensure JDK compatibility levels

The provided JUnit runner is a problem in itself, as it only supports JUnit 4 (which is ancient IMO).

• #3

Result

The results from the slicer should be pretty straight forward to work with:

• slice.log is a log file containing the java statements that are in the slice. They are ordered by execution, not by backward slice order.
• slice-graph.pdf gives us a nice dot graph to manually inspect the slice
• trace.log_icdg.log contains the intermediate statements from the trace which is used for slicing

Debug log files:

• instr-debug.log contains code instrumentation output
• trace_full.log contains trace output
• slice-file.log contains slicing output
• graph-debug.log contains graph creation output

Overall conclusion

With a few challenges in mind, we can conclude that we can use Slicer4J for dynamic slicing. Thus, checked coverage can be computed. The main challenge will be to create a robust interaction between JSR and Slicer4J to ensure that the slicing is executed correctly and robustly against e.g. different JUnit versions, JDK versions, etc.

Create IntelliJ IDEA Plugin Prototype

IDE Plugin for IntelliJ IDEA

If possible, else resort to VSCode

Improve stability and usability of JSR as a library

The ultimate goal would be to improve JSR in terms of stability and usability to make it more comofortable to use it.

• Integrate Slicer4J into the project by removing the necessety to clone the repository and to build it plus the runner(s). As long as Sl4J is not a library, this means that we have to somehow include all necessary Jars from Sl4j in our project. This requires an updating mechanism of the jars but it is still much more stable and user friendly than expecting anyone to complete the tough install procedure that is currently necessary.
• Afterwards, reduce the number of user parameters (paths). We can easily drop the slicer path (as it would be directly included in the JSR Jars), possibly also the base backage parameter by finding the biggest common bases package when parsing the source code.
• Add options for a more step-by-step execution process by exporting every step's result (e.g. parsed representations of code/test sources are not persisted at the moment) and being able to continue where the user left off.

Evaluation Benchmarks

Perform Evaluation Benchmarks:

• At least 5 Open Source Projects
• Multiple coverage metrics
• Greedy HGS vs Genetic TSR algirthm
• Mutation Testing to evaluate FDC

IDE Plugin: give user the option to disable individual TCs

This could be done by implementing a context menu when right-clicking a TC item in the results list