rjust / defects4j Goto Github PK
View Code? Open in Web Editor NEWA Database of Real Faults and an Experimental Infrastructure to Enable Controlled Experiments in Software Engineering Research
License: MIT License
A Database of Real Faults and an Experimental Infrastructure to Enable Controlled Experiments in Software Engineering Research
License: MIT License
Hi, I made some edits in the checkout repo of chart 22 fixed version, and after that I tried to generate tests cases using evosuite. However, I found this strange error
about Class not Found Unsupported major.minor version 52.0
Given that the repo is in git I just check out the original version c52ceb271453b43e8846fffae3f086d581acd414
But still getting the same error. Do you have any clue?
- EvoSuite
- Going to generate test cases for class: org.jfree.data.time.DynamicTimeSeriesCollection
- Starting client
- Connecting to master process on port 9541
- Analyzing classpath:
- /home/chart_22f/build
- /home/chart_22f/lib/servlet.jar
[MASTER] 14:59:01.917 [logback-2] ERROR TestClusterGenerator - Problem for org.jfree.data.time.DynamicTimeSeriesCollection. Class not found
java.lang.ClassNotFoundException: org/jfree/data/time/Second : Unsupported major.minor version 52.0
at org.evosuite.instrumentation.InstrumentingClassLoader.instrumentClass(InstrumentingClassLoader.java:155) ~[evosuite-0.2.0.jar:na]
at org.evosuite.instrumentation.InstrumentingClassLoader.loadClass(InstrumentingClassLoader.java:125) ~[evosuite-0.2.0.jar:na]
at org.evosuite.setup.TestClusterGenerator.addCastClassDependencyIfAccessible(TestClusterGenerator.java:265) [evosuite-0.2.0.jar:na]
at org.evosuite.setup.TestClusterGenerator.handleCastClasses(TestClusterGenerator.java:204) [evosuite-0.2.0.jar:na]
at org.evosuite.setup.TestClusterGenerator.generateCluster(TestClusterGenerator.java:170) [evosuite-0.2.0.jar:na]
at org.evosuite.setup.DependencyAnalysis.analyze(DependencyAnalysis.java:116) [evosuite-0.2.0.jar:na]
at org.evosuite.TestSuiteGenerator.generateTestSuite(TestSuiteGenerator.java:221) [evosuite-0.2.0.jar:na]
at org.evosuite.rmi.service.ClientNodeImpl$1.run(ClientNodeImpl.java:125) [evosuite-0.2.0.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
Caused by: java.lang.UnsupportedClassVersionError: org/jfree/data/time/Second : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.7.0_75]
at java.lang.ClassLoader.defineClass(ClassLoader.java:800) ~[na:1.7.0_75]
at java.lang.ClassLoader.defineClass(ClassLoader.java:643) ~[na:1.7.0_75]
at org.evosuite.instrumentation.InstrumentingClassLoader.instrumentClass(InstrumentingClassLoader.java:148) ~[evosuite-0.2.0.jar:na]
... 12 common frames omitted
Hi,
Some bug (at least one C8) have the failing tests that depend on the environment.
For example, the C8 bugs does not have failing test if the time-zone of the machine is equivalent to "Europe/Copenhagen".
Can you list the environment in which Defects4J has been made?
Thanks
Hi again :)
When running the run_evosuite.pl
script, it exits with the following error message (I found this bug for Lang-11f, Closure-74f, Closure-75f, etc.):
Can't "next" outside a loop block at /home/donghwan/defects4j/framework/bin/run_evosuite.pl line 233.
The problematic code is:
(run_evosuite.pl: lines around 230~234)
230: my $archive = "$PID-$VID-evosuite-$CRITERION.$TID.tar.bz2";
231: if (system("tar -cjf $TMP_DIR/$archive -C $TMP_DIR/evosuite-$CRITERION/ .") != 0) {
232: $LOG->log_msg("Error: cannot compress test suites!");
233: next;
234: }
As I guess, Evosuite failed generating tests because of the OutOfMemory exception (as described below), and so the run_evosuite.pl
script goes into the if
statement.
The log said:
* EvoSuite
* Going to generate test cases for class: org.apache.commons.lang3.RandomStringUtils
* Starting client
* Connecting to master process on port 9060
* Analyzing classpath:
- /tmp/run_evosuite.pl_27319_1450095235/target/classes
* Finished analyzing classpath
* Generating tests for class org.apache.commons.lang3.RandomStringUtils
* Test criterion:
- Branch Coverage
* Setting up search algorithm for whole suite generation
* Total number of test goals: 65
* Using seed 1450095246036
* Starting evolution
* Search finished after 214s and 239 generations, 96727 statements, best individual has fitness: 3.499862505156057
* Minimizing test suite
[MASTER] 21:19:58.937 [logback-1] WARN TimeController - Phase MINIMIZATION lasted too long, 63 seconds more than allowed.
[MASTER] 21:19:50.786 [logback-2] ERROR ClientNodeImpl - Error when generating tests for: org.apache.commons.lang3.RandomStringUtils with seed 1450095246036. Configuration id : null
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:2367) ~[na:1.7.0_91]
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) ~[na:1.7.0_91]
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) ~[na:1.7.0_91]
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) ~[na:1.7.0_91]
at java.lang.StringBuilder.append(StringBuilder.java:132) ~[na:1.7.0_91]
at org.evosuite.coverage.branch.ControlFlowDistanceCalculator.getRootDistance(ControlFlowDistanceCalculator.java:186)
So, I just removed next;
at line 233, and add -XX:-UseGCOverheadLimit
option for run_evosuite()
in the Project.pm
file. After this, there is no error.
Please check the potential bug and give me some feedback whether my temporal solution is problematic or not.
Thank you.
Donghwan Shin
According to the paper "Do Automatically Generated Unit Tests Find Real Faults?" bug 22 of Chart was detected in the 10 runs for all the tools. However, I've ran Evosuite 11 times, and any of the iterations has failed.
defects4j test -w /home-students/moar82/chart_22b -s >~/evosuite_chart_22b/Chart/evosuite-branch/11/Chart-22b-evosuite-branch.11.tar.bz2
Extract test suite......................................................... OK
Running ant (compile.gen.tests)............................................ OK
Running ant (run.gen.tests)................................................ OK
Failing tests: 0
Is that correct?
Defects4J currently uses CSV files as DB backend, which are not adequate for all purposes. DBI supports various DB backends, but dbi:csv is currently hardcoded.
Also, we should eliminate all textual access to CSV files and make the whole framework DBI driven.
Could you provide an example of test case execution using defects4j? I tried using the script framework/bin/run_mutation.pl ... but doesn´t work.. Could you help me use the defect4j for test case execution ?
Currently, the SingleTestRunner only supports the execution of a single test method -- it throws a NPE if the method name is omitted.
Hi @rjust,
I found that defects4j mutation -r
fails for Chart-10f
.
It returns the following error message:
donghwan@donghwan-ELOS:/tmp/Chart-10f$ defects4j mutation -r
Running ant (mutate)....................................................... OK
Cannot mutate project! at /home/donghwan/defects4j/framework/bin/d4j/d4j-mutation line 146.
Compilation failed in require at /home/donghwan/defects4j/framework/bin/defects4j line 155.
I think it fails to mutate the target class, but I do not know why.
Except this case, it is okay for all Chart, Lang, Math, Time, and Closure faults.
Is this problem only for me?
For your information, my machine uses,
$ java -version
java version "1.7.0_111"
OpenJDK Runtime Environment (IcedTea 2.6.7) (7u111-2.6.7-0ubuntu0.14.04.3)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)
$ perl -version
This is perl 5, version 18, subversion 2 (v5.18.2) built for x86_64-linux-gnu-thread-multi
Many thanks,
Donghwan
For each real fault, Defects4J should provide the corresponding bug-tracker ID. While the bug-tracker IDs are temporarily known during the bug-mining workflow, they are not stored in the database.
Hi René,
it seems that export -p cp.test
command is not working for Mockito bugs. compileLib directory (which is only created when test cases are compiled) does not exist after checking out a Mockito bug.
steps to reproduce the issue:
$ $D4J_HOME/framework/bin/defects4j checkout -p Mockito -v 1b -w Mockito-1b
$ cd Mockito-1b
$ $D4J_HOME/framework/bin/defects4j export -p cp.test
and the error message:
BUILD FAILED
~/defects4j/framework/projects/defects4j.export.xml:80: /tmp/Mockito-1b/compileLib does not exist.
Hi,
compiling is not my favourite activity but i'm trying to compile and run tests of one chart bug (c1) with new generated tests and one new library. As there is no build.xml file in the bug generated folder, is there any way for me to add my lib and execute defects4j compile without modifing Charts.build.xml ?
Hi @rjust,
I found a bug in defects4j.build.xml, especially for target name="update.all.tests"
. Because of the bug, defects4j mutation
and defects4j mutation -r
work exactly the same way.
The current version is (https://github.com/rjust/defects4j/blame/master/framework/projects/defects4j.build.xml#L346)
<target name="update.all.tests" if="d4j.relevant.tests.only">
and it must be fixed as follows
<target name="update.all.tests" if="${d4j.relevant.tests.only}">
Thanks,
Donghwan
Hi, my mentor gives me defects4j as a course project, he wants me to add more bugs in this tool. Could you give me some advice? I'll appreciate it.
For example, If I'd like to add new programs to 'Math', what should I do? Should I build a new local git repository and change the corresponding pm files? Could you explain the implementation details about it?
Thanks,
Shawn
After succesfully checkout and compiling the source code I have the following problem with the compilation of the unit test.
defects4j test
Buildfile: /home/moar82/r/xx/defects4j/framework/projects/defects4j.build.xml
init:
[echo] -------- commons-lang3 3.2-SNAPSHOT --------compile:
compile.tests:
[javac] Compiling 42 source files to /home/moar82/r/xx/lang_1_buggy/target/tests
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.5
[javac] warning: [options] source value 1.5 is obsolete and will be removed in a future release
[javac] warning: [options] target value 1.5 is obsolete and will be removed in a future release
[javac] warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
[javac] /home/moar82/r/xx/lang_1_buggy/src/test/java/org/apache/commons/lang3/reflect/TypeUtilsTest.java:524: error: incompatible types: inferred type does not conform to upper bound(s)
[javac] final Integer max = TypeUtilsTest.stub();
[javac] ^
[javac] inferred: G
[javac] upper bound(s): Comparable,Integer
[javac] where G is a type-variable:
[javac] G extends Integer
[javac] 1 error
[javac] 4 warningsBUILD FAILED
/home/moar82/r/xx/lang_1_buggy/build.xml:71: Compile failed; see the compiler error output for details.Total time: 14 seconds
Any suggestion?
Currently, the bug-mining scripts work top down (starting with the HEAD revision); they should work bottom up instead.
Hi,
After the Randoop version is upgraded (from 2.0.1 to 2.1.0, I guess), the same type of crashes randomly occurs for various subject programs.
Could you please check this issue? Thank you! I captured one of it as follows:
donghwan@donghwan-ELOS:~/Dropbox/mutationExpr/scripts$ run_randoop.pl -p Closure -v 6f -n 1 -o ../defects4j.testSuites/ -b 180
Check out 9f7a3533 to /tmp/run_randoop.pl_31036_1454493387................. OK
Init local repository...................................................... OK
Tag post-fix revision...................................................... OK
Run post-checkout hook..................................................... OK
Initialize fixed program version........................................... OK
Apply patch................................................................ OK
Initialize buggy program version........................................... OK
Diff 9f7a3533:929b1c8f..................................................... OK
Apply patch................................................................ OK
Tag pre-fix revision....................................................... OK
Check out program version: Closure-6f...................................... OK
Running ant (compile)...................................................... OK
Running ant (export.cp.compile)............................................ OK
Run Randoop (/home/donghwan/defects4j/framework/util/randoop.config)....... FAIL
policy = sun.security.provider.PolicyFile@4aafaa85
Ignoring non-visible class com.google.javascript.jscomp.MemoizedScopeCreator specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.ConstCheck specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.ProcessTweaks specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.LinkedFlowScope specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.jscomp.PassConfig specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.type.ReverseAbstractInterpreter specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.ProcessClosurePrimitives specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.rhino.jstype.PrototypeObjectType specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.TypeInference specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.JoinOp specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.jscomp.JsMessageVisitor specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.StrictModeCheck specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.jscomp.parsing.TypeSafeDispatcher specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.HotSwapCompilerPass specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.jscomp.graph.DiGraph specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.CheckGlobalThis specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.SourceAst specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.rhino.jstype.ObjectType specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.parsing.JsDocToken specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.CheckUnreachableCode specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.CleanupPasses specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.ScopeCreator specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.graph.LatticeElement specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.CheckProvides specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.deps.DependencyInfo specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.FunctionTypeBuilder specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.VariableReferenceCheck specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.type.FlowScope specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.graph.GraphvizGraph specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.jscomp.AbstractMessageFormatter specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.rhino.jstype.JSType specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.Region specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.rhino.jstype.UnresolvedTypeExpression specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.graph.SubGraph specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.ControlFlowGraph specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.jscomp.AbstractCompiler specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.SuppressDocWarningsGuard specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.CssRenamingMap specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.ControlFlowAnalysis specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.rhino.jstype.UnionTypeBuilder specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.rhino.jstype.ValueType specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.ProcessDefines specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.rhino.ErrorReporter specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.CheckGlobalNames specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.rhino.jstype.StaticScope specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.graph.Annotatable specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.InferJSDocInfo specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.TypeValidator specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.rhino.jstype.NamedType specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.rhino.jstype.StaticSourceFile specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.rhino.jstype.StaticSlot specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.DisambiguateProperties specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.jscomp.DataFlowAnalysis specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.CheckAccessControls specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.CheckDebuggerStatement specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.rhino.jstype.NoResolvedType specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.ControlStructureCheck specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.CodingConvention specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.rhino.jstype.StaticSymbolTable specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.parsing.JsDocTokenStream specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.jscomp.PassFactory specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.rhino.jstype.InstanceObjectType specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.VarCheck specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.rhino.SourcePosition specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.rhino.jstype.StaticReference specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.ReferenceCollectingCallback specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.rhino.jstype.EquivalenceMethod specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.MessageFormatter specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.rhino.jstype.IndexedType specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.RhinoErrorReporter specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.graph.GraphNode specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.TypedScopeCreator specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.SyntacticScopeCreator specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.parsing.Annotation specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.CompilerPass specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.TypeInferencePass specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.rhino.jstype.ErrorFunctionType specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.rhino.jstype.ArrowType specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.rhino.jstype.ProxyObjectType specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.jscomp.BasicErrorManager specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.PrepareAst specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.SourceExcerptProvider specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.graph.Annotation specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.jscomp.type.ChainableReverseAbstractInterpreter specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.rhino.jstype.Visitor specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.ErrorHandler specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.CheckSideEffects specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.CheckRegExp specified via --classlist or --testclass.
Ignoring non-visible class com.google.javascript.jscomp.parsing.IRFactory specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.graph.AdjacencyGraph specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.jscomp.WarningsGuard specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.ErrorManager specified via --classlist or --testclass.
Ignoring abstract class com.google.javascript.jscomp.graph.Graph specified via --classlist or --testclass.
Ignoring abstract interface com.google.javascript.jscomp.CodeChangeHandler specified via --classlist or --testclass.
PUBLIC MEMBERS=2054
Explorer = randoop.sequence.ForwardGenerator@103b813c
Progress update: test inputs generated=0, failing inputs=0 (Wed Feb 03 01:56:42 PST 2016)
Progress update: test inputs generated=465, failing inputs=1 (Wed Feb 03 01:56:47 PST 2016)Throwable thrown while handling command:java.lang.Error: Exception thrown before end of sequence
java.lang.Error: Exception thrown before end of sequence
at randoop.test.ValidityCheckingVisitor.visit(ValidityCheckingVisitor.java:42)
at randoop.test.ExtendGenerator.visit(ExtendGenerator.java:39)
at randoop.test.ExtendGenerator.visit(ExtendGenerator.java:39)
at randoop.sequence.ExecutableSequence.execute(ExecutableSequence.java:334)
at randoop.sequence.ExecutableSequence.execute(ExecutableSequence.java:263)
at randoop.sequence.ForwardGenerator.step(ForwardGenerator.java:129)
at randoop.sequence.AbstractGenerator.explore(AbstractGenerator.java:299)
at randoop.main.GenTests.handle(GenTests.java:409)
at randoop.main.Main.nonStaticMain(Main.java:77)
at randoop.main.Main.main(Main.java:39)
Randoop failed.
Last sequence under execution:diagnosticGroup0 = getter : <get>(com.google.javascript.jscomp.DiagnosticGroup:com.google.javascript.jscomp.DiagnosticGroups.UNKNOWN_DEFINES) :
diagnosticGroup_array1 = array : com.google.javascript.jscomp.DiagnosticGroup[1] : diagnosticGroup0
diagnosticGroup2 = cons : com.google.javascript.jscomp.DiagnosticGroup.<init>([Lcom.google.javascript.jscomp.DiagnosticGroup;) : diagnosticGroup_array1
void3 = setter : <set>(com.google.javascript.jscomp.DiagnosticGroup:com.google.javascript.jscomp.DiagnosticGroups.DUPLICATE_MESSAGE) : diagnosticGroup2
checkLevel4 = prim : com.google.javascript.jscomp.CheckLevel:null :
diagnosticGroupWarningsGuard5 = cons : com.google.javascript.jscomp.DiagnosticGroupWarningsGuard.<init>(com.google.javascript.jscomp.DiagnosticGroup,com.google.javascript.jscomp.CheckLevel) : diagnosticGroup2 checkLevel4
diagnosticGroup6 = getter : <get>(com.google.javascript.jscomp.DiagnosticGroup:com.google.javascript.jscomp.DiagnosticGroups.TYPE_INVALIDATION) :
b7 = method : com.google.javascript.jscomp.DiagnosticGroupWarningsGuard.disables(com.google.javascript.jscomp.DiagnosticGroup) : diagnosticGroupWarningsGuard5 diagnosticGroup6
Executed command: cd /tmp/run_randoop.pl_31036_1454493387 && java -ea -classpath /home/donghwan/defects4j/framework/lib/test_generation/generation/randoop-current.jar:/tmp/run_randoop.pl_31036_1454493387/build/classes:/tmp/run_randoop.pl_31036_1454493387/lib/args4j.jar:/tmp/run_randoop.pl_31036_1454493387/lib/guava.jar:/tmp/run_randoop.pl_31036_1454493387/lib/json.jar:/tmp/run_randoop.pl_31036_1454493387/lib/jsr305.jar:/tmp/run_randoop.pl_31036_1454493387/lib/protobuf-java.jar:/tmp/run_randoop.pl_31036_1454493387/build/lib/rhino.jar randoop.main.Main gentests --testclass=com.google.javascript.rhino.jstype.StringType --testclass=com.google.javascript.jscomp.MemoizedScopeCreator --testclass=com.google.javascript.jscomp.LightweightMessageFormatter --testclass=com.google.javascript.jscomp.ComposeWarningsGuard --testclass=com.google.javascript.jscomp.TypeCheck --testclass=com.google.javascript.jscomp.ConstCheck --testclass=com.google.javascript.jscomp.JSModule --testclass=com.google.javascript.rhino.Node --testclass=com.google.javascript.jscomp.ProcessTweaks --testclass=com.google.javascript.jscomp.LinkedFlowScope --testclass=com.google.javascript.rhino.jstype.NumberType --testclass=com.google.javascript.jscomp.PassConfig --testclass=com.google.javascript.jscomp.type.ReverseAbstractInterpreter --testclass=com.google.javascript.jscomp.ProcessClosurePrimitives --testclass=com.google.javascript.rhino.jstype.PrototypeObjectType --testclass=com.google.javascript.jscomp.TypeInference --testclass=com.google.javascript.jscomp.JoinOp --testclass=com.google.javascript.jscomp.Scope --testclass=com.google.javascript.jscomp.JsMessageVisitor --testclass=com.google.javascript.jscomp.NodeTraversal --testclass=com.google.javascript.rhino.JSDocInfoBuilder --testclass=com.google.javascript.rhino.jstype.TemplateType --testclass=com.google.javascript.jscomp.SourceFile --testclass=com.google.javascript.jscomp.StrictModeCheck --testclass=com.google.javascript.jscomp.parsing.TypeSafeDispatcher --testclass=com.google.javascript.rhino.jstype.JSTypeRegistry --testclass=com.google.javascript.jscomp.HotSwapCompilerPass --testclass=com.google.javascript.jscomp.JSError --testclass=com.google.javascript.jscomp.parsing.Config --testclass=com.google.javascript.jscomp.ErrorFormat --testclass=com.google.javascript.rhino.IR --testclass=com.google.javascript.jscomp.type.ClosureReverseAbstractInterpreter --testclass=com.google.javascript.rhino.ScriptRuntime --testclass=com.google.javascript.jscomp.DiagnosticType --testclass=com.google.javascript.jscomp.NodeUtil --testclass=com.google.javascript.jscomp.graph.DiGraph --testclass=com.google.javascript.jscomp.CheckGlobalThis --testclass=com.google.javascript.jscomp.SourceAst --testclass=com.google.javascript.rhino.jstype.ObjectType --testclass=com.google.javascript.rhino.jstype.FunctionType --testclass=com.google.javascript.jscomp.parsing.JsDocToken --testclass=com.google.javascript.jscomp.CheckUnreachableCode --testclass=com.google.javascript.jscomp.CleanupPasses --testclass=com.google.javascript.jscomp.ScopeCreator --testclass=com.google.javascript.jscomp.graph.LatticeElement --testclass=com.google.javascript.jscomp.CheckProvides --testclass=com.google.javascript.jscomp.deps.DependencyInfo --testclass=com.google.javascript.jscomp.FunctionTypeBuilder --testclass=com.google.javascript.jscomp.CompilerOptions --testclass=com.google.javascript.jscomp.VariableReferenceCheck --testclass=com.google.javascript.jscomp.type.FlowScope --testclass=com.google.javascript.jscomp.parsing.JsDocInfoParser --testclass=com.google.javascript.jscomp.graph.GraphvizGraph --testclass=com.google.javascript.jscomp.graph.LinkedDirectedGraph --testclass=com.google.javascript.jscomp.AbstractMessageFormatter --testclass=com.google.javascript.rhino.jstype.RecordTypeBuilder --testclass=com.google.javascript.rhino.jstype.JSType --testclass=com.google.javascript.jscomp.Region --testclass=com.google.javascript.rhino.jstype.UnresolvedTypeExpression --testclass=com.google.javascript.jscomp.CheckLevel --testclass=com.google.javascript.rhino.InputId --testclass=com.google.javascript.jscomp.parsing.ParserRunner --testclass=com.google.javascript.jscomp.DefaultPassConfig --testclass=com.google.javascript.jscomp.graph.SubGraph --testclass=com.google.javascript.jscomp.deps.SortedDependencies --testclass=com.google.javascript.rhino.jstype.NoType --testclass=com.google.javascript.jscomp.ControlFlowGraph --testclass=com.google.javascript.jscomp.DependencyOptions --testclass=com.google.javascript.jscomp.AbstractCompiler --testclass=com.google.javascript.jscomp.SuppressDocWarningsGuard --testclass=com.google.javascript.jscomp.CssRenamingMap --testclass=com.google.javascript.jscomp.ControlFlowAnalysis --testclass=com.google.javascript.rhino.jstype.UnionTypeBuilder --testclass=com.google.javascript.rhino.jstype.AllType --testclass=com.google.javascript.rhino.jstype.ValueType --testclass=com.google.javascript.jscomp.ProcessDefines --testclass=com.google.javascript.rhino.ErrorReporter --testclass=com.google.javascript.rhino.jstype.JSTypeNative --testclass=com.google.javascript.jscomp.CheckGlobalNames --testclass=com.google.javascript.jscomp.DiagnosticGroup --testclass=com.google.javascript.rhino.jstype.StaticScope --testclass=com.google.javascript.jscomp.graph.Annotatable --testclass=com.google.javascript.jscomp.InferJSDocInfo --testclass=com.google.javascript.rhino.jstype.SimpleSlot --testclass=com.google.javascript.jscomp.TypeValidator --testclass=com.google.javascript.rhino.JSTypeExpression --testclass=com.google.javascript.jscomp.AnonymousFunctionNamingPolicy --testclass=com.google.javascript.rhino.jstype.NamedType --testclass=com.google.javascript.jscomp.ClosureCodingConvention --testclass=com.google.javascript.rhino.jstype.StaticSourceFile --testclass=com.google.javascript.rhino.jstype.StaticSlot --testclass=com.google.javascript.jscomp.DisambiguateProperties --testclass=com.google.javascript.jscomp.DataFlowAnalysis --testclass=com.google.javascript.jscomp.CheckAccessControls --testclass=com.google.javascript.jscomp.CheckDebuggerStatement --testclass=com.google.javascript.jscomp.JSModuleGraph --testclass=com.google.javascript.rhino.jstype.NoResolvedType --testclass=com.google.javascript.jscomp.ControlStructureCheck --testclass=com.google.javascript.rhino.JSDocInfo --testclass=com.google.javascript.jscomp.CodingConvention --testclass=com.google.javascript.rhino.jstype.StaticSymbolTable --testclass=com.google.javascript.jscomp.PropertyRenamingPolicy --testclass=com.google.javascript.jscomp.parsing.JsDocTokenStream --testclass=com.google.javascript.jscomp.PassFactory --testclass=com.google.javascript.rhino.jstype.InstanceObjectType --testclass=com.google.javascript.jscomp.Compiler --testclass=com.google.javascript.rhino.jstype.FunctionBuilder --testclass=com.google.javascript.jscomp.VarCheck --testclass=com.google.javascript.jscomp.DiagnosticGroupWarningsGuard --testclass=com.google.javascript.rhino.SourcePosition --testclass=com.google.javascript.jscomp.CodingConventions --testclass=com.google.javascript.rhino.jstype.StaticReference --testclass=com.google.javascript.jscomp.ReferenceCollectingCallback --testclass=com.google.javascript.rhino.jstype.EquivalenceMethod --testclass=com.google.javascript.jscomp.MessageFormatter --testclass=com.google.javascript.jscomp.LoggerErrorManager --testclass=com.google.javascript.rhino.jstype.IndexedType --testclass=com.google.javascript.rhino.jstype.VoidType --testclass=com.google.javascript.jscomp.RhinoErrorReporter --testclass=com.google.javascript.jscomp.graph.GraphNode --testclass=com.google.javascript.rhino.jstype.UnknownType --testclass=com.google.javascript.jscomp.JsAst --testclass=com.google.javascript.jscomp.TypedScopeCreator --testclass=com.google.javascript.jscomp.SyntacticScopeCreator --testclass=com.google.javascript.jscomp.parsing.Annotation --testclass=com.google.javascript.jscomp.CompilerPass --testclass=com.google.javascript.jscomp.TypeInferencePass --testclass=com.google.javascript.rhino.jstype.EnumElementType --testclass=com.google.javascript.rhino.jstype.ErrorFunctionType --testclass=com.google.javascript.rhino.jstype.NoObjectType --testclass=com.google.javascript.rhino.jstype.NullType --testclass=com.google.javascript.rhino.jstype.EnumType --testclass=com.google.javascript.rhino.jstype.ArrowType --testclass=com.google.javascript.rhino.jstype.ProxyObjectType --testclass=com.google.javascript.jscomp.BasicErrorManager --testclass=com.google.javascript.jscomp.PrepareAst --testclass=com.google.javascript.jscomp.SourceExcerptProvider --testclass=com.google.javascript.jscomp.SourceMap --testclass=com.google.javascript.rhino.jstype.ParameterizedType --testclass=com.google.javascript.jscomp.graph.Annotation --testclass=com.google.javascript.jscomp.type.SemanticReverseAbstractInterpreter --testclass=com.google.javascript.jscomp.type.ChainableReverseAbstractInterpreter --testclass=com.google.javascript.rhino.jstype.FunctionParamBuilder --testclass=com.google.javascript.rhino.jstype.Visitor --testclass=com.google.javascript.jscomp.DiagnosticGroups --testclass=com.google.javascript.jscomp.ErrorHandler --testclass=com.google.javascript.rhino.jstype.BooleanType --testclass=com.google.javascript.jscomp.GoogleCodingConvention --testclass=com.google.javascript.jscomp.CheckSideEffects --testclass=com.google.javascript.rhino.jstype.UnionType --testclass=com.google.javascript.jscomp.CheckRegExp --testclass=com.google.javascript.jscomp.parsing.IRFactory --testclass=com.google.javascript.jscomp.graph.AdjacencyGraph --testclass=com.google.javascript.jscomp.WarningsGuard --testclass=com.google.javascript.jscomp.ErrorManager --testclass=com.google.javascript.jscomp.graph.Graph --testclass=com.google.javascript.jscomp.CodeChangeHandler --testclass=com.google.javascript.jscomp.CompilerInput --testclass=com.google.javascript.jscomp.VariableRenamingPolicy --junit-output-dir=randoop --timelimit=180 --randomseed=1006 --clear=10000 --string-maxlen=5000 --forbid-null=false --null-ratio=0.1 --no-error-revealing-tests=true 2>&1
Failed to generate tests! at /home/donghwan/defects4j/framework/bin/run_randoop.pl line 193.
Currently, Defects4J tags the buggy and the fixed version of a checked-out defect -- mainly for efficiency reasons. Some methods in Closure, however, read the current git tag. As a result, a test generation tool might generate a test for such a method that asserts on the current tag. Running such a test on the opposite version leads to a test failure, which is a false alarm as it does not reveal the actual defect.
Currently, Defects4J doesn't compile Time with all debugging information.
Hello, I want to know where exactly does dj4 store the unit tests written by the developers to uncover the bugs?
Thanks.
Hi,
I have a problem when doing mutation analysis with the test set generated by Evosuite.
Specifically, mutation.test
fails with the following message:
initialise:
compile:
[javac] /home/donghwan/defects4j/framework/projects/Chart/Chart.build.xml:50: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
compile-experimental:
[javac] /home/donghwan/defects4j/framework/projects/Chart/Chart.build.xml:99: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 4 source files to /tmp/run_mutation.pl_10654_1450680089/5f/build
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.4
[javac] 1 warning
compile.tests:
[mkdir] Created dir: /tmp/run_mutation.pl_10654_1450680089/5f/build-tests
[javac] /home/donghwan/defects4j/framework/projects/Chart/Chart.build.xml:116: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 373 source files to /tmp/run_mutation.pl_10654_1450680089/5f/build-tests
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.4
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 1 warning
update.all.tests:
mutation.test:
[echo] Running mutation analysis ...
[junit] MAJOR: Mutation analysis enabled
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[junit] MAJOR: �[1;31morg.jfree.data.xy.XYSeries_ESTest[initializationError] failed!�[0m
[junit] MAJOR: �[1;31m -> java.lang.LinkageError�[0m
[junit] MAJOR: �[1;31m -> "loader constraint violation: when resolving method "org.evosuite.runtime.thread.ThreadStopper.<init>(Lorg/evosuite/runtime/thread/KillSwitch;J[Ljava/lang/String;)V" the class loader (instance of org/apache/tools/ant/loader/AntClassLoader5) of the current class, org/jfree/data/xy/XYSeries_ESTest_scaffolding, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for resolved class, org/evosuite/runtime/thread/ThreadStopper, have different Class objects for the type untime/thread/KillSwitch;J[Ljava/lang/String;)V used in the signature"�[0m
BUILD FAILED
/home/donghwan/defects4j/framework/projects/defects4j.build.xml:214: Test org.jfree.data.xy.XYSeries_ESTest failed
Total time: 3 seconds
I guess SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
is the problem, but I cannot fix this issue by myself.
I tested the test_evosuite.sh
script given in `defects4j/framework/test/', and it failed with the same error message.
For your information, this is my test set generated by Evosuite (Chart-5f): https://www.dropbox.com/s/ndn76zkaufcbx4q/Chart-5f-evosuite-branch.1.tar.bz2?dl=0
Help me! Thank you!
Donghwan Shin
p.s. In the same environment, the test set generated by Randoop has no error for mutation analysis.
Hi, I have some issues when executing ./init.sh
I've got the following error:
: No such file or directory
If I execute:
sh ./init.sh
: command not found
: command not found
: command not found
Setting up project repositories ...
: No such file or directoryme-students/rmorales/tarfPlus/defects4j
: command not found
: command not found echo
Setting up Major ...
http://mutation-testing.org/downloads%0D/major-1.1.8%0D_jre7.zip%0D:
2016-11-02 13:36:34 ERROR 404: Not Found.
wget: unable to resolve host address “\r”
./init.sh: line 41: syntax error near unexpected token &&' '/init.sh: line 41:
&& unzip -o $MAJOR_ZIP > /dev/null \
I've run the code in a Fedora Linux and also in CentOS, and I have installed all the requirment tools.
Thanks.
It has been solved now.
From the conversation I had with René (via email), "the naming of the directory (patches) in the framework is ambiguous and probably confusing".
Here there are some details from René:
"Each patch provided by Defects4J (such as
defects4j/framework/projects/Closure/patches/1.src.patch) is a
bug-inducing rather than a bug-fixing diff. In other words, applying
such a patch to the fixed program version reinserts the bug. This is
exactly what Defects4J does to checkout a buggy program version. If
you want to view or use the bug-fixing diff instead, you can use a
tool such as interdiff to invert the diff."
Hi @rjust
I think the current version of Major does not mutate statements with the 'ternary operator'.
condition ? expression1 : expression2
I think we should mutate the condition
part to, for example, true
or false
.
I attempted to solve this using the Major-MML feature, but I failed.
Could you please give me any suggestions?
Many thanks,
Donghwan
Hello,
I do my research with your nice tools. I really appreciate you!
One problem when running the following command:
run_evosuite.pl -p Lang -v 1b -n 2 -c strongmutation -o Research/testSuties/ -b 10 -a 10 -D
The command return with the following error message:
[MASTER] 05:58:35.644 [pool-1-thread-3] ERROR TestSuiteGenerator - No Java compiler is available. Are you running with the JDK?
Of course, I have installed jdk7. For example,
echo $JAVA_HOME
returns:
/usr/lib/jvm/java-7-openjdk-amd64
javac -version
returns:
javac 1.7.0_85
java -version
returns:
java version "1.7.0_85" OpenJDK Runtime Environment (IcedTea 2.6.1) (7u85-2.6.1-5ubuntu0.14.04.1) OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)
Please take a look at the attached log files for the details: log.txt
Many Thanks!
Hi,
I have been generating test cases for all classes of each project/bug and found that there is at least one dependency missing, i.e., not included in the cp.compile
property of Closure bugs.
The class that is throwing a java.lang.NoClassDefFoundError
is com.google.javascript.jscomp.ant.AntErrorManager
of Closure-1:
Throwable thrown while handling command: java.lang.NoClassDefFoundError: org/apache/tools/ant/Task
java.lang.NoClassDefFoundError: org/apache/tools/ant/Task
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
at java.lang.Class.getDeclaredConstructors(Class.java:1914)
And, as you can see below, the lib/ant.jar
(the jar file with the org.apache.tools.ant.Task class) is only returned by cp.test
and not by cp.compile
:
## cp.compile of Closure-1f
$ $D4J_HOME/framework/bin/defects4j export -p cp.compile
/tmp/Closure-1f/build/classes:/tmp/Closure-1f/lib/args4j.jar:/tmp/Closure-1f/lib/guava.jar:/tmp/Closure-1f/lib/json.jar:/tmp/Closure-1f/lib/jsr305.jar:/tmp/Closure-1f/lib/protobuf-java.jar:/tmp/Closure-1f/build/lib/rhino.jar
## cp.test of Closure-1f
$D4J_HOME/framework/bin/defects4j export -p cp.test
/tmp/Closure-1f/build/classes:/tmp/Closure-1f/lib/args4j.jar:/tmp/Closure-1f/lib/guava.jar:/tmp/Closure-1f/lib/json.jar:/tmp/Closure-1f/lib/jsr305.jar:/tmp/Closure-1f/lib/protobuf-java.jar:/tmp/Closure-1f/build/lib/rhino.jar:/tmp/Closure-1f/lib/ant-launcher.jar:/tmp/Closure-1f/lib/ant.jar:/tmp/Closure-1f/lib/caja-r4314.jar:/tmp/Closure-1f/lib/jarjar.jar:/tmp/Closure-1f/lib/junit.jar:/tmp/Closure-1f/build/test
Should lib/ant.jar
be included in the cp.compile
?
--
Cheers,
Jose
This is a great project, but I'm worried that this property may be abused by researchers:
In Defects4J, an automated step removes all tests that fail
on Vfix from the test suite before attempting to expose the
bug in Vbug — the majority of program versions were accompanied
by failing tests. The automated step then executes
the test suite on Vbug , and a bug is considered reproducible
if at least one test case fails on Vbug due to the fault.
If I run, defects4j info -p Lang -b 1
Root cause in triggering tests:
- org.apache.commons.lang3.math.NumberUtilsTest::TestLang747
--> java.lang.NumberFormatException: For input string: "80000000"
--------------------------------------------------------------------------------
List of modified sources:
- org.apache.commons.lang3.math.NumberUtils
If I look at the fixing commit: apache/commons-lang@d1a45e9
This test case did not exist before the fix. It is actually a regression test to make sure the fix stays in place. The danger here is that any research trying to use test cases to help "find bugs" is missing the fact that the "answer" was provided by the developer after they did all the hard work of fault localization and fixing the bug.
Is there a better way to mark tests that existed at the same time as bugs versus tests that are added with fixes. My hypothesis is that a) failing tests for bugs do not frequently exist b) and if did exist, they would be harder for a developer (and automated technique) to use to locate bugs than regression tests.
Dear René,
For Closure-7f, I generated many tests using randoop (v.2.1.3) and the bug detection result said there is no bug in 7f. However, I found that one of the generated test (i.e., RegressionTest1468) triggered the fault when it is executed alone.
In other words, when I run the bug_detection.pl
script for Closure-7f with the whole tests including RegressionTest1468.java
, it returns no bug-triggering tests, while when I run the bug_detection.pl
script with only RegressionTest1468.java
, it returns that RegressionTest1468.java
triggers assertion failed.
I think this is abnormal.
I attached the bz2 file which contains the tests generated by randoop for Closure-7f (including RegressionTest1468.java
): https://www.dropbox.com/s/9ostuc8gc1naho0/Closure-7f-randoop.0.tar.bz2?dl=0
Please check this issue.
Thank you!
Donghwan Shin
Hi,
defects4j is a good tool.
I have the test suite produced by evosuite.
How can I produce coverage matrix( two dimension array consist of 0 and 1) by the test suite?
Hi @rjust
I found a bug in mutation with option -t
(single test method).
When I execute defects4j mutation -t <test_class>::<test_method>
, I expect the result of executing only the single test method <test_method>
in the test class <test_class>
. However, it returns the result of executing all test methods in the <test_class>
.
For example, for the Chart-12f
, the following commands return exactly the same results:
defects4j mutation -t org.jfree.chart.plot.junit.MultiplePiePlotTests::testConstructor
defects4j mutation -t org.jfree.chart.plot.junit.MultiplePiePlotTests::testEquals
defects4j mutation -t org.jfree.chart.plot.junit.MultiplePiePlotTests::testCloning
defects4j mutation -t org.jfree.chart.plot.junit.MultiplePiePlotTests::testSerialization
and, in fact, the results are the same as the result of executing the org.jfree.chart.plot.junit.MultiplePiePlotTests
test class as a whole.
I need your help.
Many thanks,
Donghwan
Dear René,
I try to perform weak mutation testing in Defects4J.
In the Major documentation, I found the mutation coverage map, but I am not so sure the mutation coverage map refers the state infection results of mutants for tests.
If there is another way to find the weak mutation results, please let me know. Thank you.
Regards,
Donghwan
Major's version of ant does not support Java 8.
Hi, I wonder if it is possible to obtain information about in which bugs EvoSuite was able to detect them. I checkout some bugs, and I noticed that 1) the coverage of evosuite is really low (14% Mockito 23b) and 2) that it does not detect the real bug, or the source class of the bug. Is this information available somwhere?
Thanks,
Kind regards,
Rodrigo.
If an error occurs (e.g., if a download fails or a file cannot be created on the file system) the init script should report the error and fail.
Hi @rjust,
I have been trying to run mutation-analysis on some automatically generated test suites by EvoSuite, but no success so far.
It seems that ant runs the mutation-analysis without using the EvoSuiteRunner/EvoSuite's Scaffolding, therefore test cases start to fail (e.g., due of JVM non deterministic stuff like time) and no mutation data is returned. Interesting point is that when I execute "$D4J_HOME/framework/util/fix_test_suite.pl"
script or manually execute the test suites, no error is returned. On the other hand, when I execute "$D4J_HOME/framework/bin/defects4j" mutation
the test suites fail.
I prepared a minimal working example (Time-9f-evosuite.0.zip) and here are the steps to trigger the issue:
## Get D4J
$ mkdir /tmp/mutation_issue
$ cd /tmp/mutation_issue
$ git clone https://github.com/rjust/defects4j.git
$ cd defects4j
$ git checkout java-8-support
$ ./init.sh
$ export D4J_HOME=`pwd`
$ cd ..
## Get Time-9f
$ "$D4J_HOME/framework/bin/defects4j" checkout -p Time -v 9f -w Time-9f
$ cd Time-9f
# dowload the attached Time-9f-evosuite.0.zip file to here
# convert zip to tar.bz2 (as Github does not allow tar.bz2 attachments)
$ unzip Time-9f-evosuite.0.zip
$ tar -jcvf Time-9f-evosuite.0.tar.bz2 org/
$ rm -rf org/ Time-9f-evosuite.0.zip
## Is there any failing test case that needs to be fixed/removed?
$ "$D4J_HOME/framework/util/fix_test_suite.pl" -p Time -d . -v 9f
--------------------------------------------------------------------------------
Time-9f-evosuite.0.tar.bz2
--------------------------------------------------------------------------------
Check out 96f586cb to /tmp/fix_test_suite.pl_30815_1479901514.............. OK
Init local repository...................................................... OK
Tag post-fix revision...................................................... OK
Run post-checkout hook..................................................... OK
Initialize fixed program version........................................... OK
Apply patch................................................................ OK
Initialize buggy program version........................................... OK
Diff 96f586cb:08a3a0b9..................................................... OK
Apply patch................................................................ OK
Tag pre-fix revision....................................................... OK
Check out program version: Time-9f......................................... OK
Extract test suite......................................................... OK
Running ant (compile.gen.tests)............................................ OK
Running ant (run.gen.tests)................................................ OK
Running ant (compile.gen.tests)............................................ OK
Running ant (run.gen.tests)................................................ OK
Running ant (compile.gen.tests)............................................ OK
Running ant (run.gen.tests)................................................ OK
Running ant (compile.gen.tests)............................................ OK
Running ant (run.gen.tests)................................................ OK
Running ant (compile.gen.tests)............................................ OK
Running ant (run.gen.tests)................................................ OK
$ cat fix_test_suite.log
###### Start fixing tests /tmp/mutation_issue/defects4j/framework/util/fix_test_suite.pl: 2016-11-23 03:45:14 ######
- Found 1 test archive(s)
###### End fixing tests /tmp/mutation_issue/defects4j/framework/util/fix_test_suite.pl: 2016-11-23 03:46:12 ######
## Execute the generated test suite
$ "$D4J_HOME/framework/bin/defects4j" compile
$ CP=$("$D4J_HOME/framework/bin/defects4j" export -p cp.compile)
$ mkdir evosuite-tests
$ tar -jxvf Time-9f-evosuite.0.tar.bz2 -C evosuite-tests/
$ cd evosuite-tests
$ javac -cp .:$CP:"$D4J_HOME/framework/lib/formatter/lib/junit-4.11.jar":"$D4J_HOME/framework/lib/test_generation/runtime/evosuite-rt.jar" org/joda/time/DateTimeZone_ESTest.java
$ java -cp .:$CP:"$D4J_HOME/framework/lib/formatter/lib/junit-4.11.jar":"$D4J_HOME/framework/lib/test_generation/runtime/evosuite-rt.jar" org.junit.runner.JUnitCore org.joda.time.DateTimeZone_ESTest
JUnit version 4.11
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
.
Time: 1.155
OK (1 test)
## Run mutation analysis
$ cd ..
$ "$D4J_HOME/framework/bin/defects4j" mutation -s Time-9f-evosuite.0.tar.bz2
Running ant (mutate)....................................................... OK
Extract test suite......................................................... OK
Running ant (compile.gen.tests)............................................ OK
Running ant (mutation.test)................................................ FAIL
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1G; support was removed in 8.0
Buildfile: /tmp/mutation_issue/defects4j/framework/projects/defects4j.build.xml
Executed command: cd /tmp/mutation_issue/Time-9f && PATH=/tmp/mutation_issue/defects4j/major/bin:$PATH && /tmp/mutation_issue/defects4j/major/bin/ant -f /tmp/mutation_issue/defects4j/framework/projects/defects4j.build.xml -Dd4j.home=/tmp/mutation_issue/defects4j -Dbasedir=/tmp/mutation_issue/Time-9f -Dbuild.compiler=major.ant.MajorCompiler -Dd4j.test.dir=/tmp/mutation_issue/Time-9f/.test_suite -Dd4j.test.include=*.java -Dmajor.exclude=/tmp/mutation_issue/Time-9f/exclude.txt -Dmajor.kill.log=/tmp/mutation_issue/Time-9f/kill.csv -logfile /tmp/mutation_issue/Time-9f/.mutation.log mutation.test 2>&1
test-offline:
get-deps:
joda-time-from-maven.compile:
[javac] /tmp/mutation_issue/Time-9f/maven-build.xml:78: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Since compiler setting isn't classic or modern, ignoring fork setting.
[javac] Since compiler setting isn't classic or modern, ignoring fork setting.
joda-time.compile:
compile:
[java] Writing zoneinfo files
[java] Writing ZoneInfoMap
joda-time-from-maven.compile-tests:
[javac] /tmp/mutation_issue/Time-9f/maven-build.xml:117: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Since compiler setting isn't classic or modern, ignoring fork setting.
[javac] Since compiler setting isn't classic or modern, ignoring fork setting.
joda-time.compile.tests:
compile.tests:
update.all.tests:
mutation.test:
[echo] Running mutation analysis ...
[junit] MAJOR: Mutation analysis enabled
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[junit] MAJOR: org.joda.time.DateTimeZone_ESTest[test01] failed!
[junit] MAJOR: -> AssertionFailedError
[junit] MAJOR: -> "expected:<1392409281320> but was:<1479901909169>"
BUILD FAILED
/tmp/mutation_issue/defects4j/framework/projects/defects4j.build.xml:221: Test org.joda.time.DateTimeZone_ESTest failed
Total time: 3 seconds
Could you please take a look at this asap? Could it be somehow due to Java-8 (btw, I'm using version 1.8.0_111)?
--
Thanks,
Jose
Dear René,
I found that the Major employs the non-redundant version of mutation operators (as far as I know COR and ROR) by default. I want to generate all mutants (including redundants). I just wonder Is there any option for that in Major.
Have a nice day!
Best regards,
Donghwan
Hi,
It seems that mutation analysis of Mockito bugs only works with Java 7.
With Java-8, the mutate target failed due to a Class not found: major.ant.MajorCompiler
.
Here the commands to reproduce the issue:
$ mkdir -p /tmp/mockito-major-issue
$ cd /tmp/mockito-major-issue
## Get D4J
$ git clone https://github.com/rjust/defects4j.git
$ cd defects4j
$ git checkout java-8-support
$ ./init.sh
$ export D4J_HOME=$(pwd)
$ cd ../
## Checkout a Mockito bug
$ $D4J_HOME/framework/bin/defects4j checkout -p Mockito -v 17f -w Mockito-17f
$ cd Mockito-17f
## Run mutation-analysis
$ $D4J_HOME/framework/bin/defects4j mutation -r
Running ant (mutate)....................................................... FAIL
Buildfile: /tmp/mockito-major-issue/defects4j/framework/projects/defects4j.build.xml
check.classes.uptodate:
mutate:
compile:
gradle.compile:
ant.compile:
gradle.compile.mutants:
ant.compile.mutants:
clean:
prepare:
[mkdir] Created dir: /tmp/mockito-major-issue/Mockito-17f/target/classes
[mkdir] Created dir: /tmp/mockito-major-issue/Mockito-17f/target/test-classes
[mkdir] Created dir: /tmp/mockito-major-issue/Mockito-17f/target/reports/junit
compile:
[javac] /tmp/mockito-major-issue/Mockito-17f/build.xml:76: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
BUILD FAILED
/tmp/mockito-major-issue/defects4j/framework/projects/defects4j.build.xml:173: The following error occurred while executing this line:
/tmp/mockito-major-issue/defects4j/framework/projects/Mockito/Mockito.build.xml:133: The following error occurred while executing this line:
/tmp/mockito-major-issue/defects4j/framework/projects/Mockito/Mockito.build.xml:123: The following error occurred while executing this line:
/tmp/mockito-major-issue/Mockito-17f/build.xml:76: Class not found: major.ant.MajorCompiler
Total time: 1 second
Executed command: cd /tmp/mockito-major-issue/Mockito-17f && ant -f /tmp/mockito-major-issue/defects4j/framework/projects/defects4j.build.xml -Dd4j.home=/tmp/mockito-major-issue/defects4j -Dbasedir=/tmp/mockito-major-issue/Mockito-17f -Dbuild.compiler=major.ant.MajorCompiler mutate 2>&1
Cannot mutate project! at /tmp/mockito-major-issue/defects4j/framework/bin/d4j/d4j-mutation line 146.
Compilation failed in require at /tmp/mockito-major-issue/defects4j/framework/bin/defects4j line 155.
Is this somehow related to the issue/fix #34 ?
--
Cheers,
Jose
Hi,
I noticed that for (at least one) Mockito's bug defects4j export
deletes the test classes directory. For example, suppose that I checkout Mockito-25b and then I compile it:
$ $D4J_HOME/framework/bin/defects4j checkout -p Mockito -v 25b -w Mockito-25b
$ cd Mockito-25b
$ $D4J_HOME/framework/bin/defects4j compile
at this point I have a target directory with two sub-directories: classes and test-classes, each one with the respective .class files.
But, if I execute $D4J_HOME/framework/bin/defects4j export -p cp.compile
or $D4J_HOME/framework/bin/defects4j export -p cp.test
the target/test-classes is deleted.
The world will not end tomorrow just because of this, but anyway... 😃
Mutation analysis of Mockito project versions is currently not working; this will be fixed for the next official release.
Bug-mining scripts should exclusively use the working directory until bugs are promoted to the main database. Currently, a newly configured project and a project's layout map are directly written to the main database.
Hi Defects4j Team,
I frequently had this error (see below) when I use the defects4j checkout feature.
Applying patch ... FAIL fatal: invalid path <absolute_path>
I order to solve this issue I replace this line (https://github.com/rjust/defects4j/blob/master/framework/core/Vcs/Git.pm#L54):
return "git --work-tree=$work_dir apply --directory=$work_dir/$path $patch_file 2>&1";
by this one:
return "git --work-tree=$work_dir apply --unsafe-paths --directory=$work_dir/$path $patch_file 2>&1";
I don't like disable security checks but I think in this case it's not a problem.
Hi,
I checked the patches of the bug Math 97 and the Math 99 and I observed that the two patches are exactly the same.
Is it normal?
Thanks
Hi René,
I've been running a sanity check on all test methods using $D4J_HOME/framework/lib/formatter.jar
and $D4J_HOME/framework/bin/defects4j test -t ...
, and there are two test methods to which the outcome of each tool is different:
Lang,64,org.apache.commons.lang.enums.ValuedEnumTest::testCompareTo_classloader_different
Lang,64,org.apache.commons.lang.enums.ValuedEnumTest::testCompareTo_classloader_equal
for example, the test method testCompareTo_classloader_different
passes with $D4J_HOME/framework/bin/defects4j test -t
$ $D4J_HOME/framework/bin/defects4j checkout -p Lang -v 64b -w Lang-64b
$ cd Lang-64b
$ $D4J_HOME/framework/bin/defects4j compile
$ $D4J_HOME/framework/bin/defects4j test -t org.apache.commons.lang.enums.ValuedEnumTest::testCompareTo_classloader_different
however, it fails if I use $D4J_HOME/framework/lib/formatter.jar
:
$ $D4J_HOME/framework/bin/defects4j checkout -p Lang -v 64b -w Lang-64b
$ cd Lang-64b
$ CP=$($D4J_HOME/framework/bin/defects4j export -p cp.test)
$ $D4J_HOME/framework/bin/defects4j compile
java -cp "$CP:$D4J_HOME/framework/lib/formatter.jar" edu.washington.cs.mut.testrunner.SingleTestRunner org.apache.commons.lang.enums.ValuedEnumTest::testCompareTo_classloader_different
any clue why the custom JUnit runner reports a failing test method and the other approach not?
What about running the test class from the command line with org.junit.runner.JUnitCore
?
$ $D4J_HOME/framework/bin/defects4j checkout -p Lang -v 64b -w Lang-64b
$ cd Lang-64b
$ CP=$($D4J_HOME/framework/bin/defects4j export -p cp.test)
$ $D4J_HOME/framework/bin/defects4j compile
$ java -cp "$CP" org.junit.runner.JUnitCore org.apache.commons.lang.enums.ValuedEnumTest
well, it seems that there are indeed 3 failing test cases and not just 1 as reported by $D4J_HOME/framework/projects/Lang/trigger_tests/64
.
and actually, those 3 failing test cases seem to be triggering test cases, as they do not fail on the fixed version, i.e., Lang-64f.
Hi,
In my experimentation workflow, I first generate many tests using randoop, and then run fix_test_suite.pl since I found there are many broken tests returning from run_bug_detection.pl.
Usually, fix_test_suite scipt works well, but not for the Math project. In that project, several versions do not end. For example, for the following code in bash:
fix_test_suite.pl -p Math -v 11f -d tests/Math/randoop/1/ -o results/
returns:
(...)
Check out program version: Math-11f........................................ OK
Running ant (compile.gen.tests)............................................ OK
Running ant (run.gen.tests)................................................
and does not end.
This is not the only one; there are many versions in Math that do not end at run.gen.tests
.
I temporarily add some timeout code at the run.gen.tests
task in the ant bulild code defects4j.build.xml
, but I am not sure this is the best solution. I think you may know this issue. If you have any suggestion, please let me know.
Thank you!
Donghwan Shin
p.s. For your information, I attached my test suite file for Math 11f generated by randoop: https://www.dropbox.com/s/mbcepflumd019qk/Math-11f-randoop.1.tar.bz2?dl=0
p.s.2. Quick question: is it okay to simultaneously run several run_randoop.pl
and run_mutation.pl
scripts on different terminals in the same machine?
It would be really helpful if the "info" method could provide additional information of the requested version ID.
In particular, it would be helpful to have the following additional info:
I want to know the revision id of the bugs to retrieving information of a development history of the software.
The information is needed for my research about bug fixing.
I've looked through readme and your paper, but I couldn't find information about the revision id of the bugs.
Is there any way to know the revision id?
Hello,
Is it possible to configure the tool to list all executed tests no matter the result? Or even better, to provide the files that JUnit generates after a run?
Could you provide an example how to use run_coverage.pl to run a coverage analysis using the tests that are already part of a project?
I tried to run this command and no test was executed:
lang_1_buggy$ run_coverage.pl -p Lang -d src/test/java/ -o /tmp/result_db
However, this was the output of running such command:
Smartmatch is experimental at /vagrant/defects4j/framework/bin/run_coverage.pl line 128.
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.