maxbechtold / golden-master Goto Github PK
View Code? Open in Web Editor NEWA harness for writing Golden Master tests in Java, painlessly.
License: MIT License
A harness for writing Golden Master tests in Java, painlessly.
License: MIT License
Currently, inputs are to be created based on an index. This requires the test setup to (deterministically) build inputs from the index. It should prove useful to explore generators like those of JQwik to "smartly" create inputs.
It is then the user's responsibility to register those inputs with the goldenmaster JUnit extension to bring them together with the outputs. Something like File output = goldenMaster.register("input-representation", inputObject)
perhaps?
Thanks @BlueIce, @rolandsand, @JanMuellerAndrena, @deeteear for the fruitful discussion!
This package provides approval extensions and should be part of that library
It doesn't matter if there are methods annotated with @GoldenMasterRun
or not.
Observed with golden-master 0.6-SNAPSHOT.
Stacktrace:
Suppressed: java.lang.NullPointerException
at maxbe.goldenmaster.approval.ApprovalIdResolver.getApprovalId(ApprovalIdResolver.java:21)
at maxbe.goldenmaster.approval.ApprovalIdResolver.resolveApprovalIdFor(ApprovalIdResolver.java:16)
at maxbe.goldenmaster.junit.extension.RunInvocationContextProvider.afterEach(RunInvocationContextProvider.java:97)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachCallbacks$11(TestMethodTestDescriptor.java:245)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$12(TestMethodTestDescriptor.java:256)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$13(TestMethodTestDescriptor.java:256)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAllAfterMethodsOrCallbacks(TestMethodTestDescriptor.java:255)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAfterEachCallbacks(TestMethodTestDescriptor.java:244)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:141)
... 43 more
Looking at #7, it seems the used Mockito version doesn't work with Java 11
Thus, users get more insight into how to use the JUnit extension.
With more than one test class being run, the approval file is overwritten for each class.
If the final class produces no change, the approval script of a test class previously executed is deleted.
At this time, users of the golden-master harness must perform several steps to get an initial set of master files:
There's no good reason for step 2) which I'd like to eliminate.
Also, give hints on setting up a distinct project for the Golden Master test so as to make sure to use Java 8
... as the runs share the same *.forapproval.approved
file which can only be moved once to *.approval
. However, the result is still that all files are re-approved (if the script runs through, only checked Windows 10)
In particular, if only one of the same-ID runs has failing repetitions, what are the semantics of re-approving the changed behavior (which will affect all runs)?
This is related to #2 in that it allows approval on-the-fly, but in a more elaborated manner.
E.g. there's a use case to adjust a program while it is only covered by golden master tests. This could be as simple as adapting the output format, or more complex changes. For very regular changes (e.g replace space by tab) there could be an automatic way to approve changes, if the output changes in the described way. This could leverage the power of tools like kdiff3 and sed.
Questions:
assess(input, output)
?ExecutableDifferenceReporter
)?The docs clearly mention to use an Integer index
, however I myself wrote a method like such
@GoldenMasterRun
void instrumentProgramUnderTest(int index) {
// Do some hard work whose output will be compared to that of previous runs
}
This way, tests will end in errors due to a ParameterResolutionException. This should at least hint to using the boxed types, or even auto-unbox parameters if possible.
...as it is not related to the JVM concept (https://docs.oracle.com/javase/8/docs/api/java/lang/instrument/package-summary.html) - good point by @JanMuellerAndrena
In my use case of a concurrent program it proved to be useful to be somewhat lenient when approving (large) outputs against the master files.
The output of two threads was in a race condition, leading to switched lines in the output files. The harness contains a MAX_DEVIATION
constant (currently set to 0.0) that can be used to allow minor variations in the output. Non-perfect matches with the master files are then reported as "aborted" tests (equivalent to failed assumptions up to JUnit 4), which are then neither "green" nor "red."
Of course, this can only work for minor differences. If the program output is very loosely related to the input or contains many timestamps (e.g. through logging), this approach is too simple.
The harness is working, but there's still some clean up to do.
Right now, the user has to manually overwrite the current master files in order to approve a new set of master files (or execute the created approval script)
It would be better to have this closer to the test execution, i.e. perform the approval as part of a test execution.
Currently, test executions resulting from a @GoldenMasterRun
will be named as [0], [1], and so forth. In particular with #16 it would be interesting to allow to give descriptive names to such executions that help to identify the input.
Similarly, it should be checked if "Run" is a good name for this test annotation, or if this rather marks a descriptor for runs or executions. Also, @GoldenMasterTest
might be confusing to add to the test class rather than its methods.
Credits to @deeteear for the feedback.
The file API is a bit simple. It might improve usability to provide Sinks that the user can write outputs to, e.g.
These need to have a corresponding read method to be used for approval/comparison.
The output should still be represented in a textual format in order to present a diff to the user.
The project isn't even listed on https://app.travis-ci.com/github/maxbechtold anymore, maybe just move to GitHub actions while at it
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.