maxbechtold/golden-master

A harness for writing Golden Master tests in Java, painlessly.

golden-master's Issues

Easier input generation

Currently, inputs are to be created based on an index. This requires the test setup to (deterministically) build inputs from the index. It should prove useful to explore generators like those of JQwik to "smartly" create inputs.

It is then the user's responsibility to register those inputs with the goldenmaster JUnit extension to bring them together with the outputs. Something like File output = goldenMaster.register("input-representation", inputObject) perhaps?

Thanks @BlueIce, @rolandsand, @JanMuellerAndrena, @deeteear for the fruitful discussion!

Move maxbe.goldenmaster.approval to the approval library

This package provides approval extensions and should be part of that library

@Test methods with @GoldenMasterRun annotation fail

It doesn't matter if there are methods annotated with @GoldenMasterRun or not.

Observed with golden-master 0.6-SNAPSHOT.

Stacktrace:

Suppressed: java.lang.NullPointerException
		at maxbe.goldenmaster.approval.ApprovalIdResolver.getApprovalId(ApprovalIdResolver.java:21)
		at maxbe.goldenmaster.approval.ApprovalIdResolver.resolveApprovalIdFor(ApprovalIdResolver.java:16)
		at maxbe.goldenmaster.junit.extension.RunInvocationContextProvider.afterEach(RunInvocationContextProvider.java:97)
		at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachCallbacks$11(TestMethodTestDescriptor.java:245)
		at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$12(TestMethodTestDescriptor.java:256)
		at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
		at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$13(TestMethodTestDescriptor.java:256)
		at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
		at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAllAfterMethodsOrCallbacks(TestMethodTestDescriptor.java:255)
		at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAfterEachCallbacks(TestMethodTestDescriptor.java:244)
		at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:141)
		... 43 more

Ensure tests still run with Java 11+

Looking at #7, it seems the used Mockito version doesn't work with Java 11

Configure build to pack a sources.jar for JitPack provisioning

Thus, users get more insight into how to use the JUnit extension.

Approval script only works for a single test class

With more than one test class being run, the approval file is overwritten for each class.

If the final class produces no change, the approval script of a test class previously executed is deleted.

Simplify initial approval

At this time, users of the golden-master harness must perform several steps to get an initial set of master files:

1. Run the golden-master tests, which fail since there are no approval files to compare the output against
2. Run the golden-master tests again, which compares against the empty approval files created in the previous step, thus fail
3. Execute the provided approval script to approve the output of the previous run
4. Run the golden-master tests again, which finally succeed (unless your instrumentation yields a non-deterministic output)

There's no good reason for step 2) which I'd like to eliminate.

Provide an example for how to use the Golden Master harness

Also, give hints on setting up a distinct project for the Golden Master test so as to make sure to use Java 8

When using different runs with identical IDs, the approval script fails

... as the runs share the same *.forapproval.approved file which can only be moved once to *.approval. However, the result is still that all files are re-approved (if the script runs through, only checked Windows 10)

In particular, if only one of the same-ID runs has failing repetitions, what are the semantics of re-approving the changed behavior (which will affect all runs)?

Approve behavioural changes by some "oracle"

This is related to #2 in that it allows approval on-the-fly, but in a more elaborated manner.

E.g. there's a use case to adjust a program while it is only covered by golden master tests. This could be as simple as adapting the output format, or more complex changes. For very regular changes (e.g replace space by tab) there could be an automatic way to approve changes, if the output changes in the described way. This could leverage the power of tools like kdiff3 and sed.

Questions:

• is it enough to let the user provide an "oracle" method like assess(input, output)?
• how to ensure this "oracle" is only active once/for the current change?
• how does this relate to the comparison built into FastApproval (cf. ExecutableDifferenceReporter)?

Parameter resolution breaks with primitive int

The docs clearly mention to use an Integer index, however I myself wrote a method like such

  @GoldenMasterRun
  void instrumentProgramUnderTest(int index) {
    // Do some hard work whose output will be compared to that of previous runs
  }

This way, tests will end in errors due to a ParameterResolutionException. This should at least hint to using the boxed types, or even auto-unbox parameters if possible.

Reconsider term "program instrumentation" in docs

...as it is not related to the JVM concept (https://docs.oracle.com/javase/8/docs/api/java/lang/instrument/package-summary.html) - good point by @JanMuellerAndrena

Consider fuzzy approval for non-deterministic programs

In my use case of a concurrent program it proved to be useful to be somewhat lenient when approving (large) outputs against the master files.

The output of two threads was in a race condition, leading to switched lines in the output files. The harness contains a MAX_DEVIATION constant (currently set to 0.0) that can be used to allow minor variations in the output. Non-perfect matches with the master files are then reported as "aborted" tests (equivalent to failed assumptions up to JUnit 4), which are then neither "green" nor "red."

Of course, this can only work for minor differences. If the program output is very loosely related to the input or contains many timestamps (e.g. through logging), this approach is too simple.

Clean up code and provide tests

The harness is working, but there's still some clean up to do.

Accelerate approval with an auto-approve option

Right now, the user has to manually overwrite the current master files in order to approve a new set of master files (or execute the created approval script)

It would be better to have this closer to the test execution, i.e. perform the approval as part of a test execution.

Allow meaningful DisplayName for test executions

Currently, test executions resulting from a @GoldenMasterRun will be named as [0], [1], and so forth. In particular with #16 it would be interesting to allow to give descriptive names to such executions that help to identify the input.

Similarly, it should be checked if "Run" is a good name for this test annotation, or if this rather marks a descriptor for runs or executions. Also, @GoldenMasterTest might be confusing to add to the test class rather than its methods.

Credits to @deeteear for the feedback.

Replace File API with a Sink concept

The file API is a bit simple. It might improve usability to provide Sinks that the user can write outputs to, e.g.

• StringSink#write(String text)
• SerializableSink#write(Serializable Object)
• XMLSink#write(DOM tree)

These need to have a corresponding read method to be used for approval/comparison.

The output should still be represented in a textual format in order to present a diff to the user.

Travis CI Build badge broken - move to GitHub Actions?

The project isn't even listed on https://app.travis-ci.com/github/maxbechtold anymore, maybe just move to GitHub actions while at it