franiglesias/golden

Golang Library for snapshot testing

golden's Issues

Improve BetterDiffReported separating lines

Describe the bug

Not exactly a bug, but the new BetterDiffReported should separate lines.

Customization of reporters

Managing no deterministic output

Manage not deterministic data using scrubbers.

Scrubbers can work using regexp to locate specific info and replace with a static representation. It could be something like "removed for testing", or maybe a valid example of data, but I thing it should be marked some way. This can be a problem if we want to use the snapshot as an example.

We should be able to pass any number of scrubbers.

We should be able to ask for predefined scrubbers, because there are lots of common cases.

The place for scrubbing should be the Normalizer.

Customization of snapshot name

t.Name() provides a good default for snapshot naming after the test name, but...

• If the test name is changed, the snapshot file will change name and we end with some unused snapshot files
• what if we need several snapshots in the same test. Not the best practice, but it could happen
• this should allow us to use snapshot files created or customized from other sources, like an output generated by another system that we need to replicate

API

Golden.UseSnapshot(name string) Golden

Customization of snapshots folder

Combinatorial tests and Golden Master

Combinatorial testing requires to create a simple interface that can be used in multiple situations.

1. We need to pass a way to invoke the Subject Under Test with the needed parameters
2. We should allow a list of possible values for each parameter
3. We need to generate all possible combinations of the multiples values
4. Then, we need to execute SUT for every parameters combination
5. And, finally, register that in a snapshot file, so we can re-run the test and verify that nothing changed

API

golden.GoldenMaster(t, function, ...values)

Some ideas

It could be good to create the snapshot file as an array of json objects. This could make easier for users to understand the snapshot and review the results.

Json Field scrubber; replace a json field content with somethign (JsonFieldScrubber(field, replacement)

Approvals

Provide a way to force the approval of a snapshot before it can be used as matching criteria.

In "snapshot mode", a snapshot that is created for first time is supposed to represent the behaviour of the subject under test.
In "approval mode", a snapshot that is created for first time has to wait to be approved by a human actor before being used as matching criteria.

Known approaches

• Approvals test uses the file name to mark a snapshot file as approved of pending. You need to rename the file in order to mark as approved.
• Other libraries uses an env variable to run the test. If the env variable has certain value, the snapshot is considered approved.

Proposal

Do this in the test

First, you execute the test in "approval mode", so test will never pass.

golden.ToApprove(t, subject)

Once you are happy with the output, back to verify

golden.Verify(t, subject)

Pros

• Explicit in the test code. You always know if your snapshot is valid or not
• Convenient: you don't need to move away from code, manage files o worried about env vars, even if they are temporal

Cons

• You need to be sure that all snapshots are approved before committing changes, given that tests with ToApprove status are going to fail in CI. (This could be Pro, if you ask me, and it is no different from other systems)

API

golden.ToApprove(t, subject)

This will kind of a decorator around Verify. It will set a flag so the system can decide if it needs to compare snapshot and subject. Snapshot creation has to be forced even if previous snapshot exists.

A reminder to approve the snapshot can be added to the report.

API for avoiding manual type assertions

Is your feature request related to a problem? Please describe.
Type assertions are annoying.

Describe the solution you'd like
Make things more statically typed & easier to use. Also reduce number of places to change when you add/remove/change parameters.

Describe alternatives you've considered
I'm using this helper function which does type assertions for me. Would be nice to have something like that built into the library:

func toGoldenMasterFn(fn interface{}) func(args ...any) any {
	return func(args ...any) any {
		fnVal := reflect.ValueOf(fn)
		if fnVal.Kind() != reflect.Func {
			panic("wrapFunctionWithAnyArgs: provided interface is not a function")
		}

		fnType := fnVal.Type()
		if fnType.IsVariadic() {
			panic("wrapFunctionWithAnyArgs: variadic functions are not supported")
		}

		if len(args) != fnType.NumIn() {
			panic(fmt.Sprintf("wrapFunctionWithAnyArgs: incorrect number of arguments. Expected %d, got %d", fnType.NumIn(), len(args)))
		}

		in := make([]reflect.Value, len(args))
		for i, arg := range args {
			in[i] = reflect.ValueOf(arg)
		}

		results := fnVal.Call(in)
		if len(results) != 1 {
			panic("wrapFunctionWithAnyArgs: function does not return exactly one result")
		}

		return results[0].Interface()
	}
}

// Usage:
func TestGoldenMaster(t *testing.T) {
    // define the wrapper function
    f := toGoldenMasterFn(func(title, part string, span int) any {
        return Border(title, part, span)
    })
    
    // define the input values to combine
    titles := []any{"Example 1", "Example long enough", "Another thing"}
    parts := []any{"-", "=", "*", "#"}
    times := []any{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}

    // run the test
    golden.Master(t, f, golden.Combine(titles, parts, times))
}

Additional context
Another option, maybe even better one - have a set of generic functions for accepting N generic paramters, like:

golang.MasterOf2[T1, T2](t *testing.T, fn func(a T1, b T2) any, params1 []T1, params2 []T2)
golang.MasterOf3[T1, T2, T3](t *testing.T, fn func(a T1, b T2, c T3) any, params1 []T1, params2 []T2, params3 []T3)
// make a bunch more of these to cover all realistic use-cases...

This looks a bit goofy, but in practice it's not that bad - similar pattern is widely used in Elm, for example, and it's just fine https://package.elm-lang.org/packages/elm/json/latest/Json-Decode#map

With this approach, example above could look like:

// Usage:
func TestGoldenMaster(t *testing.T) {
    // define the wrapper function
    f := func(title, part string, span int) any {
        return Border(title, part, span)
    })
    
    // define the input values to combine
    titles := []any{"Example 1", "Example long enough", "Another thing"}
    parts := []any{"-", "=", "*", "#"}
    times := []any{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}

    // run the test
    golden.MasterOf3(t, f, titles, parts, times)
}

Which is a bit cleaner, and in addition to that you can't swap parameters of different types by accident.

Apply scrubber in json path if possible.i.e: "event.id"

During approval mode add a message to the report stating how to approve the snapshot

Add a reminder to approve the test when executing in Approval mode