matryer/is

Professional lightweight testing mini-framework for Go.

is

Professional lightweight testing mini-framework for Go.

• Easy to write and read
• Beautifully simple API with everything you need: is.Equal, is.True, is.NoErr, and is.Fail
• Use comments to add descriptions (which show up when tests fail)

Failures are very easy to read:

Usage

The following code shows a range of useful ways you can use the helper methods:

func Test(t *testing.T) {
	is := is.New(t)
	signedin, err := isSignedIn(ctx)
	is.NoErr(err)            // isSignedIn error
	is.Equal(signedin, true) // must be signed in
	body := readBody(r)
	is.True(strings.Contains(body, "Hi there"))
}

Color

To turn off the colors, run go test with the -nocolor flag, or with the env var NO_COLOR (with any value).

go test -nocolor

NO_COLOR=1 go test

is's Issues

gofmt all the things

we're only getting a B because of bad gofmting: https://goreportcard.com/report/github.com/matryer/is#gofmt

Potential issue when running go test -json

Hey Mat, long time. Ran into a curious issue worth filing.

Summary

Running go test -json ... and using this package reports incorrect test names in JSON events. This could very well be a bug in Go (test2json), but figured I'd start here because if I remove is and use regular std. lib primitives it reports the correct thing.

Reproducible example

I noticed this happening in https://github.com/pressly/goose and put together a PR to showcase the problem.

In this commit I purposefully changed a test to trigger a failure. Note the name of the test is TestNotAllowMissing.

pressly/goose@9b7c732#diff-080f0d7ba408eaf6181e1f61388995ebcd57fdd1468ffafffbf789176531418fR30

When I run go test -v -json -count=1 ./tests/e2e -dialect=postgres I happily get JSON output. But, the test name in the JSON event doesn't line up with the Output.

I was expecting Output that is prints to be associated with test named TestNotAllowMissing, but instead it was associated with another test named TestMigrateFull. (the name is non-deterministic, it changes between runs).

The raw output can be found here . But I've copied and trimmed the relevant bits:

{"Test":"TestMigrateFull","Output":"\t\u001b[90mallow_missing_test.go:30: \u001b[39m7 != 8\n"}
{"Test":"TestNotAllowMissing","Output":"--- FAIL: TestNotAllowMissing (3.73s)\n"}

The first JSON event refers to allow_missing_test.go:30, in the code base the call site on line 30 is located within the test named: TestNotAllowMissing, but the test name is TestMigrateFull.

The second JSON event correctly outputs the failure under the test named TestNotAllowMissing.

This could very well be a bug in how go test prints JSON events, but I suspect it may be something within this library. I say this because if I remove is from that specific test (TestNotAllowMissing) I see the expected output associated with the correct Test name:

Raw output can be found here

{"Test":"TestNotAllowMissing","Output":"    allow_missing_test.go:35: got 7: want: 8\n"}
{"Test":"TestNotAllowMissing","Output":"--- FAIL: TestNotAllowMissing (2.81s)\n"}

Release v1.5.0

I think it is a good time to make a new release 😉

add New and NewRelaxed methods to support sub tests

Adding New and NewRelaxed methods to the I object will allow subtests to work using the normal pattern:

func Test(t *testing.T) {
  is := is.New(t)
  t.Run("sub1", func(t *testing.T){
    is := is.New(t)
    is.Equal(1, 1) // should be equal
  })
}

Proposal: show var names in output

It would be cool to have variable names in the output.
Example:

is.Equal(signedin, true) // must be signed in

could be rendered like this:

is_test.go:175: not true: signedin<false> == true // must be signed in

Add a Contains method

Hi!
Watched the talk you gave on The Art of Testing and really liked it. That's why I'm trying out this package, and I really like its simplicity.

One thing though that I'm missing is a Contains method (basically this).

As there is none so far, I guess there is some other way to do it?
For example, I have a function that returns [][]string, and I wanna check if some []string are in that slice, but do not wanna specify the order.

Thanks!

Create release with MIT license

The current v1.0.0 release still contains the old GPL license. Could we create a new release with the new MIT license or move the existing release tag to include it?

My editor thinks this is bad practice

Hi Mat!

Big fan (of Go of course 😏), long time listener of GoTime and working with Go full time for 4 years next month!

So... I just tried this nice little assert library and the first thing I notice is that my editor thinks I shouldn't do this:

is := is.New(t)

It says: Variable 'is' collides with imported package name (And I hate to disappoint my editor!)

What is your defence? (Trying to be funny 😂)

Panics when comparing two nil slices or two nil maps

The test panics with panic: runtime error: comparing uncomparable type map[string]string if two nil slices or two nil maps are compared.

This can be simulated by adding the following test cases:

{
	N: "Equal(nilSlice1, nilSlice2)",
	F: func(is *I) {
		var s1 []string
		var s2 []string
		is.Equal(s1, s2) // nil slices
	},
	Fail: "",
},
{
	N: "Equal(nilMap1, nilMap2)",
	F: func(is *I) {
		var m1 map[string]string
		var m2 map[string]string
		is.Equal(m1, m2) // nil maps
	},
	Fail: "",
},

Consider using a more open license

Is there a specific reason why this project uses the restrictive GPL 3 license? I for one would be able to use it in a much more relaxed way if it used a more open license like MIT.

@matryer, would you consider switching?

Is the project still alive?

There are new commits, but no new release or tag for 2 years ...

Proposal: is.Less and is.Greater

Would you consider adding these functions to the API? I could see it being useful for numeric comparisons but also understand that you want to keep the API to a minimum.

Thanks.

Ditch the indentation?

Hey there, thanks for creating this package. I use it in all my tests. Would you be open to removing the tab in front?

The misalignment with other printed logs bugs me a bit (very subjective of course):

And with the colors, I think there's enough visual separation:

If you're up for this change, I'd be happy to open a PR.

Look for `// ` (with space) not `//`

Otherwise, URLs appear as comments

Lines should be escaped before printing

The following test produces an incorrect message when failing, as the lines passed to decorate() are not being escaped before being passed into log()'s Sprintf().

func TestFormatStringEscape(t *testing.T) {
	is := New(t)
	is.Equal("20% VAT", "0.2 VAT")
}

=== RUN   TestFormatStringEscape
	is_test.go:296: 20%!V(MISSING)AT != 0.2 VAT
--- FAIL: TestFormatStringEscape (0.00s)

New Release?

It seems like there hasn't been a release that uses the IS_NO_COLOR environment variable mentioned in the README. Would it be possible to tag a new release?

P.S. I enjoy listening to you and friends on Go Time! 😄

Proposal: value matcher interface in is.Equal

I sometimes have cases, where I would like to alter or extend the way is.Equal is deciding, if the values are considered equal. After trying several approaches, I came up with the following solution, which does not extend the current API surface, but adds a maximum of flexibility and freedom to the user in regards to how is.Equal decides if two inputs are equal.

The proposal is to value a newly defined matcher interface, which is defined as follows (the matcher interface does not need to become part of the public API of github.com/matryer/is, in fact it can just be defined inline where needed):

type matcher interface{
	Match(interface{}) bool
}

The actual change, that I am proposing is, to extend the existing areEqual function like this (lines 9-11):

// areEqual gets whether a equals b or not.
func areEqual(a, b interface{}) bool {
	if isNil(a) && isNil(b) {
		return true
	}
	if isNil(a) || isNil(b) {
		return false
	}
	if matcher, ok := a.(interface{ Match(interface{}) bool }); ok {
		return matcher.Match(b)
	}
	if reflect.DeepEqual(a, b) {
		return true
	}
	aValue := reflect.ValueOf(a)
	bValue := reflect.ValueOf(b)
	return aValue == bValue
}

In theory, this change does alter the working of is.Equal. In practice, I assume the chances of this change having negative side effects for users of this package to be extremely low. Only users, that use this package to compare types, that implement this exact interface would be affected. I rate the profit of this change to be way higher than the risk of negative side effects.
If you think, that the current signature (Match(interface{}) bool) is too likely to cause problems, the name of the method can easily be altered such that the chance of a collision become negligible (e.g. MatRyerIsMatch(interface{}) bool 😜).

If I find acceptance for this proposal, I am happy to provide the necessary PR to update the code, the tests and the documentation.

---

As a side note, the areEqual function could be minimally simplified by replacing:

	if isNil(a) && isNil(b) {
		return true
	}
	if isNil(a) || isNil(b) {
		return false
	}

with

	if isNil(a) || isNil(b) {
		return isNil(a) && isNil(b)
	}

Color escape codes break vim

vim can't handle terminal escape codes in compiler/test output, and vim-go does not strip them out. A failing test is rendered in the Quickfix window as:

^[[90mhtml_test.go|14| ^[[39merr: unexpected input^[[32m // should succeed^[[39m

And tries to open a file called ^[[90mhtml_test.go

Revisiting `is.Equal` and the role of `reflect.DeepEqual`

I started switching from google/go-cmp to is in pursuit of the latter's more pleasant and succinct syntax, but unfortunately have run into issues with how times are compared by is.

In the code being tested, a time is created using time.Now(), stored to a database, and then later retrieved. When comparing the two times (or the structs containing time values), is.Equal reports that the times are different because one has a monotonic counter and the other does not. This differs from how the standard library time.Equal() functions, which is recommend by the Go team instead of directly comparing time values.

For example, consider the following excerpt where time1 has a monotonic counter, and time2 is the same time without a monotonic counter:

time1 := time.Now()
time2 := time1.Round(0) // returns time1 stripped of any monotonic clock reading but otherwise unchanged (see https://pkg.go.dev/time#Time.Round)
fmt.Printf("time1.Equal(time2) = %v\n", time1.Equal(time2))
is.Equal(time1, time2)

which results in the following:

time1.Equal(time2) = true
test.go:25: 2023-03-10 19:18:46.3692787 -0500 EST m=+0.083328801 != 2023-03-10 19:18:46.3692787 -0500 EST

Is there a better way to handle these comparisons in is? Some options I can imagine include:

1. Switch to using time.Equal() if the values being compared are times.
2. Adopting a model similar to google/go-cmp where all "Types that have an Equal method may use that method to determine equality".
3. Use the tried-and-true google/go-cmp library itself to perform underlying equality comparisons.

-nocolor flag not supported by Go Test in go-1.11.4

go test without -nocolor works:

sam@macbook:~/go/src/github.com/sbward/s (master)
$ go test ./model/
	permissions_test.go:11: model.Permission(2) != int(1)
--- FAIL: TestPermissionBits (0.00s)
FAIL
FAIL	github.com/sbward/s/model	0.009s

But go test with -nocolor gives an error:

sam@macbook:~/go/src/github.com/sbward/s (master)
$ go test ./model/ -nocolor
flag provided but not defined: -nocolor
...

Add semver tag/release.

I would like to use the library, but it has no tag/release. Would be great to have semver tag (for vgo/dep, etc).
Thanks!

flag.Parse in init() prevents own flags in tests

With the commit 46663c1 the parsing of the flags has changed to the use of flag.Parse().
Because flag.Parse() is called in init(), it prevents the source package (the one, that uses github.com/matryer/is) from defining and using it's own flags, because the calls to e.g. flag.Bool(...) will be after the execution of flag.Parse() and therefore these flags are not considered and the execution of go test fails with flag provided but not defined: ....

This leaves us with two options:

1. Revert the above mentioned commit
2. Under the assumption, that this package is mainly (only) used for testing with go test, then it would also be save to remove the call to flag.Parse() in init(), because go test will execute flag.Parse()

Proposal: Use generics to add type safety support to is.Equal

I think it is quite common to got caught into pitfalls getting errors like "unit8(0) != int(0)" (by calling is.Equal with an untyped constant for example).

Now that Go has added generics support, it is easy to add type safety support to equal, like:

func (is *I) EqualSafe[T any](a, b T) {
    is.Helper()
    return is.Equal(a, b)
}

Small simplification

The following code

	if isNil(a) || isNil(b) {
		is.logf("%s != %s", is.valWithType(a), is.valWithType(b))
	} else if reflect.ValueOf(a).Type() == reflect.ValueOf(b).Type() {
		is.logf("%v != %v", a, b)
	} else {
		is.logf("%s != %s", is.valWithType(a), is.valWithType(b))
	}

can be simplified to

        if isNil(a) || isNil(b) || reflect.ValueOf(a).Type() != reflect.ValueOf(b).Type() {
		is.logf("%s != %s", is.valWithType(a), is.valWithType(b))
	} else {
		is.logf("%v != %v", a, b)
	}

Additional error assertations

Hi, let me first say that the library is great, thank you for creating it!

I find myself repeating few checks for errors that could be made into assertation.
It would be my pleasure to contribute this upgrades if we agree on their form.

1. is.NoErr(err) is great but in lots of situations I need to check the opposite, that error is present.
   Using is.True(err != nil) works but does not have the same approach to assertation.
   I suggest adding is.Err(err) that would assert err is not nil.

2. Usually, I add context to errors but I do not use custom error types as there is no behavior associated with this errors. When I have this situation, I would like to check that error contains a particular substring.
   I suggest adding is.ErrContains(err, "part of error message") that would assert if error contains given substring.

What do you think about this two additions?