dave/courtney

Courtney is a coverage tool for Go

courtney's Issues

Doesn't handle switch statements

... Should work just like if.

Typo in readme

Just an FYI, someone using that might get caught up.
notificaitons

Github Action fails - error getting a locales package?

I'm not quite sure where this is coming from since GH is definitely not down:

here is the area where i install/run courtney:

      - name: Install courtney for coverage
        run: go install github.com/dave/courtney@latest

      - name: Run Tests
        run: |
          courtney ./tests ./controllers

and my output is now oddly:

Run courtney ./tests ./controllers
open  downloading github.com/go-playground/locales v0.[1](https://github.com/vulncheck/api/actions/runs/3166862695/jobs/5156898345#step:12:1)4.0: no such file or directory
Error reading files from  downloading github.com/go-playground/locales v0.14.0
github.com/dave/courtney/tester.(*Tester).processDir
	/home/runner/go/pkg/mod/github.com/dave/[email protected]/tester/tester.go:1[9](https://github.com/vulncheck/api/actions/runs/3166862695/jobs/5156898345#step:12:10)5
github.com/dave/courtney/tester.(*Tester).Test
	/home/runner/go/pkg/mod/github.com/dave/[email protected]/tester/tester.go:59
main.Run
	/home/runner/go/pkg/mod/github.com/dave/[email protected]/courtney.go:66
main.main
	/home/runner/go/pkg/mod/github.com/dave/[email protected]/courtney.go:44
runtime.main
	/opt/hostedtoolcache/go/1.19.1/x64/src/runtime/proc.go:250
runtime.goexit
	/opt/hostedtoolcache/go/1.19.1/x64/src/runtime/asm_amd64.s:[15](https://github.com/vulncheck/api/actions/runs/3166862695/jobs/5156898345#step:12:16)94
Test
main.Run
	/home/runner/go/pkg/mod/github.com/dave/[email protected]/courtney.go:67
main.main
	/home/runner/go/pkg/mod/github.com/dave/[email protected]/courtney.go:44
runtime.main
	/opt/hostedtoolcache/go/1.[19](https://github.com/vulncheck/api/actions/runs/3166862695/jobs/5156898345#step:12:20).1/x64/src/runtime/proc.go:250
runtime.goexit
	/opt/hostedtoolcache/go/1.19.1/x64/src/runtime/asm_amd64.s:1594

Option to exclude all generated files

There is a lot of code generator in go, and also there is a convention for the generated code to identify themselves as generated code. sometimes we simply want to skip the coverage for them, so it is a good idea to add a switch to ignore them.

This is very easy to detect see https://github.com/golang/lint/blob/85993ffd0a6cd043291f3f63d45d656d97b165bd/lint.go#L125

Bug excluding blocks

package a

func wrap(error) error

func a() error {
	var a bool
	var err error
	if err != nil {
		if a { // this line will not be excluded!
			return wrap(err) // *
		}
		return wrap(err) // *
	}
	return nil
}

Broken builds with new version

Travis-ci builds are breaking with the new version of courtney, because of using /tmp for temporary writing.

Logs: https://travis-ci.org/nkprince007/uipgo/builds/315662168

Fails on Go1.11 beta2 with GO111MODULE=on

Trying out vgo and courtney fails with errors similar to:

go: cannot find main module root; see 'go help modules'
/usr/local/go/src/os/user/lookup.go:53:9: undeclared name: lookupGroupId
FILENAME:13:7: could not import gopkg.in/volatiletech/null.v6 (cannot find package "gopkg.in/volatiletech/null.v6" in any of:
	/usr/local/go/src/gopkg.in/volatiletech/null.v6 (from $GOROOT)
	/Users/mehcode/.go/src/gopkg.in/volatiletech/null.v6 (from $GOPATH))

---

Really awesome tool here by the way. Been loving it.

Option for excluding directories

While using the godep tool, a vendor subdirectory is created. It should be possible to exclude that from tests.

Fails with recursive relative test paths

If I run courtney in my local checkout for a subdir recursively, it fails:

simon@simon-x220-deb:/media/ext4_data/Coding/go/src/github.com/syncthing/syncthing
$ courtney -short ./lib/...
Dir not found for ./lib

With a single package it runs but without result:

simon@simon-x220-deb:/media/ext4_data/Coding/go/src/github.com/syncthing/syncthing
$ courtney -short ./lib/db
No results

With the package spec from github it runs just fine producing results both recursively and not.

Option to exclude returns by context

When using contexts extensively, coverage tends to be flaky because different exit paths may be taken. Some sentry type selects look like this:

	select {
	case <-ctx.Done():
		return ctx.Err()
	default:
	}

however in the general case it's just on case in a bigger select statement:

	select {
	[...]
	case <-ctx.Done():
		return ctx.Err()
	}

Being able to filter that out would be really nice.

Add readme to the temporary dir

... explaining that it can be deleted

Text-based summary

First up, thanks for recommending this tool to me, Dave. It performs it's role very well!

The only gripe I have with it as it stands is that you either get a ton of output, or little to no output. Nothing much in-between. I'd be awesome if there was some output similar to what you get when you run something like $ go test -cover normally. There is of course $ courtney -v, but I sort of feel like the output you get from that is closer to what could be $ courtney -vv.

For example:

$ go test -cover (go list ./... | grep -v proto | grep -v vendor)
ok  	github.com/eidolon/console	0.004s	coverage: 96.6% of statements
ok  	github.com/eidolon/console/parameters	0.003s	coverage: 100.0% of statements
ok  	github.com/eidolon/console/specification	0.002s	coverage: 100.0% of statements

To me, this is useful for development when writing the tests, because I can just sit there with something watching every 5 seconds or so running the tests and get feedback about how my tests are going in an easy to parse way.

vs. well, I had to make a gist for it:

https://gist.github.com/SeerUK/0b256343d9d0ee2e8dff3b5e2eef4500

If I get a chance and can grok it, I might take a look at implementing this myself, but would also like your input on how you would see something like that being used (in terms of what flag you'd need to pass in).

go module support

Heyho,
I'm trying to run this on a go module and it doesn't seem to find any of the packages.

Here's the output of go test vs courtney

lenny@localhost ~/g/xxxx> go test ./...
? gitlab.com/xxx/yyy/xxxx/cmd/service [no test files]
? gitlab.com/xxx/yyy/xxxx/config [no test files]
? gitlab.com/xxx/yyy/xxxx/model [no test files]
ok gitlab.com/xxx/yyy/xxxx/pkg/extractparsers 0.011s
ok gitlab.com/xxx/yyy/xxxx/pkg/filerule 0.005s
? gitlab.com/xxx/yyy/xxxx/pkg/http/rest [no test files]
? gitlab.com/xxx/yyy/xxxx/pkg/http/rest/context [no test files]
? gitlab.com/xxx/yyy/xxxx/pkg/http/rest/middleware [no test files]
? gitlab.com/xxx/yyy/xxxx/pkg/log [no test files]
ok gitlab.com/xxx/yyy/xxxx/pkg/rule 0.006s
ok gitlab.com/xxx/yyy/xxxx/pkg/zapi 0.004s
ok gitlab.com/xxx/yyy/xxxx/util 0.003s

lenny@localhost ~/g/xxxx> courtney ./...
Package not found for /home/lenny/gits/xxxx

Package not found

karl@magic:~/Projects/go-concise-encoding$ go get -u github.com/dave/courtney
go: finding github.com/dave/courtney v0.3.0
go: downloading github.com/dave/courtney v0.3.0
go: extracting github.com/dave/courtney v0.3.0
go: finding github.com/dave/patsy latest
go: finding github.com/dave/brenda v1.1.0
go: finding github.com/dave/astrid latest
go: downloading github.com/dave/patsy v0.0.0-20170606133301-2245ba804d71
go: downloading github.com/dave/brenda v1.1.0
go: downloading github.com/dave/astrid v0.0.0-20170323122508-8c2895878b14
go: finding golang.org/x/tools latest
go: downloading golang.org/x/tools v0.0.0-20200606014950-c42cb6316fb6
go: extracting github.com/dave/patsy v0.0.0-20170606133301-2245ba804d71
go: extracting github.com/dave/brenda v1.1.0
go: extracting github.com/dave/astrid v0.0.0-20170323122508-8c2895878b14
go: extracting golang.org/x/tools v0.0.0-20200606014950-c42cb6316fb6
go: finding github.com/pkg/errors v0.9.1
go: downloading github.com/pkg/errors v0.9.1
go: extracting github.com/pkg/errors v0.9.1
karl@magic:~/Projects/go-concise-encoding$ go test
PASS
ok  	github.com/kstenerud/go-concise-encoding	0.093s
karl@magic:~/Projects/go-concise-encoding$ courtney 
Package not found for /home/karl/Projects/go-concise-encoding

allow adding extra context to `notest` comments

having the ability to exclude code from being tested is a must for getting some value out of the test coverage, so I'm glad I've found this package after browsing the go issue tracker and finding multiple issues asking for this feature to be supported natively

one of the things that I like a lot about golangci-lint is the encouragement to add some additional context when suppressing a linter, feature which we're using constantly, as it's much easier to know at a glance why we're explicitly deciding that a linter is not important for a specific piece of code

when marking code as not important to be covered by test, via de notest comment, I'd also like to be able to supply some additional context, as we'll know not only what code we explicitly decided not to test, but also some context about why that decision was taken, preventing unnecessary guesswork and discussions when going through the codebase and finding chunks of code marked as excluded from the coverage

example

var CmdListDicts = cli.Command{
	Name:        "list",
	// notest // glue code, mostly error handling and some cli sugarcoat
	Action: func(ctx *cli.Context) error {
		...
	}
}

Update Dependencies for golang 1.20

After updating from golang 1.19.6 to 1.20.1 I was unable to run tests with courtney because I received errors like this:

internal error: package "encoding/json" without types was imported from "<<MY PACKAGE NAME>>"

I reverted to 1.19.6 as a workaround and that stopped the errors. Search results suggest that others have resolved similar errors by building with a more current version of dependencies like golang.org/x/tools, for example:

golang/go#37617

Thank you for this project, I find it very helpful!

Should not unwrap /... when running tests?

"notest" comment does not work in my setup

Hi, it's great to find this tool.
But after trying for hours, I'm still having trouble to have it work.

pac.go: (you can see I have tried adding "notest" everywhere..

package newpac

import "fmt"

func max(a, b int) int {
	if a > b {
		// notest
		fmt.Println("gt") // notest
		return a          // notest
	} else {
		fmt.Println("lt")
		return b
	}
}

pac_test.go

package newpac

import (
	"fmt"
	"testing"
)

func Test1(t *testing.T) {
	res := max(3, 5)
	fmt.Println(res)
}

Run command:

go install github.com/dave/courtney@latest
go install github.com/axw/gocov/gocov@latest
go install github.com/AlekSi/gocov-xml@latest
go install github.com/matm/gocov-html@latest

courtney -v -o c.out && gocov convert c.out | gocov-html > C.html 

But still getting same report as before,

My Go version is 1.17, and I have downgraded to 1.16 but still got same result. Am I missing anything?

Where to download the executable and checksum?

I would like to download an executable and check the checksum. E.g. https://github.com/030/go-yq/releases/tag/2.0.0

Add mode to ingest pre-prepared coverage files

@haya14busa:

Why not accepting coverage report text files instead of wrapping go test and executing tests?
I like the concept but I don't want to replace go command as much as possible.

Feature: % of code coverage

Hi Dave, thanks for Courtney.

Quick question, is it possible to specify a % of code against the enforce command? I'd like to use this option in an automated build, but I dont want to enforce 100% coverage.

How to enforce covrage

Hi,

Very nice repo!

1. Is there a way to enforce coverage like above 90% otherwise the build should failed (travis) ?
2. when I use your tool it generate a new file with the following output

mode: set go-myproject/main.go:10.13,11.46 1 0 go-myproject/main.go:11.46,14.3 2 0

what does it mean and how should I run it ?

Thanks

Thank you

Hello @dave,

just wanted to thank you for courtney, greatly appreciated!

marco