Comments (18)
As that StackOverflow thread pointed out, it's possible to use the -p=1 flag to force the maximum number of packages to be ran in parallel (with 1 effectively serializing it all). This doesn't seem to be a problem with testify - but rather, just how go test works.
Another thread for this issue is here:
http://stackoverflow.com/questions/23715302/go-how-to-run-tests-for-multiple-packages
from testify.
I guess it wouldn't hurt to document this in the godoc or README or something. Looks like some people already stumbled into this and where clueless. Me included.
from testify.
Interesting - is this a bug or feature? :)
On May 2, 2014, at 2:15 PM, seantalts [email protected] wrote:
If use go test on all of your packages at once and there are multiple test suites, the suites will end up running in parallel with other suites.
—
Reply to this email directly or view it on GitHub.
from testify.
I think it's a bug, as normally go test will only execute tests in parallel if they mark themselves as parallel by calling *testing.T.Parallel().
from testify.
I've been running in to this lately, and have been trying to find a fix. Work on it is slow, though - I'm swamped.
from testify.
Thanks for looking in to this, @nelsam :)
from testify.
I believe our team is running into this too today.
Basically I created a new test file in a particular package with some barebone test structure - no actual tests...just an empty struct type that embeds suite.Suite, and a function that takes in a *testing.T object and calls suite.Run() on said struct. This immediately caused all our other tests to start failing indeterministically. The nature of the failures were associated with database unique key integrity violations on inserts and deletes into a single Postgres DB. This is leading me to believe that the tests were being run concurrently without calling our setup methods to prepare the environment properly between tests. Needless to say, the moment I move this test file to another package, everything magically works!
We are not choosing to run any of the tests concurrently.
from testify.
Would an option to run synchronously be the solution?
from testify.
The issue really boils down to the fact that I don't know what's causing suites to run in parallel. I've done some research (again, I'm swamped), but haven't really gotten much worthwhile about what's causing it.
Technically, tests that can be run in parallel should call t.Parallel()
, but all suite methods are being run in parallel without having that called.
I'll see what I can find out today, but no promises. This is on my priority list, but it's below several other things I'm working on.
from testify.
@tylerb how about two eyes on it?
On Oct 29, 2014, at 16:51, nelsam [email protected] wrote:
The issue really boils down to the fact that I don't know what's causing suites to run in parallel. I've done some research (again, I'm swamped), but haven't really gotten much worthwhile about what's causing it.
Technically, tests that can be run in parallel should call t.Parallel(), but all suite methods are being run in parallel without having that called.
I'll see what I can find out today, but no promises. This is on my priority list, but it's below several other things I'm working on.
—
Reply to this email directly or view it on GitHub.
from testify.
They're running in parallel because of the implementation of golang's
testing.Run that testify's suite.Run calls. It spawns a new goroutine and
communicates with it over a channel.
HTH, on a bus sorry.
On Oct 29, 2014 5:53 PM, "Mat Ryer" [email protected] wrote:
@tylerb how about two eyes on it?
On Oct 29, 2014, at 16:51, nelsam [email protected] wrote:
The issue really boils down to the fact that I don't know what's causing
suites to run in parallel. I've done some research (again, I'm swamped),
but haven't really gotten much worthwhile about what's causing it.Technically, tests that can be run in parallel should call t.Parallel(),
but all suite methods are being run in parallel without having that called.I'll see what I can find out today, but no promises. This is on my
priority list, but it's below several other things I'm working on.—
Reply to this email directly or view it on GitHub.—
Reply to this email directly or view it on GitHub
#52 (comment).
from testify.
Found it.
It basically looks like testing.RunTests and testing.InternalTest are not intended for use externally. I'll see what I can do. I think our original version just walked through methods calling them individually (instead of adding them to a slice and calling testing.RunTests), and I think that would be a better choice than what we currently have; however, I'd love it if I could get individual test methods to respect both testing.T.Parallel() and the -parallel
flag.
For the time being, I'll probably just strip out the testing.InternalTest and testing.RunTests usage, because (as I mentioned previously) I'm swamped and don't have much time to dedicate to this. However, I'll keep an item on my todo list to figure out a way to make use of the parallel test stuff. I'm pretty sure GoCheck works with those flags, but their solution is pretty complicated, and I think I'd rather have a simple solution than a complete solution. Anyone who really needs that functionality can probably just use GoCheck.
EDIT: Actually, I think I have a better idea. Sorta. Maybe. We'll see how much it breaks things.
from testify.
... Argh.
Okay, so, I was pretty sure I had found the problem, but I was (to put it simply) wrong. I added channels to suite.Run() to ensure that only one instance of suite.Run() was calling testing methods at a time, and then I just removed everything so that suite.Run() called every method in sequence (without using testing.InternalTest or testing.RunTests). Same issue, both times.
My boss found this, which suggests that the issue isn't exclusive to our suites: http://stackoverflow.com/questions/15721238/go-serial-execution-of-package-tests
It looks like relying on serial execution when using go test ./...
just isn't supported without tweaking. I'm going to try to find a way to swap things around so that I can use transactions, because I still want the queries to actually run against my database engine (for checking query syntax); that might require a pretty big change to my application code, but we'll see.
from testify.
Can confirm @MysticHLE 's solution works. Go documentation matches: https://golang.org/cmd/go/#hdr-Compile_packages_and_dependencies
-p n
the number of programs, such as build commands or
test binaries, that can be run in parallel.
The default is the number of CPUs available, except
on darwin/arm which defaults to 1.
Ticket should be closed I think.
from testify.
just -parallel 1
may not be sufficient, see golang/go#19280
Looks like in the next release there will be a parallel=-1
to make tests sequential https://go-review.googlesource.com/c/go/+/44530
from testify.
How to reproduce this problem, is there a gist? I cannot reproduce by
package main
import (
"testing"
"time"
"github.com/stretchr/testify/suite"
)
type ExampleTestSuite struct {
suite.Suite
}
func TestExampleTestSuite(t *testing.T) {
suite.Run(t, new(ExampleTestSuite))
}
func (s ExampleTestSuite) TestSimple() {
time.Sleep(2*time.Second)
s.True(true)
}
type ExampleTestSuite2 struct {
suite.Suite
}
func TestExampleTestSuite2(t *testing.T) {
suite.Run(t, new(ExampleTestSuite2))
}
func (s ExampleTestSuite2) TestSimple1() {
time.Sleep(2*time.Second)
s.True(true)
}
It executes in 4 seconds.
from testify.
How to reproduce this problem, is there a gist? I cannot reproduce by
package main import ( "testing" "time" "github.com/stretchr/testify/suite" ) type ExampleTestSuite struct { suite.Suite } func TestExampleTestSuite(t *testing.T) { suite.Run(t, new(ExampleTestSuite)) } func (s ExampleTestSuite) TestSimple() { time.Sleep(2*time.Second) s.True(true) } type ExampleTestSuite2 struct { suite.Suite } func TestExampleTestSuite2(t *testing.T) { suite.Run(t, new(ExampleTestSuite2)) } func (s ExampleTestSuite2) TestSimple1() { time.Sleep(2*time.Second) s.True(true) }It executes in 4 seconds.
try to put suites in different files
from testify.
It's how the go test works. Please see this answer.
Yes, as other answers already pointed, tests inside one _test.go file are running in parallel only if you add t.Parallel() and run with -parallel tag.
BUT - by default the command go test runs in parallel tests for different packages, and default is the number of CPUs available (see go help build)
So to disable parallel execution you should run tests like this go test -p 1 ./...
from testify.
Related Issues (20)
- Assert: go test output shows subtests that fail as PASS HOT 3
- `YAMLEq` does not validate YAML HOT 1
- Is it possible to assert for multiple allowed values? HOT 2
- Proposal: guard or support comparing with untyped nil HOT 5
- assert/require.Len doesn't print anything if the slice is too long HOT 7
- Allow user to skip (ignore) specific caller frames in assert.CallerInfo() HOT 1
- Unexpected call when interface is parameter HOT 4
- How can I get coverage in the Suite package HOT 6
- Fix CVE-2022-28948 - Remove `gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c` HOT 3
- assert: Equal does not consider Zone information in time.Time
- Update Github actions workflows from nodejs 16 to nodejs 20 before "Spring 2024" HOT 1
- mock: Call.NotBefore still expects calls removed with Call.Unset HOT 6
- unnecessary/incorrect log call in AssertExpectations HOT 2
- Data race when mock is called with refernce to same object as expectation HOT 1
- assert fails and expects to dereference a reference HOT 1
- Add GoLang 1.22 to CI Testing Matrix for Package Compatibility Verification HOT 2
- error while importing github.com/stretchr/testify/suite: read C:\Program Files\Go\src\math\huge_test.go: unexpected NUL in input HOT 1
- AnythingOfType is marked deprecated on pkgsite HOT 1
- .On().Return() doesn't enforce expectation for the returned value HOT 1
- v1.9.0 breaks float comparisons with EqualValues HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from testify.