Prow issue where kind clusters are surviving into new test runs

[kubernetes-sigs]

Prow issue where kind clusters are surviving into new test runs

[johnSchnake]

Idea: I think that perhaps this was due to a situation where the gexe was starting some processes but not necessarily blocking to wait for them. As a result things like the creation/deletes wouldn't complete and there would be orphaned containers.

One way to mitigate this is just to do something in make test which will run a clean routine first. That way we can clear up any existing clusters.

[dims]

@johnSchnake isn't it being created here? https://github.com/kubernetes-sigs/e2e-framework/blob/main/klient/k8s/resources/main_test.go#L96

[johnSchnake]

So the issue isn't that I dont know where it got made in code, the point is that it exists on the host before it ever should, presumably from a previous run.

For instance, https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_e2e-framework/63/pull-e2e-framework-test/1448519284388007936#1:build-log.txt%3A46 is the first time in those tests that it is making a cluster but there are already 5 that show up as having been created (just a few lines below that linked spot)

As I am pasting all the "supporting confusion" I do see something that partially explains it though. Despite the logs being written the way they are, the timestamps show that all the Creating kind cluster ... code is getting executed near the same time.

So I thought that all of this was happening in series (I even saw a PR about wanting tests in parallel eventually), and the logs look like those creation events are in series separated by lots of time, they are actually all happening around the same moment.

This also helps explain why when I added more things like

• why does kind get installed multiple times? Because all the checks happen before kind finishes getting installed the first time
• why do we get the file is busy error even though we thought the tests we going in series? There seems to be more contention around the use of that file than I thought since the tests (or the setup at least) happens in parallel.

I think the issue is that the way it is invoked go test ... $(go list ./...); I'm guessing that the go tool runs each of those packages in parallel, but each package in series? Something like that.

[johnSchnake]

https://medium.com/@xcoulon/how-to-avoid-parallel-execution-of-tests-in-golang-763d32d88eec describes exactly the same issue; tests werent in parallel but go was running packages in parallel causing racy behavior.

This also explains why I can't repro locally (different GOMAXPROCS set I'm assuming)

Should get a fix for this soon if all I have to do is change -p 1