Starscream shows successfully retried tests as errors

[cfournie]

Starscream shows successfully retried tests as errors

[wvanbergen]

How are JUnitXML files generated in your stack?

[cfournie]

py.test creates it for us after it's finished running our tests (which we call in our test runner).

[wvanbergen]

In Ruby's MiniTest, we had to override some behaviour so it treats requeued/retried tests as "skipped" to make all the tooling that is unaware of retries work out of the box. My python knowledge and skills are not enough to know whether that is feasible in your case. What do you think?

[cfournie]

We mark as skipped and requeue:

ci-queue/python/ciqueue/pytest.py

Lines 95 to 99 in 7a601e0

	# Only attempt to requeue if the test failed.
	# The method will return `False` if the test couldn't be requeued
	if test_failed and self.queue.requeue(test_name):
	outcomes.mark_as_skipped(call, item, self.terminalreporter.stats, "WILL_RETRY")
	self.terminalwriter.write(' WILL_RETRY ', green=True)

But I guess not well enough because py.test is still writing out a failure :|

[solackerman]

We modify the stats to look like the test never happened, but we don't remove the test itself.

ci-queue/python/ciqueue/_pytest/outcomes.py

Lines 78 to 82 in 7a601e0

	def clear_out_stats(key):
	if key in stats:
	stats[key] = [i for i in stats[key] if i.nodeid != item.nodeid]
	if not stats[key]:
	del stats[key]

[jules2689]

I think we need to override py tests' Junit then. JUnit output will be very important to CIs and to additional user experience. We're currently writing a command that will pull junit artifacts to run failed tests locally, for example. So I think it's in our best interest to invest some time into making this better in python-land.

@sabidib might be interested in this as I was talking to him about JUnit being required for this.