expected failures

[nek5000]

expected failures

[maxhutch]

Great idea!

[RonRahaman]

The unittest.expectedFailure decorator breaks Twisted Trial's test runner (https://twistedmatrix.com/trac/ticket/4811), which is used in buildbot. I'd really like to maintain interoperability with buildbot, so the how should be mark the expected failures?

I know that the unittest.skip decorator works. This won't run the decorated test method at all, so the failure won't appear. Is this acceptable?

[maxhutch]

Looks like the fix in Twisted Trial went stale. unittest.skip seems ok for now, but can we leave this open with a blocked externally label or something?

[RonRahaman]

Upon consideration, I'm not satisfied with using unittest.skip. I'd like to always run the expected failures to see exactly how/why they fail. That information is valuable.

I think we can mark the expected failures at the level of buildbot (not at the level of unittest). Each TestCase is run by buildbot using a Trial BuildStep. Like all BuildSteps, there are parameters can control how failures and warnings are handled. For the TestCases that are expected failures, we can set flunkOnFailure=False and warnOnFailure=True. That way, the expected failures are still run, but if (or rather, when) they fail, buildbot will not mark the overall build as a failure. I think they'll also be colored yellow (instead of red) in the GUI.

What do you think?

[maxhutch]

Sounds good. Can we also annotate them in case someone is running locally?

On Wed, Jul 27, 2016, 06:04 Ron Rahaman [email protected] wrote:

Upon consideration, I'm not satisfied with using unittest.skip. I'd like
to always run the expected failures to see exactly how/why they fail. That
information is valuable.

I think we can mark the expected failures at the level of buildbot (not at
the level of unittest). Each TestCase is run by buildbot using a Trial
http://docs.buildbot.net/0.8.8/manual/cfg-buildsteps.html#trial
BuildStep. Like all BuildSteps, there are parameters
http://docs.buildbot.net/0.8.8/manual/cfg-buildsteps.html#common-parameters
can control how failures and warnings are handled. For the TestCases that
are expected failures, we can set flunkOnFailure=False and
warnOnFailure=True. That way, the expected failures are still run, but if
(or rather, when) they fail, buildbot will not mark the overall build as a
failure. I think they'll also be colored yellow (instead of red) in the GUI.

What do you think?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#7 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABd7pJ0g3kEsLvmVVyYyBCd2ebXKc_Zgks5qZ1djgaJpZM4JNeA6
.