Comments (11)
Reproduce with:
mkdir green-60
cd green-60
git clone --depth 1 --branch green-60 https://github.com/krisztianfekete/lib.git
cd lib
./test
from green.
I have managed to reproduce it with a smaller code, see testing-cabal/testtools#144
Even if the original problem is elsewhere I still think green
needs a fix too, as I get no help from it diagnosing the problem.
from green.
Ah, thank you! The smaller code helps. Sorry for the slow response, I have been swamped by life in general at the moment.
I agree, this should definitely be handled by green.
from green.
Okay, I finally got some time to sit down and look at this. Unfortunately, I can't reproduce your issue!
What am I missing?
I'm using this code (adapted from your post you linked to above):
# test.py
import unittest
class Test_testtools(unittest.TestCase):
def test_1(self):
pass
def test_2(self):
raise SystemExit(0)
def test_3(self):
pass
$ green test.py
.E.
Error in test.Test_testtools.test_2
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "test.py", line 9, in test_2
raise SystemExit(0)
SystemExit: 0
Ran 3 tests in 0.001s
FAILED (errors=1, passes=2)
from green.
the first code fragment is the one - with testtools in it, as testtools causes the problem:
# test.py
import unittest
import testtools
class Test_testtools(testtools.TestCase):
def test_1(self):
pass
def test_2(self):
raise SystemExit(0)
def test_3(self):
pass
if __name__ == '__main__':
unittest.main()
from green.
It appears to be a conscious design choice that testtools will consider KeyboardInterrupt
and SystemExit
as being outside the scope of something that you want to test. In other words, they are letting those exceptions go through to the main python interpreter on purpose.
If you would like, you may review the relevant testtools code yourself (in their current master branch) at the following locations:
testtools/runtest.py:217-219 # Explanation of intent to re-raise certain exceptions
testtools/test_testcase.py:906-915 # Demonstrates they expect SystemExit to escape TestCase.run()
testtools/matchers/_exception.py:81-121 # Actual location that implements re-raising SystemExit and KeyboardInterrupt
That is contrary to the design used by unittest, unittest2, py.test, trial, nose, etc. -- at least as far as I am aware, all of the other tools that provide their own TestCase subclasses have designed them to catch all exceptions. Green relies on the internal exception-handling of those subclasses as part of the "Traditional" aspect of things (see green's feature list on the readme).
Since this is a design choice that testtools has made, for whatever reason, I'm not going to start battling them by overriding their behavior when tests are run by green. Going down that road would eventually lead to me battling other behavior of other tools and eventually creating ill will among their users and developers. Definitely not something that's worth it to me.
So, I think your options for handling it in your own code are:
- Catch the
SystemExit
in your own test. - Ask testtools to catch
SystemExit
like most other TestCase implementations do. - Stop using
testtools.TestCase
I recommend number 1. Quick, easy, and doesn't pick fights with anyone's design decisions.
Good luck!
from green.
I am disappointed, maybe I have failed to convey my problem.
Let me try again:
I expect from a test runner
- to run tests
- provide feedback
- show information about failures
I have presented a test case, where all three above was compromised. This is clearly a problem.
Whatever caused the failure, green has failed as well.
The whatever in this case is a widely used test library, these are testtools' statistics on PyPI at the time of writing:
Downloads (All Versions):
10394 downloads in the last day
76699 downloads in the last week
240422 downloads in the last month
testtools unexpectedly from such a library breaks compatibility with its ancestors (L from SOLID).
green
was not robust to handle the deviance.
Can I trust green (unittest, etc) anymore when third party libraries as widely used as testtools can cause them to fail silently? Zero exit status, no failure, no traceback!
When I found this problem it was due to a failing test: a library (argparse
) unexpectedly for me raised SystemExit. I did not expect SystemExit.
"My options":
- Do you seriously propose to catch all errors in every test case?
- testing-cabal/testtools#144
- it is hard, as it is very nice to use with fixtures, but this option did occur to me
it looks quite funny (or not) when applied to the test in question and green
;-) :
- Do not write such a test case or carefully read all the library sources/documentations that you use and be aware of all strange design decisions, test libraries, frameworks included
- Ask
green
to gracefully handle failures in customTestCase
implementations (#60) - Stop using
green
Option 1 is DRY and assume and follow convention.
Option 2 is being nice.
I do not like option 3.
from green.
I agree that catching it in your one test won't prevent this situation from occurring again in other tests.
Catching the specific instance of SystemExit
that is causing you problems would be a good, pragmatic solution to your current problem test in the short term.
Solving it in the long-term is always better.
It would be great for the testtools to conform to standard python TestCase behavior so that other python testing tools, including alternate test runners like green, will not have to each individually compensate for their behavior.
The testtools project appears to have established a good pattern of accepting pull requests. If you proposed a specific change in a pull request, maybe that would be enough to get them to change the behavior.
from green.
Patching it in testtools will be hard and time consuming, it looks like a conscious decision and they even defended that decision over the years.
But even if it is patched green does not print stack trace next time. Would you accept the pull request that prints a stack trace?
from green.
I think your idea has merit. Global handling of any unanticipated exceptions would be an excellent way to handle any corner cases caused by poorly-coded testing frameworks (including my own, if I fall into the same trap). I will move over to the pull request and continue commenting there, instead of on this issue.
from green.
Version 1.10.0 was just released which contains the change that addresses this (and more...)
from green.
Related Issues (20)
- Feature Request: unittest timeout HOT 5
- Python 3.9.6 threading compatibility HOT 14
- django.db.utils.ProgrammingError: relation "users_user" does not exist HOT 3
- Combine methode name and doc string in output HOT 2
- Question: What is a "test" in context of --processes HOT 1
- SyntaxError not catched when named explicte but unittest does HOT 4
- Confusing error when I have errors in source code HOT 4
- Problem with -r (coverage) output HOT 1
- Missing git tag for 3.4.2 release HOT 1
- green might hang on async tests HOT 3
- shutil cleanup sometimes fails on python 3.8 HOT 5
- running green -vvv green.test.test_runner on PyPy + macOS errors out HOT 2
- Feature request: option to debug tests
- Python 3.12 support HOT 4
- Join The Project Team HOT 7
- stopTest issue with Python 3.12.1 and skipped tests HOT 15
- Move Release Process to GitHub Actions HOT 7
- Please add me to the project as a maintainer HOT 2
- Fix lack of coverage in CI HOT 3
- Change default branch on coveralls to "main" HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from green.