die() in tested class still result in "Success"

[atoum]

die() in tested class still result in "Success"

[mageekguy]

There is no way to handle direct call to die() or exit() in a tested class.
The only solution is to use a dedicated public method to stop the script and mock it in the test.

stop('Stop because i want !'); } } } namespace vendor\project\tests\units; class foo extends \mageekguy\atoum\test { public function testSoSomethingAndCallStop() { $this->mockTestedClass(); ``` $foo = new \mock\vendor\project\foo(); $foo->getMockController()->stop = function() {}; $foo->doSomethingAndCallStop(); $this->assert ->mock($foo)->call('stop')->withArguments('Stop because i want !')->once() ; } ``` } ?>

[tentacode]

Just to be sure, I don't want to test the call to die(), I want to prevent someone to call a die() somewhere and still my CI is green.

[ratibus]

But how can you return a global Success status though the status line prints 3/5 ?

[mageekguy]

"Success" means "no fail, no error and no exception".
Currently, it does not mean that all test methods was executed.

[tentacode]

Maybe adding a "--strict" option a la Behat, where skipped test returns Error could solve this issue ?

[ratibus]

Yep I see.
When you say "Currently", do you mean that you consider it as a bug (I do) ?
In fact there can be multiple exit scenarii that you cannot intercept like fatal errors, that lead to such a case.

[mageekguy]

When you say "Currently", do you mean that you consider it as a bug (I do) ?

Not sure that it's a bug, see below.

In fact there can be multiple exit scenarii that you cannot intercept like fatal errors, that lead to such a case.

atoum can handle fatal error and it's a failure.
atoum can also handle segfault without any problem, but it's not a failure, because it's a PHP problem (bug in the Zend Engine), not a code problem (not a bug in the tested class).

I think that it's a semantic problem.
What means exactly "SUCCESS" ?
In my current opinion, "SUCCESS" = 0 fail, 0 error, 0 exception.
Segfault is a problem, but it's not a test failure.
Direct call to die() or exit() are in fact similar to segfault : the php script stop, and it's not a test failure.

So, perhaps there is not enough visibility in the report about the fact that some test methods stop "abnormaly".

try this :

It's a failure ? Not sure...
Currently, atoum say :

atoum version DEVELOPMENT by Frédéric Hardy (/Users/fch/Atoum/repository)
PHP path: /usr/local/bin/php
PHP version:
=> PHP 5.3.8 (cli) (built: Sep 21 2011 23:14:37)
=> Copyright (c) 1997-2011 The PHP Group
=> Zend Engine v2.3.0, Copyright (c) 1998-2011 Zend Technologies
=> with Xdebug v2.1.1, Copyright (c) 2002-2011, by Derick Rethans
mageekguy\atoum\tests\units\adapter...
[.___________________________________________________________][0/1]
=> Test duration: 0.01 second.
=> Memory usage: 0.00 Mb.
Total test duration: 0.01 second.
Total test memory usage: 0.00 Mb.
Code coverage value: 0.00%
=> Class mageekguy\atoum\adapter: 0.00%
==> mageekguy\atoum\adapter::__call(): 0.00%
==> mageekguy\atoum\adapter::invoke(): 0.00%
Running duration: 0.16 second.
Success (1 test, 1 method, 0 assertion, 0 error, 0 exception) !

[ratibus]

According to me, Success means : every test passed.
One test that did not run is a failure. What is the purpose of this test if when it does not run, the global status is still ok ?
I write tests for them to be run :)

[mageekguy]

I write tests for them to be run :)

But it can't run with a die() or a exit() in it...
So, if there is a die() in it, it's certainly because you don't want run it, so it's not a failure.

[marmotz]

if all tests was run and passed => Success
if one or more tests wasn't run or not passed => Fail (or Error, or Warning, or whatever)

[ratibus]

In fact we put a die in it to debug something. And we saw that the test still pass. That's why we opened this issue in the first place :D
The test should have failed. That surprised us it did not.

[mageekguy]

Currently, a die() in a test method is equivalent to an empty test method.
And an emtpy test method is not a failure.
So i have a problem, because this case is like the schrodinger cat : it may be failure, or not.

[ratibus]

I see your problem :)
But what is the point of an empty test method ? According to me it should be considered as a failure.

[mageekguy]

But what is the point of an empty test method ? According to me it should be considered as a failure.

Why ?
No assert = no failure = success.

[ratibus]

No assert = useless = failure :)
Different point of view :D

[mageekguy]

No assert = useless = failure :)
Different point of view :D

Yes, just need to know if all users are the same point of view than you.

FYI, It's not a problem to change this behaviour.
Just need to add a "status" in the score to handle these cases (segfault, direct call to die()/exit(), empty test method).
Currently, there are "fails", "errors" and "exceptions" in score.
But i have no idea for the name of the new status.
If you have a suggestion...

[marmotz]

"problem" ?

[ratibus]

"not completed" ?

[marmotz]

+1

[mageekguy]

I have find during the name, It will be "uncompleted".