Allow Code review to fail without preventing simpletest to run.

[Decipher]

Allow Code review to fail without preventing simpletest to run.

[Decipher]

See https://github.com/squizlabs/PHP_CodeSniffer/wiki/Configuration-Options#ignoring-errors-when-generating-the-exit-code

[Decipher]

This is no longer the case with the change to DCR, but it should probably be configurable still.

[AlexSkrypnyk]

I suggest using the approach most of CI platforms use: each runner is independently run, but results are combined and returned for the whole build.

So even if code review fails, the other runner still runs, but the whole build fails.

Now, if you still like to have the whole build pass when some runners fail, there should be a per-runner result severity setting (error, warning, info) and DCIR would have a --fail-level=severity_level option to set desired fail-level. For example, if DCR issues warnings and --fail-level=error, the build would not fail. But if --fail-level=warning, it will fail.

[Decipher]

I think there is a legitimate case for a quick fail case, where if a single runner fails then CI exits, as otherwise you might end up waiting full run time to determine there was a failure within the first few seconds/minutes.

But I also believe it should be configurable.