Support soft failures

[buildkite-plugins]

Support soft failures

[toote]

There are only 3 failure scenarios for the code of this plugin execution:

• mandatory options are not defined
  I believe that these should still be failures

• files to upload are not found
  These only cause a failure if the step's command did not fail. While we can add an option to ignore these, I'm not sure you'd want to use it as if you have configured your plugin to upload results from your step and the results are not there, something is really wrong, don't you think?

• spurious command errors
  I think that these are the ones you are most concerned about. The failure comes from the set -e in bash and executables in the code path not being allowed to fail. This is a good practice in general on scripts, but - from what I'm seeing- the only such occurrence would be the curl call, and even then I'm not completely sure as it is part of the upload function which is only called conditionally to absorb errors

Could you show an example of the failures you want to avoid (as much of them as you can)? It is likely we are missing something in the code or a use-case for the plugin :)

[djudd]

Hi, thanks for the quick turnaround!

The case in which I surfaced the issue was JUnit XML not being present. I'd still prefer at least an option to avoid hard-failing there because it gives me more freedom to mess with our test telemetry without worrying about breaking builds - e.g. if there's some individual test that does something weird that I don't know about. Something would be wrong, but I could find it in the logs without disrupting anyone else, at least during a rollout phase.

But yeah, the case I'm most concerned about is the case of the curl call failing. I don't know how to reproduce that easily but coworkers report that we've had issues before where we e.g. ran out of some quota on your side for Test Analytics and then builds started failing. If we believe that won't happen anymore, this issue probably isn't a blocker for us, but I'm not sure how to easily test that.

[toote]

While working on a new functionality I actually tested what would happen if curl fails with an exit code that is not 0 and the plugin succeeds and just shows an error in the output saying Error uploading, will continue.

Another alternative is that you were not using the latest version of the plugin. In 1.4.0 we changed the way the curl call is handled when failing to avoid the scenario you are describing and in 1.5.0 we fixed an incompatibility with Bash 4.3 that could cause you to see unbound variable errors. Both of those versions were released in the last month so they are relatively new.

With all that said, I am closing this issue but if you have one scenario that is not being taken into account please re-open it and we'll continue to work on it :)

[BoraxTheClean]

The case in which I surfaced the issue was JUnit XML not being present. I'd still prefer at least an option to avoid hard-failing there because it gives me more freedom to mess with our test telemetry without worrying about breaking builds - e.g. if there's some individual test that does something weird that I don't know about. Something would be wrong, but I could find it in the logs without disrupting anyone else, at least during a rollout phase.

I have a similar use case to this. My test results can be cached, and when they are, I don't produce any JUnit XML. I'd like to be able to soft_fail when the plugin doesn't find any JUnit XML.

[toote]

My test results can be cached, and when they are, I don't produce any JUnit XML. I'd like to be able to soft_fail when the plugin doesn't find any JUnit XML.

That is indeed an interesting use-case I hadn't thought of @BoraxTheClean. I have created PR #48 with a possible solution

[BoraxTheClean]

Thank you @toote !