Comments (6)
There are only 3 failure scenarios for the code of this plugin execution:
-
mandatory options are not defined
I believe that these should still be failures -
files to upload are not found
These only cause a failure if the step's command did not fail. While we can add an option to ignore these, I'm not sure you'd want to use it as if you have configured your plugin to upload results from your step and the results are not there, something is really wrong, don't you think? -
spurious command errors
I think that these are the ones you are most concerned about. The failure comes from theset -e
in bash and executables in the code path not being allowed to fail. This is a good practice in general on scripts, but - from what I'm seeing- the only such occurrence would be thecurl
call, and even then I'm not completely sure as it is part of theupload
function which is only called conditionally to absorb errors
Could you show an example of the failures you want to avoid (as much of them as you can)? It is likely we are missing something in the code or a use-case for the plugin :)
from test-collector-buildkite-plugin.
Hi, thanks for the quick turnaround!
The case in which I surfaced the issue was JUnit XML not being present. I'd still prefer at least an option to avoid hard-failing there because it gives me more freedom to mess with our test telemetry without worrying about breaking builds - e.g. if there's some individual test that does something weird that I don't know about. Something would be wrong, but I could find it in the logs without disrupting anyone else, at least during a rollout phase.
But yeah, the case I'm most concerned about is the case of the curl call failing. I don't know how to reproduce that easily but coworkers report that we've had issues before where we e.g. ran out of some quota on your side for Test Analytics and then builds started failing. If we believe that won't happen anymore, this issue probably isn't a blocker for us, but I'm not sure how to easily test that.
from test-collector-buildkite-plugin.
While working on a new functionality I actually tested what would happen if curl
fails with an exit code that is not 0 and the plugin succeeds and just shows an error in the output saying Error uploading, will continue
.
Another alternative is that you were not using the latest version of the plugin. In 1.4.0 we changed the way the curl
call is handled when failing to avoid the scenario you are describing and in 1.5.0 we fixed an incompatibility with Bash 4.3 that could cause you to see unbound variable
errors. Both of those versions were released in the last month so they are relatively new.
With all that said, I am closing this issue but if you have one scenario that is not being taken into account please re-open it and we'll continue to work on it :)
from test-collector-buildkite-plugin.
The case in which I surfaced the issue was JUnit XML not being present. I'd still prefer at least an option to avoid hard-failing there because it gives me more freedom to mess with our test telemetry without worrying about breaking builds - e.g. if there's some individual test that does something weird that I don't know about. Something would be wrong, but I could find it in the logs without disrupting anyone else, at least during a rollout phase.
I have a similar use case to this. My test results can be cached, and when they are, I don't produce any JUnit XML. I'd like to be able to soft_fail when the plugin doesn't find any JUnit XML.
from test-collector-buildkite-plugin.
My test results can be cached, and when they are, I don't produce any JUnit XML. I'd like to be able to soft_fail when the plugin doesn't find any JUnit XML.
That is indeed an interesting use-case I hadn't thought of @BoraxTheClean. I have created PR #48 with a possible solution
from test-collector-buildkite-plugin.
Thank you @toote !
from test-collector-buildkite-plugin.
Related Issues (18)
- Hook error disguises previous build error HOT 1
- readarray in the pre-exit hook doesn't work on Amazon Linux 2.
- path which includes symlink won't find any files
- Support configurable API endpoint URL for debugging purposes
- Hook error disguises previous build error (not fixed) HOT 3
- Support absolute paths HOT 1
- Multiple entires for 'files' HOT 5
- Too slow when there are many tests HOT 6
- missing-error ENV flag does not appear to be respected HOT 2
- Nested JUnit testsuite causes duplication HOT 5
- Annotation printing `\n` instead of newlines HOT 1
- Performance of test-collector is unacceptably slow. HOT 2
- error parsing report url
- If multiple steps use the plugin the step that runs last will be the only context reported.
- Messages with parentheses do not get reported correctly
- Unexpected output in logs
- Dependency Dashboard
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from test-collector-buildkite-plugin.