Comments (8)
This is the correct behavior according to the Cucumber OSS community. This was also clarified here:
#68
Hook results count towards the overall status of the test case.
from cluecumber.
Then in the first instance I think it would be prudent to either request that feature from Karate or contribute it to the Karate project.
from cluecumber.
Thanks @bischoffdev . I missed mentioning that ours is a API test suite written in karate DSL.
And to my observations, it seems behavior is different in karate. A failure of afterScenario hook does not mark the test as failed in karate.
This seems to be the reason of discrepancies i am seeing in karate vs Cluecumber report.
Do you have any thoughts on how this can be handled?
from cluecumber.
Is this documented somewhere for Karate? If so, then I think there should be a dedicated configuration parameter like ignoreHooksFailure
. What do you think?
from cluecumber.
@bischoffdev I did not find any reference in karate documentation but this is the behavior we are currently observing. To give more context, we are using this after scenario hook from our config to publish a link to splunk and other logging tool. This afterhook fails when the scenario is aborted due to no response being found.
I really like your idea of a boolean flag so that it can be controlled by user based on their use case. It certainly gives flexibility to the testing teams.
from cluecumber.
Karate is not Cucumber though. Due to a shared historical origin they do happen to share a report format, but they're very different otherwise.
So I would suggest using the reports Karate provides. There are likely to be more accurate and always match Karate's semantics.
from cluecumber.
@mpkorstanje Agreed on karate not being cucumber. And we are using karate report itself. But karate report does not provide separate count for skipped scenarios. Because of that, we started exploring this project to get more granular report with separate count for skip.
The reason we want skip count is because of extensive use of karate.abort() function to handle different environment, test data, dynamic response limitation.
from cluecumber.
@mpkorstanje I fully agree that Karate is not Cucumber. And the first point of reference is certainly the original Cucumber JSON format. Karate's output format is so close to the Cucumber JSON format that it is not too much effort to support it as well.
In this special case, though, I would opt for not supporting it but indeed request a change on Karate side.
Karatelabs itself recommends not using karate.abort
in their example, so this is an absolute edge case:
(source: https://github.com/karatelabs/karate/blob/master/karate-demo/src/test/java/demo/abort/abort.feature)
from cluecumber.
Related Issues (20)
- The iFrame element shows empty content under HTML attachment Scenario Step HOT 7
- Adding status "Passed with warning" HOT 1
- Consolidated Scenario feature improvements
- Latest version is not present in maven central HOT 1
- Improve the filtering of "Search" fields HOT 2
- Separate steps per linked glue code and name (e.g. for Karate) HOT 1
- expandBeforeAfterHooks setting is ignored
- Introduce states on the scenario detail expansion buttons
- Deeply nested sub sections can break the layout HOT 1
- Upload 3+ releases to Maven Central HOT 3
- Possibility to easily define a favicon to all report web pages HOT 5
- Unable to generate cluecumber reports on test failures and unable to download reports HOT 1
- Query: How to add custom log steps in cluecumber reports for failed steps HOT 1
- Report generation very slow HOT 5
- Don't show failures of previous runs in the graphs unless expanded
- Test scenario is marked as 'Skipped' instead of 'Failed' when flag to 'failScenariosOnPendingOrUndefinedSteps' is enabled HOT 5
- Generate Reports Should Accepts Multiple json files HOT 1
- single-file html generation HOT 1
- Support for new cucumber message format (ndjson) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cluecumber.