Comments (7)
The problem is that Monkeyble works as a callback plugin. It's not aware of a side executions.
I can relate to the limitations of running within a callback interface, even though what we can do is sometimes surprising :p
@briantist it's probably not what you are looking for but once playbook task results are recorded in ara the data is available to query over its API.
If the data you need is in the results, you could find them and use them that way.
For example:
- a task result: https://demo.recordsansible.org/results/782089.html
- this task's API representation: https://demo.recordsansible.org/api/v1/results/782089
from monkeyble.
Hi,
Actually this kind of feature is mostly present in unit testing of real language. Because test are really independent.
In the context of monkeyble we it's end to end. So I'm not sure it would be pertinent to give this info as we cross the full playbook to validate a scenario. It means that, even if a task is skipped, you pass through it.
What do you have in mind exactly?
from monkeyble.
Sure that's a good point, and I'm not sure I have a lot of specifics in mind. But for example on a given run, skipping a task would probably be the equivalent of a line not being tested. Then ideally the combination of scenarios we test does end up covering all paths.
I suppose conditionals get a bit complex. If a task has a conditional, we want to test it with both true
(does execute) and false
(skips), so in that case I guess:
- a skip that's not because of a
when:
conditional is a miss - a skip due to a
false
conditional is a partial - an execution due to a
true
conditional is a partial - the combination of the above two is fully tested
I don't think we'd need to dig into the particulars of complicated conditionals (when
with a list for example), for this purpose I think we only care about the final result of it, if that makes sense.
from monkeyble.
And so Monkeyble already answer to that. You can test both scenario with true
or false
.
from monkeyble.
And so Monkeyble already answer to that. You can test both scenario with
true
orfalse
.
Right exactly, the idea is to generate some standard coverage report that we use in coverage tools like https://codecov.io , that makes it easier to see if we missed some pathways or something and want to test more scenarios
from monkeyble.
The problem is that Monkeyble works as a callback plugin. It's not aware of a side executions.
So we cannot know that we passed in a particular task before and that the result of the condition was true
or false
.
from monkeyble.
The problem is that Monkeyble works as a callback plugin. It's not aware of a side executions. So we cannot know that we passed in a particular task before and that the result of the condition was
true
orfalse
.
That makes sense, I'm not suggesting that a single report needs to be created that takes into account all of the other scenarios, rather each run would only have to be concerned with what that run covered, multiple scenarios results in multiple reports, and the coverage tool takes all of those into account.
I think that's how it usually works with other tools, though I am a bit out of my depth on the particulars.
from monkeyble.
Related Issues (9)
- The should_failed check should preferably read should_fail instead HOT 1
- 'Task' object has no attribute 'play' HOT 18
- ✨Feature Request - generate flow charts from test scenarios HOT 3
- CLI: add test name to be executed HOT 1
- CLI: add monkeyble list command HOT 1
- When task is skipped: test if it should have failed HOT 1
- mock - ansible.builtin.copy - different behaviour depening on whether the dest file exists or not
- Add extra_vars in scenario
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from monkeyble.