hewlettpackard/monkeyble

End-to-end testing framework for Ansible

monkeyble's Issues

✨Feature Request - generate flow charts from test scenarios

This is kind of a half-baked idea, but monkeyble seems uniquely positioned to generate a flow chart of a play/playbook/role, which could be a really nice addition to role documentation and such.

It could use Mermaid syntax, use the task names, represent looping and branching and such?
So maybe for example, you run several scenarios against a specific role or playbook, and a result a flowchart can be generated that shows various paths through the role.

An example of mermaid (quote or edit this post to see the syntax, better examples are out there):

graph TD
    A[Thievery] -->|Get money| B(Go shopping)
    B --> C{Let me think}
    C -->|One| D[Laptop]
    C -->|Two| E[iPhone]
    C -->|Three| F[fa:fa-car Car]

✨Feature Request - support for generating coverage reports

It would be really cool if we cool if we could generate coverage reports with monkeyble so we could see coverage in our CI/CD

CLI: add monkeyble list command

monkeyble list

Playbook   | Scenario        |
-----------+-----------------|
 play1.yml | validate_test_1 |
 play1.yml | validate_test_2 | 
 play2.yml | validate_this   | 
 play2.yml | validate_that   | 

CLI: add test name to be executed

We should be able to execute a single test from its name

monkeyble test validate_test_1

Or multiple selected test

monkeyble test validate_test_1 validate_test_2

The should_failed check should preferably read should_fail instead

As far as I understand the documentation, the intent of Monkeyble is to provide a declarative way of writing unit tests for Ansible tasks. To that end, the YAML keys used to describe the expected outcome of tasks should have a meaning when read out loud.

While it is the case for should_be_changed and should_be skipped checks, reading should_failed seems a little odd. I would suggest to change it to should_fail so that the reader can understand this task should fail when reading the scenario.

When task is skipped: test if it should have failed

Here the method should test that the task was not expected to fail.

If a task is "ok" or "fail" we check if the task should have failed. But this test is not performed in case of skipped task.

'Task' object has no attribute 'play'

Hi,
I have been trying to install Monkeyble on a fresh install of Ansible today, but I keep getting the same error : Failure using method (v2_runner_on_start) in callback plugin (<ansible.plugins.callback.monkeyble_callback.CallbackModule object at 0x7f2fedaa1fd0>): 'Task' object has no attribute 'play'.
My ansible.cfg file is exactly the same as the first one provided in the Hello World doc, with just the path to monkeyble changed.

Is this a known issue?

Thank you