Paving the way for greater and more powerful automation

[chaostoolkit]

Paving the way for greater and more powerful automation

[0rc]

one more point:

8. Support tolerance for probe inside method. Perhaps also support tolerance for action.

[Devatoria]

Hello there. I'm actually giving a shot to the chaos toolkit to test some use cases we have at Datadog and it's sure that some of the listed features in this issue are things we definitely need. What's the actual state on your side? Do you think you want to keep it simple or add those features but with the complexity it'll generate? I'd really like to have a quick overview of where the project should go in your mind.

Thank you for the answer and for the work you provided here :)

[Lawouach]

Hey @Devatoria,

Thanks for kind words on the project. :)

We are currently having lots of discussions out of band around the toolkit and lots of interests around its direction so you will have to assume that some data points may be missing in what I am saying next :p

I think in general, users who have reached out to us because they want to invest on the toolkit have expressed the need for features that fall into three baskets IMO:

1. functionalities that are part of the scope and remit of the toolkit itself (basically, anything that is about the experiment itself)
2. functionalities that are more around the visibility, organization and scheduling of chaos experiments running (think organizations and collaboration)
3. functionalities that are not tied to the toolkit but likely more an expression of a specific need

With this in mind, how the project evolves depends on which category they fall into.

At this stage, the toolkit should (finally) see its v1 being released to make a stand that effort people have put into using the toolkit (and contributing to it) until now will remain. Stability is essential to us.

Future minor versions will see some of points, shown in my initial comment, likely be included in some fashion. Not all of them may make it however.

I don't have a v2 definition in my mind yet. I think, v2 is likely one or two years away and will come after the community has grown more with a clearer idea of where the project could go to.

Some functionalities that we see asked more and more are around collaboration and visibility. Those concerns will likely be under the Chaos Platform umbrella as it is its scope. That project is under this org and you can see its code already but it's a month away from being fully documented to appreciate its scope.

Finally, where we would like to also take the project is towards becoming a respected citizen of the Open Chaos Initiative. From a functionality front, it means we plan on finally making chaos discover fully featured in respect of analysing your system to suggest recommendations of experiments you would want to run. But also allow you, if you wanted to, to share aspects of your experiments and learnings, for everyone to benefit from (with the idea that it's all knowledge we may benefit from at an industry level).

I know this is high level, so please let me know if you have more specific questions and I'll do my best to reply :)

[0rc]

@Lawouach thanks for great summary
I would also add a point of conditional logic to your list (which somewhat correlates with point 5 in your list):

• Be able to run or not run certain steps based on the outputs of previous steps

To me and the use-cases I see the most important would be points 1, 3, 4

[0rc]

another idea I have for improvement:
9. Multi-stage experiments. In other words, possibility to have multiple methods and steady state hypothesis in a specific order within one file, or perhaps referencing files from the master file

[MrBW]

As we discussed:

Is there a possibility to execute the so-called probe more than once or to execute it several times for a certain time? So far I have used the HTTP provider, which only sent 1 request at a time. Is there a possibility to execute this multiple times as described and to aggregate the results?
So far, for example, I have used Apache Benchmarking for HTTP requests.

During my last workshop, more people than just me came up with this idea and asked me about it. In my opinion it would make sense to use more than 1 request for verification.

[Lawouach]

Hey @0rc, sorry for the delay.

Thank you so much for you input. This is what this issue is all about! :)

It feels that 8. and 9. could be towed to 6. (adding a test mode).

It is an interesting point, the toolkit was meant to be used as a discovery tool (tell me something about my system that I don't know yet). However, as users move on with their experiments and want to run them continuously, having more control over their execution is expected.

While I feel comfortable with adding a certain level of control, I certainly wonder how far we can go with the declarative approach.

In other words, there is a huge discussion to have here and that's exciting :)

[Lawouach]

Just a note to myself this might be of interest for this discussion too chaostoolkit/chaostoolkit-lib#32

[Lawouach]

One for the road chaostoolkit/chaostoolkit-lib#106

[gtfisher]

One of the things I looked for was the ability to run multiple experiments for one call to chaos run, by passing a list of files or possible wildcards. I realise this leads to complexity such as sequential or parallel execution

[Lawouach]

Might be of interest too chaostoolkit/chaostoolkit-lib#137

[Lawouach]

See #148 and #149 for work undetaken.

[Lawouach]

Another useful feature to consider. #162

[github-actions]

This Issue has not been active in 365 days. To re-activate this Issue, remove the Stale label or comment on it. If not re-activated, this Issue will be closed in 7 days.

[github-actions]

This Issue was closed because it was not reactivated after 7 days of being marked Stale.