when using probe in `method` tolerance is not validated

[chaostoolkit]

when using probe in `method` tolerance is not validated

[SaydChada]

This should be available in some way... What i saw is that most of the chaostoolkit libs (as for exemple k8s) uses probs func as a way to raise chaoslib.exceptions.ActivityFailed when it fails.
For those who are not familar with python, having a way to validate tolerance activity probs in method can be very helpful.

[0rc]

here
I'd prefer to have something that will fail the experiment when activity is probe and it is not in defined tolerance

Also (just thinking) it can be useful to specify tolerance for action too to have more control over the experiment

[russmiles]

It's an excellent point @0rc . At the moment this is working as intended as probes in an experiment's method are intended to collect additional information during an experiment's execution, and not to provide any early-experiment-aborting logic.

That's not to say that such control is a bad thing, but it's not currently the intention of these probes.

[russmiles]

This might be a further refinement/requirement on this issue: chaostoolkit/chaostoolkit#74

[0rc]

Yes I will add this point to chaostoolkit/chaostoolkit#74

And yes, I just double-checked the documentation and it doesn't say tolerance is supported when using probe in method. But still it may be better to mention explicitly it's not supported unless we can add it soon

But what are other opinions, do you guys think it's useful to have tolerance supported inside method?

[0rc]

Another example:
When running experiment, I want to make sure there is no active scaling (up or down) activity because of other reasons, not related to experiment (scheduled, traffic spike, whatever) and if there is any I find it's unsafe to run an experiment.
At the same time, I don't want this to be steady state hypothesis as after experiment actions block it is quite normal to have scaling activities which will bring things back to normal.

[maximmold]

My use case is that I want to kill a node and ensure the node is actually killed before proceeding with the steady state hypothesis which I would initially expect to fail on my one node cluster. Currently, it just drops through to the steady state hypothesis and passes before azure has actually stopped the node which is a false positive.

[Lawouach]

I think we are getting more and more ready for this to be an implemented feature considering how many folks need it :)

[github-actions]

This Issue has not been active in 365 days. To re-activate this Issue, remove the Stale label or comment on it. If not re-activated, this Issue will be closed in 7 days.

[github-actions]

This Issue was closed because it was not reactivated after 7 days of being marked Stale.