Order retries / circuit breaker in the order they're configured

[failsafe-lib]

Order retries / circuit breaker in the order they're configured

[whiskeysierra]

Got it.

Composing policies is in theory like composing functions.

That actually pretty much what I'm doing in riptide. We have a Plugin interface that allows to decorate executions (which are basically just callables). I'm using failsafe for retries and circuit breaker, see riptide-failsafe.

Other plugins provide other functionalities that you also described here or elsewhere:

• timeouts (not per execution, but applied to the outer most decorated function covering any kind of retires/circuit breaker/delay/etc.)
• backup requests, can be thought of as early retries
• metrics, effectively measuring duration of executions
• transient fault detection, basically error handling that can then be used to control retries (using retryOn)

[whiskeysierra]

See also #96

[jhalterman]

Thinking about this issue some more, I'm not sure that having a circuit breaker before a retry makes sense. Anyone have thoughts on this?

[whiskeysierra]

I'm not sure that having a circuit breaker before a retry makes sense.

Isn't that the current behavior?

[jhalterman]

The current behavior is a blend of retry and circuit breaker handling, which I don't think is ideal. I'd like to separate them more clearly so that only one occurs at a time, and so they can be composed in some order.

[jhalterman]

Trying out an implementation... will post a PR for review if it looks good.

[jhalterman]

See #153 for a discussion topic related to this issue.

[jhalterman]

I'm back to this PR and hope to push something to look at soon.

The current Failsafe implementation assumes a specific ordering of policies. Arbitrarily composing synchronous policy executions is straightforward, but composing asynchronous executions is a bit of a challenge. Since async callbacks have no context, we need to re-create the context in which the async execution began, by moving through the composed policies back to the location where the async execution came from before we can continue.

Overall this will be a nice improvement as it unifies a few things internally and will open the door to let policies be more standalone things.

[whiskeysierra]

Since async callbacks have no context, we need to re-create the context in which the async execution began, by moving through the composed policies back to the location where the async execution came from before we can continue.

What kind of context are we talking about exactly?

[jhalterman]

What kind of context are we talking about exactly?

The thread/execution context. Composing policies is in theory like composing functions. Given some callable, we can think of the policies composing around it like:

fallback(retryPolicy(circuitBreaker(callable)))

If the callable is synchronous, this composition is simple and works as expected. If it's asynchronous, the callable is executed on a separate thread, and when it's done we're no longer nested within the other policies. So we have to re-create that nesting before we can handle the callable's result.

The way we do this is treating the composition like a chain. To first execute the callable we work our way along the chain and call the callable on a separate thread:

fallback -> retryPolicy -> circuitBreaker -> callable.callAsync

Since the result of the callable comes to us on a separate thread without any context, we have to re-create the context by working our way back down the chain to where we triggered the async call, then we can handle the result synchronously as we normally would:

fallback -> retryPolicy -> circuitBreaker.handleResult(result)
fallback -> retryPolicy.handleResult(result)
fallback.handleResult(result)

This isn't the best explanation, but hope it conveys the idea.

[jhalterman]

Fixed by 5d919d5. The sync and async implementations of the problem described above are here.