Failure detector closes connection if linkerd receives data faster than it can write

[hmhagberg]

Failure detector closes connection if linkerd receives data faster than it can write

[adleong]

The failure detector is designed to help detect network issues, especially in cases where requests are infrequent. In your situation it sounds like your network is fairly reliable. I think turning it off should be fairly safe for you, especially if your connections don't have long periods of idleness.

[hmhagberg]

I created PR #2399 with the fix/workaround I mentioned in the description. I haven't been able to reproduce the issue with that patch.

However, I'm not sure if this is the best way to fix the problem. As said, it's still possible that flushing 128 messages delays ping for too long, especially if RTT is already high. However, to me it sounds anyway reasonable to limit the time one task can be running in event loop but I'm not too familiar with netty/linkerd internals.

[adleong]

Thanks for this fantastic and detailed report, @hmhagberg! I will look into your PR but we'll have to be cautious with this kind of change because the details of buffering and flushing are subtle. As a workaround, you can disable H2 failure detection by starting Linkerd with this flag -com.twitter.finagle.liveness.sessionFailureDetector=none.

[hmhagberg]

I will look into your PR but we'll have to be cautious with this kind of change because the details of buffering and flushing are subtle.

Yes, absolutely. I noticed that when debugging this :)

Thanks for the workaround. That was unknown to us but it seems to avoid the issue.

[cpretzer]

@hmhagberg glad to hear that the workaround avoids the issue. With regard to the PR, that code is pretty sensitive, so we prefer to leave it as is.

If you find that the workaround is not sufficient, please add a comment to this ticket and we'll revisit.

[hmhagberg]

Understandable.

I'd like a clarification on the downsides of the workaround though. If I've understood correctly, the ping failure detector is only used for load balancing. In cases where there is only one endpoint available, the failure detector does not bring much benefit since requests would be failing anyway if that endpoint dies?

[hmhagberg]

Ping @cpretzer, any comments on the question above? Atm we are planning on rolling out the workaround but it would be good to know if it potentially has any other effects.