Peer closes connection before other peer has read all the data

[ShahakShama]

Peer closes connection before other peer has read all the data

[thomaseizinger]

Which transport are you using? Does this issue occur with QUIC? On TCP, we are layering many technologies together, like yamux on top of noise etc. It may be that "flushing" isn't correctly forwarded for some of them.

Note that we cannot directly achieve what you are suggesting because it is a two-generals problem: The message that we have read all the data may get lost, or more generally: The last ACK can always get lost.

The only thing we can in theory ensure is that all data has been flushed to the network and it may be that this isn't true for a composed TCP stack.

This might be related in case you are using yamux: rust-lang/futures-rs#2746.

Your best chance is to use QUIC as there are less layers at work there.

[ShahakShama]

I indeed used TCP. I swapped to QUIC and now it can process about 10 times more messages before this issue returns.
So raising the timeout and using QUIC are the only possible solutions for this issue? If yes, I think it should be documented somewhere that the connection may be closed before all the messages were flushed. Maybe here

[thomaseizinger]

I indeed used TCP. I swapped to QUIC and now it can process about 10 times more messages before this issue returns. So raising the timeout and using QUIC are the only possible solutions for this issue? If yes, I think it should be documented somewhere that the connection may be closed before all the messages were flushed. Maybe here

I don't think it is right to describe bugs in the documentation :)

The issue is somewhat a combination of many (complicated aspects):

• As I documented above, it is a fundamental distributed systems problem that you can never be fully sure that the other party has received your message. See https://en.wikipedia.org/wiki/Two_Generals%27_Problem.
• Because of the above, the request-response abstraction cannot know when the response was received.
• There may actually be bugs in our forwarding of flushing calls.

I'd suggest one or more of the following:

• A non-zero default for connection timeout can generally be useful (depends on your application). Also see #4912.
• Re-design your application protocols such that the essential information is in the request, then receipt of the response acts as an ACK that the request was received.
• If receipt of the response needs to be acknowledged, don't use request-response but design a three or more message protocol. #5027 may be useful for that.

[0x7CFE]

I also constantly experiencing this issue with streams (especially on slower connections) and it's really annoying that you may never know if data were delivered.

I hacked it to work by manually sending ACK byte from the other side. But it feels weird given that underlying TCP guarantees the delivery.

[0x7CFE]

Why not just keep the connection open if at least one stream is active?

[thomaseizinger]

Why not just keep the connection open if at least one stream is active?

That is the (intended) behaviour today.