Why open a new connection for every message?

[cosmo0920]

Why open a new connection for every message?

[cosmo0920]

Because fruently does not send message with chunk option.
Fluentd forward protocol v1 (and v0) requests that server SHOULD close the connection silently with no response when the chunk option is not sent.

ref: https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1#response

[jgrimes]

Thanks for the response.

I was prompted to look into this due to very high CPU usage by fluentd workers. I made a fork and changed fruently's code to just maintain a single connection and saw CPU usage drop to .2x what it was previously on an 8 core machine with 8 workers. From about 400% to 75-90%. Nothing changed in fluentd's behavior other than that.

The part you mentioned is for the server side of the protocol, right? It seems odd to me to close the client connection after every message given the costs associated with it. There is also mention of keep-alive in the protocol docs. Note: not arguing with the protocol, just trying to figure out if it is indeed their intention to have the client close the connection after every message.

[cosmo0920]

Yep, this is server side.
But your implementation is dangerous.
If you want to use single connection on Forward protocol, you must implement require ack response mechanisim.
A patch which implements this mechanisim should be acceptable.