Trying to consume from one topic and produce to another hangs

[jroland]

Trying to consume from one topic and produce to another hangs

[warrenfalk]

Nevermind. It appears that it is actually Kafka that is failing to send the response to the metadata request after the fetch request.

[warrenfalk]

OK, after more investigation, there is actually a problem with kafka-net. The problem is that a kafka broker (in 0.8.2.1 at least) will not process a produce request while a fetch request is waiting for data. This is a problem with kafka-net because as soon as it yields the results of one fetch request, it sends a new fetch request immediately in another thread. This ultimately makes it impossible to use the BrokerRouter to do any publishes that end up being sent to the same broker.

Note: I think this is really poor behavior by Kafka. Its network protocol is described as though it could work this way (with correlation ids and all) but it does not seem to work that way.

[warrenfalk]

In fact, they document that it does not work this way because all requests on a particular socket are ordered:

"The server guarantees that on a single TCP connection, requests will be processed in the order they are sent and responses will return in that order as well. The broker's request processing allows only a single in-flight request per connection in order to guarantee this ordering. Note that clients can (and ideally should) use non-blocking IO to implement request pipelining and achieve higher throughput. i.e., clients can send requests even while awaiting responses for preceding requests since the outstanding requests will be buffered in the underlying OS socket buffer. All requests are initiated by the client, and result in a corresponding response message from the server except where noted."

[warrenfalk]

See discussion here: http://www.mail-archive.com/[email protected]/msg27430.html

This reply suggests that Consumers and Producers should actually not reuse the connection used by the BrokerRouter at all, and each instance should create its own connection for fetch/produce requests. The claim by the protocol wiki page that you should only ever need one connection between each broker and a client is no longer true and is going to be removed. The new Java consumers will all maintain their own connections.

[Jroland]

Thanks warren, ill have a close read of that. Got pretty sick the last several weeks. So I am just getting back into things now.