Consumer stops to read messages after the node is recovered

[jroland]

Consumer stops to read messages after the node is recovered

[jmucza]

Hi Ivan,

The current version of kafka-net does not implement consumer rebalancing. When a consumer dies, consumers in the same group won't rebalance over topic partitions. Similarly when the leader broker of topic partition(s) a consumer consumes goes down, the consumer won't rebalance over remaining brokers.

I would suggest taking a look at how Zookeeper is used by native scala client to implement fault tolerance and load balancing on the consumer side. To detect a broker failure a consumer would need to monitor changes to children of /brokers/ids zookeeper path, refresh topic metadata when children have changed, close existing connections and open new ones to those brokers who host in-sync replicas of consumed topic partition(s).

KafkaNet producer is fault tolerant, i.e. when a message can't be sent to a given broker, the producer will try the next broker in the cluster.

Best regards,
Janos

[IvanDrian]

Hi Janos,
Thank you very much for the explanation. I would like to add that consumer works well if node which is not leader for given topic is failed and recovered i.e. no re-balancing happened.