How is it different from KafkaUtils.createStream of spark-streaming-kafka_2.10?

[bzz]

How is it different from KafkaUtils.createStream of spark-streaming-kafka_2.10?

[dibbhatt]

Hi,
The primary difference is this Consumer is using Kafka Low Level Consumer API whereas Spark-streaming-Kafka package consumer is using Kafka High Level Consumer API.

In Spark 1.1.x the Spark provided High Level Consumer has a data loss issue on Receiver failure, which prompt me to write this consumer.

In Spark 1.2 they have provided a reliable version of High Level Consumer , which solves the data loss issue, but it has other issues related to recovery from failure, it stops intermittently , not available once failed etc. Also using Kafka High Level API itself has issues which you can find here : https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design

Now this Low Level Consumer is till now is most reliable way to consume from Kafka without any data loss and can recover from any underlying failures ( Spark worker failures, Kafka broker failures , Spark internal BlockManager failures etc). We at Pearson has been using this for months without any downtime.

Just to mention , Spark 1.3 has another version of Reliable Consumer API ( This time they used Low Level API ) and promised to solved the exactly-once semantics of message processing. But this is yet to be tested in production scenarios.

[bzz]

Thank you very much for publishing your work and the detailed explanation!

On Wed, Mar 4, 2015 at 3:12 PM, Dibyendu Bhattacharya <
[email protected]> wrote:

Hi,
The primary difference is this Consumer is using Kafka Low Level Consumer
API whereas Spark-streaming-Kafka package consumer is using Kafka High
Level Consumer API.

In Spark 1.1.x the Spark provided High Level Consumer has a data loss
issue on Receiver failure, which prompt me to write this consumer.

In Spark 1.2 they have provided a reliable version of High Level Consumer
, which solves the data loss issue, but it has other issues related to
recovery from failure, it stops intermittently , not available once failed
etc. Also using Kafka High Level API itself has issues which you can find
here :
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design

Now this Low Level Consumer is till now is most reliable way to consume
from Kafka without any data loss and can recover from any underlying
failures ( Spark worker failures, Kafka broker failures , Spark internal
BlockManager failures etc). We at Pearson has been using this for months
without any downtime.

Just to mention , Spark 1.3 has another version of Reliable Consumer API (
This time they used Low Level API ) and promised to solved the exactly-once
semantics of message processing. But this is yet to be tested in production
scenarios.

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

Kind regards,
Alexander.