Consumer not able to read kafka topic and write to HDFS

[dibbhatt]

Consumer not able to read kafka topic and write to HDFS

[dibbhatt]

Can you please share Logs of the consumer .

What happened when you restart the job ? It fetching Zero messages ?

Is there any difference between Kafka or Spark version between CDH upgrade ?

If I look at the logs, I will be able to tell something about the issue.

[sbellary300s]

Hello, thanks for your prompt response, i have just shared the logs via email to [email protected]

Kafka and zookeeper version did not change after the upgrade. Spark version changed from 1.5.0 to 1.6.0

additionally please check our spark-submit if it helps you address this issue:
spark-submit
--class com.comp.consumer.kafka.client.Consumer
--conf spark.driver.maxResultSize=4G
--conf spark.driver.extraJavaOptions=-Denv=${ENV}
--conf spark.yarn.executor.memoryOverhead=3000
--conf spark.akka.frameSize=1024
--executor-memory 6G
--driver-memory 16G
--master yarn-cluster
--driver-cores 12
--num-executors 27
--executor-cores 5
$JAR $DATE "$CONSUMER" &

[krishna383]

@dibbhatt We are observing that the offset is not getting committed to zookeeper. We are not seeing the consumerid in the zkcli

[dibbhatt]

Did you tried running kafka console consumer from Spark Exeutor and see if it able to consume messages ? You are not seeing offsets in ZK mean consumer is not consuming . Can you share the properties files with all consumer specific properties you specified ?

[dibbhatt]

Are you still seeing the issue ? If not you can close the ticket.

[dibbhatt]

hi @sbellary300s @krishna383 if the issue is fixed, please close this ticket.

[dibbhatt]

closing this as I have not heard anything yet @sbellary300s . if you still having the issue, please let me know more details about the the same.