kafkacat topic info doesn't match that of kafka-topics

[edenhill]

kafkacat topic info doesn't match that of kafka-topics

[edenhill]

kafkacat uses the Kafka broker protocol's MetadataRequest to directly query the broker for cluster and topic metadata information, while kafka-topics.sh queries zookeeper for the same information.

Clearly one of those are out of sync in your printouts, and the information retrieved directly from Kafka brokers is usually authoritative (e.g., kafkacat > kafka-topics.sh), unless some of the brokers are acting up.

Could you log in to one of your broker VMs and run kafkacat once for each broker, explicitly providing -b <broker> for each call (where broker is the hostname reported in the metadata printout above, e.g.: ip-10-0-246-113.ec2.internal:9092)?
While this is probably exactly what dsh does, I'd like it to be a bit more explicit to rule out the case where just one of the brokers were queried.

[fsaintjacques]

Are you running with unsafe leader election?

[charity]

Sorry for the delay. Found the problem -- kafka treats the "." character as a wildcard in topic names. And kafkacat will silently create a new topic if you describe it (unlike kafka-topic --describe). You can see in the first example, the topic is "hound-prod.retriever.mutation", and the second it is hound-prod.retriever-mutation".

Suggest making "kafkacat -D -t $topic" not create the topic if it doesn't exist, and instead behave the way "kafka-topics --topic --describe" does :)

[edenhill]

The automatic topic creation is actually a broker side feature and config (auto.create.topics.enable=true), to avoid it you can list all topics (leave out the -t ..topic.. switch) and then do output filtering (preferably with JSON / jq)