Zookeeper quorum and Kafka broker host configs both required

[pinterest]

Zookeeper quorum and Kafka broker host configs both required

[HenryCaiHaiying]

Secor is written before kafka 0.8.2.

The kafka hosts are specified in seed.broker.host, secor was also using
Kafka's low-level SimpleConsumer, you need to specify at least one seed
broker for SimpleConsumer.

The ZK path are need because secor is mostly using Kafka's high level
consumer, and secor needs to look at the offset stored by Kafka's high
level consumer in those ZK path.

I am not aware you can get the broker list from ZK through kafka's
documented API.

On Mon, Sep 21, 2015 at 2:07 PM, fabeschan [email protected] wrote:

Mainly just wondering if there's a reason why Secor needs to know both the
Zookeeper hosts and a Kafka broker in order to function. As far as I know,
for older versions of kafka (pre 0.82), talking to zookeeper will be able
to provide info of kafka brokers as needed, while for newer versions (0.82
and beyond) clients can manage zookeeper offsets through kafka itself.

But looking at Secor's config files, it seems like both are required. Is
there a particular reason?

—
Reply to this email directly or view it on GitHub
#134.

[keithdotpower]

This would be useful to help recover from errors too. Currently, if the bootstrap broker is unavailable when secor is brought up it can't do anything, even if more brokers are available.

I'm not sure about the official status of this but the list of active broker info is maintained by kafka in zookeeper under /broker/ids and this hasn't changed in the past couple of versions. There's an example of using this at: http://stackoverflow.com/questions/29490113/kafka-get-broker-host-from-zookeeper
We're using it in our system so I can confirm it works.

[HenryCaiHaiying]

I think retrieve broker info from ZK would work for now, but kafka is
moving away from storing info on ZK. E.g. in kafka 0.9, all the consumer
offsets are stored in broker rather than ZK (I haven't checked the broker
info in ZK itself).

On Tue, May 10, 2016 at 5:38 AM, keithdotpower [email protected]
wrote:

This would be useful to help recover from errors too. Currently, if the
bootstrap broker is unavailable when secor is brought up it can't do
anything, even if more brokers are available.

I'm not sure about the official status of this but the list of active
broker info is maintained by kafka in zookeeper under /broker/ids and this
hasn't changed in the past couple of versions. There's an example of using
this at:
http://stackoverflow.com/questions/29490113/kafka-get-broker-host-from-zookeeper
We're using it in our system so I can confirm it works.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#134 (comment)

[HenryCaiHaiying]

Also you can specify multiple brokers in the bootstrap server to minimize
the dependency on one broker, e.g.

broker1:9092,broker2:9092,broker3:9092

On Tue, May 10, 2016 at 8:59 AM, Henry Cai [email protected] wrote:

I think retrieve broker info from ZK would work for now, but kafka is
moving away from storing info on ZK. E.g. in kafka 0.9, all the consumer
offsets are stored in broker rather than ZK (I haven't checked the broker
info in ZK itself).

On Tue, May 10, 2016 at 5:38 AM, keithdotpower [email protected]
wrote:

This would be useful to help recover from errors too. Currently, if the
bootstrap broker is unavailable when secor is brought up it can't do
anything, even if more brokers are available.

I'm not sure about the official status of this but the list of active
broker info is maintained by kafka in zookeeper under /broker/ids and this
hasn't changed in the past couple of versions. There's an example of using
this at:
http://stackoverflow.com/questions/29490113/kafka-get-broker-host-from-zookeeper
We're using it in our system so I can confirm it works.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#134 (comment)

[keithdotpower]

I had looked around the code and got the impression from this that only a single host was used:

secor/src/main/java/com/pinterest/secor/common/KafkaClient.java

Lines 63 to 66 in 52a2a8b

	consumer = createConsumer(
	mConfig.getKafkaSeedBrokerHost(),
	mConfig.getKafkaSeedBrokerPort(),
	"leaderLookup");

I'll look into it more, if it works that will definitely help. Thanks for that.

[HenryCaiHaiying]

Looks like you are right, secor does put the restriction on one bootstrap
servers. It's probably not that hard to extend that to cover a list of
bootstrap brokers.

On Fri, May 13, 2016 at 1:39 AM, keithdotpower [email protected]
wrote:

I had looked around the code and got the impression from this that only a
single host was used:

secor/src/main/java/com/pinterest/secor/common/KafkaClient.java

Lines 63 to 66 in 52a2a8b

	consumer = createConsumer(
	mConfig.getKafkaSeedBrokerHost(),
	mConfig.getKafkaSeedBrokerPort(),
	"leaderLookup");

I'll look into it more, if it works that will definitely help. Thanks for
that.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#134 (comment)