In producer mode, it sends the messages but throws errors while starting up

[edenhill]

In producer mode, it sends the messages but throws errors while starting up

[edenhill]

Ah, the brackets in [localhost]:9092 probably means it is an IPv6 address, i.e., that your localhost entry resolves to both an IPv4 and IPv6 address (usually ip6-localhost is used for the IPv6 address).
And if the broker is only listening on the IPv4 port this will fail when rdkafka attempts to connect to the IPv6 address, on the next connection attempt it will use the next address which is IPv4 (roundrobin).

Can you double check if that is the case? E.g., grep localhost /etc/hosts or host localhost.
Mine gives:

$ host localhost
localhost has address 127.0.0.1
localhost has IPv6 address ::1

[edenhill]

I'm guessing your broker only listens to the IPv4 port, not IPv6, but its advertised host entry resolves to an IPv6 address (or both IPv6 and IPv4 in which case the addresses will be roundrobined).
You can verify which address family the broker is listening to with these commands:

Check for IPv4 broker bind: netstat -anp -A inet | grep :9092
Check fopr IPv6 broker bind: netstat -anp -A inet6 | grep :9092
(thats on Linux)

[edenhill]

Is there a broker running on localhost:9092?
What -b .. arguments are you giving to kafkacat?

[the100rabh]

below is exact arguments and output I got.

Kafka broker is indeed running at localhost:9092 because the consumer showed the output properly which was given after the messages below.

$ kafkacat -P -b localhost:9092 -t test -p 0 
%3|1418300191.571|FAIL|rdkafka#producer-0| localhost:9092/bootstrap: Failed to connect to broker at [localhost]:9092: Connection refused
%3|1418300191.571|ERROR|rdkafka#producer-0| localhost:9092/bootstrap: Failed to connect to broker at [localhost]:9092: Connection refused
%3|1418300191.571|ERROR|rdkafka#producer-0| 1/1 brokers are down

[the100rabh]

Thanks @edenhill this was indeed the issue. Great job debugging the issue friend.

Guess you could document this in the wiki somewhere.

[pbertoni89]

Hi there, having the same problem, but running examples of librdkafka. Same output of the mentioned grep localhost /etc/hosts. I didn't understand the solution, pardon.
My command has switched from
./rdkafka_example -P -t test -b localhost:9092
to
./rdkafka_example -P -t test -b ::1:9092
just obtaining the change from
Failed to connect to broker at localhost:9092: Connection refused
to
Failed to connect to broker at [::0.1.144.146]:9092: Network is unreachable
What am I doing wrong ? Thanks.

[edenhill]

If this turns out to be the problem you can limit what address families librdkafka will try to connect to with the broker.address.family configuration property (set to v4 or v6, defaults to any)

[pbertoni89]

Well, this is embarassing. Kafka just wasn't installed neither running. I apologize for this waste of time 😆

[edenhill]

He, no worries :)

[akuzayah]

Hi, i new in kafka, after I installed librdkafka, then run an example file, i have the same issue. When using rdkafka_example_cpp to set up producer:

rdkafka_example_cpp -P -t my_topic -p 1 -b 127.0.0.1:9092

i got these outputs:

% Created producer rdkafka#producer-0
LOG-3-FAIL: 127.0.0.1:9091/bootstrap: Failed to connect to broker at new-host:9091: Connection refused

and when using rdkafka_example_cpp to set up consumer:

rdkafka_example_cpp -C -t my_topic -p 3 -b 127.0.0.1:9093

a got this these outputs :

% Created consumer rdkafka#consumer-0
LOG-3-FAIL: 127.0.0.1:9093/bootstrap: Failed to connect to broker at new-host:9093: Connection refused
ERROR (Local: Broker transport failure): 127.0.0.1:9093/bootstrap: Failed to connect to broker at new-host:9093: Connection refused
ERROR (Local: All broker connections are down): 1/1 brokers are down

I've checked the IP version used, and I think the problem is not there.is there any suggestion, what's wrong with above command?

[edenhill]

Connection refused means librdkafka can't TCP connect to a given broker.
One of your brokers advertises itself as new-host:9093, but there is no broker running on that host+port, or it is firewalled.

Check your connectivity (you can use telnet new-host 9093 from the host kafkacat runs on to see if you have connectivity.

[scamosa]

Hi! I am having the same issue when trying to run the solution given in https://stackoverflow.com/questions/35619980/how-to-accept-twitter-stream-using-tweepy-in-streamparse-spout-and-pass-the-twee. An error is cropping up stating:

Failed to connect newly created broker for 127.0.0.1:9092
Socket disconnected during metadata request for broker 127.0.0.1:9092. Continuing.

When I ran "grep localhost /etc/hosts" on terminal as per above suggestion, the following response was received:
127.0.0.1 localhost
::1 ip6-localhost ip6-loopback

Hence localhost is configured correctly. Am I correct? If so, what would think would be the issue please? Sorry I am a little new to Storm and Kafka (and I am still getting more familiar with Python and Ubuntu).

I thank you in advance for your help and support.

[edenhill]

@scamosa Not really sure what that has to do with kafkacat, there is no mention of it.

[scamosa]

I have a main Python method with the following code:
from listener import MyStreamListener MyStreamListener()

Inside listener.py you will find:

import tweepy
from streamparse.spout import Spout
from pykafka import KafkaClient

class TweetSpout(Spout):
words = []

def initialize(self, stormconf, context):
    client = KafkaClient(hosts='127.0.0.1:9092')

    self.topic = client.topics[str('tweets')]

def next_tuple(self):
    consumer = self.topic.get_simple_consumer()
    for message in consumer:
        if message is not None:
            self.emit([message.value])
    else:
        self.emit()

class MyStreamListener(tweepy.StreamListener):

def on_status(self, status):
# print(status.text)
client = KafkaClient(hosts='127.0.0.1:9092')

topic = client.topics[str('tweets')]
with topic.get_producer(delivery_reports=False) as producer:

    # print status.text
    sentence = status.text
    for word in sentence.split(" "):
        if word is None:
            continue
        try:
            word = str(word)
            producer.produce(word)
        except:
            continue

def on_error(self, status_code):
if status_code == 420: # exceed rate limit
return False
else:
print("Failing with status code " + str(status_code))
return False

auth = tweepy.OAuthHandler(API_KEY, API_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)
api = tweepy.API(auth)
myStreamListener = MyStreamListener()
myStream = tweepy.Stream(auth=api.auth, listener=myStreamListener)

myStream.filter(track=['is'])

---

I am getting the following error:

Failed to connect newly created broker for 127.0.0.1:9092 Socket disconnected during metadata request for broker 127.0.0.1:9092. Continuing. Failed to connect newly created broker for 127.0.0.1:9092 Socket disconnected during metadata request for broker 127.0.0.1:9092. Continuing. Failed to connect newly created broker for 127.0.0.1:9092 Socket disconnected during metadata request for broker 127.0.0.1:9092. Continuing.

[edenhill]

I'm sorry but what does this have to do with kafkacat?

[scamosa]

I googled for a solution and this site cropped up. Given that the same output was given, I thought it might be related. In that case, I am sorry for wasting your time and I thank you for looking into my problem. Hope will find a solution.

[Beksultan-Aksakalov]

Hi everyone!

1. Docker Desktop on Windows

2. There are commands for up kafka , zookeeper, kafka-manager:

• docker run -d --name kafkadocker_zookeeper_1 dockerkafka/zookeeper
• docker run -d --name kafkadocker_kafka_1 --link kafkadocker_zookeeper_1:zookeeper dockerkafka/kafka
• docker run -d -it --rm --link kafkadocker_zookeeper_1:zookeeper --link kafkadocker_kafka_1:kafka -p 9000:9000 -e ZK_HOSTS=zookeeper:2181 dockerkafka/kafka-manager

3. Containers

4. Kafka - manager: cluster name is test, the topic name is test-topic

5. Confluent's .NET Client for Apache Kafka. Link: https://github.com/confluentinc/confluent-kafka-dotnet

5.1 Producer

5.2 Visual Studio Console

Can you help me solve this problem?

If there are any answers write to me in gmail: [email protected]

Thank you all !!! 💯