Losing messages on failover test

[edenhill]

Losing messages on failover test

[edenhill]

Hi Thomas,

can you describe your test case in a little more detail, specifically how you are running kafkacat in this test case? Is it a while loop with a small number of files, or one kafkacat invocation with a large number of files?

I'm guessing you are using the latest kafkacat master which provides proper exit codes when messages fails delivery.

Thanks

[thomas-manage]

Hi Magnus,

Thanks for taking the time to look at this. Yes, I am using the latest with your fix for the exit code issues.

I started my consumer with the following parameters:

[[email protected] /tmp/kafkacat-f7209ae187345995a3d82732a4bb45a6a9a9c74c]# ./kafkacat -C -b hadoop-dev01,hadoop-dev02 -t tkwan-test3 -o end -u | tee /tmp/consumed

For my producer, I started it using script

cat /tmp/test1 | ./kafkacat -P -b hadoop-dev01,hadoop-dev02 -t tkwan-test3 -p 1 || exit 10
echo "1"
cat /tmp/test2 | ./kafkacat -P -b hadoop-dev01,hadoop-dev02 -t tkwan-test3 -p 1 || exit 10
echo "2"
cat /tmp/test3 | ./kafkacat -P -b hadoop-dev01,hadoop-dev02 -t tkwan-test3 -p 1 || exit 10
echo "3"
cat /tmp/test4 | ./kafkacat -P -b hadoop-dev01,hadoop-dev02 -t tkwan-test3 -p 1 || exit 10
echo "4"
cat /tmp/test5 | ./kafkacat -P -b hadoop-dev01,hadoop-dev02 -t tkwan-test3 -p 1 || exit 10
echo "5"
cat /tmp/test6 | ./kafkacat -P -b hadoop-dev01,hadoop-dev02 -t tkwan-test3 -p 1 || exit 10
echo "6"
cat /tmp/test7 | ./kafkacat -P -b hadoop-dev01,hadoop-dev02 -t tkwan-test3 -p 1 || exit 10
echo "7"
cat /tmp/test8 | ./kafkacat -P -b hadoop-dev01,hadoop-dev02 -t tkwan-test3 -p 1 || exit 10
echo "8"
echo "Exit Code:" $?

Each /tmp/test* file has 30,000 lines of json.

I used 2 brokers for my testing. When I killed one brokers, I saw a couple errors but no non-zero exits.

[[email protected] /tmp/kafkacat-f7209ae187345995a3d82732a4bb45a6a9a9c74c]# sh /tmp/a.sh
1
2
%3|1430531634.699|FAIL|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Receive failed: Disconnected
%3|1430531634.699|FAIL|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Receive failed: Disconnected
%3|1430531634.699|ERROR|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Receive failed: Disconnected
%3|1430531634.699|ERROR|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Receive failed: Disconnected
%3|1430531634.699|FAIL|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531634.699|ERROR|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531634.699|FAIL|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531634.699|ERROR|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
3
%3|1430531635.245|FAIL|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531635.245|ERROR|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531635.249|FAIL|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531635.249|ERROR|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
4
%3|1430531636.774|FAIL|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531636.774|ERROR|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531636.788|FAIL|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531636.788|ERROR|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
5
%3|1430531637.240|FAIL|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531637.240|ERROR|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531637.269|FAIL|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531637.269|ERROR|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
6
%3|1430531638.717|FAIL|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531638.717|ERROR|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531638.754|FAIL|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531638.754|ERROR|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
7
%3|1430531639.316|FAIL|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531639.316|ERROR|rdkafka#producer-0| hadoop-dev01:9092/bootstrap: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531639.335|FAIL|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
%3|1430531639.335|ERROR|rdkafka#producer-0| hadoop-dev01.manage.com:9092/1: Failed to connect to broker at hadoop-dev01.manage.com:9092: Connection refused
8
Exit Code: 0

Then, I would do a "wc -l /tmp/consumed" on the consumer. The total should be 240,000 (8 * 30,000). Sometimes, i have less than 240,000.

thanks again

[thomas-manage]

Seems like adding "-X topic.request.required.acks=2" to the producer helps. In my test environment, I have 2 brokers. Will test more.

[edenhill]

Ah, yes, what broker version are you on?
request.required.acks>1 is a legacy that is now (since 0.8.2) replaced with request.required.acks=-1 and the broker property min.isr property that specifies the number of In-Sync-Replicas needed to fullfil a request.required.acks=-1 produce request.

Search this blog post for "min.isr":
http://blog.confluent.io/2014/12/02/whats-coming-in-apache-kafka-0-8-2/

[thomas-manage]

ah, ok, thanks for checking. We are on 0.8.1.1

[thomas-manage]

will close this as a user-error, thanks

[thomas-manage]

actually, magnus, have you considered to use -1 as the default for request.required.acks instead of 1 in kafkacat?

[edenhill]

Considered, yes, but I'm very hesitant to changing defaults since existing applications may depend on the current behaviour.
I'd be very happy to link a blog post, if you wrote one, describing this usage though :)

[thomas-manage]

Fair concern. I will close this issue. Thanks again for helping!