Min wall clock time per single kafkacat invokation

[edenhill]

Min wall clock time per single kafkacat invokation

[edenhill]

You are absolutely right, there are a couple of internal timeouts in librdkafka that are 1s.
To find the best solution I'd like to ask you what would an acceptable exit-delay be for you?

[ibalashov]

Thanks for confirming this!
Hm.. Anything less than 100ms would serve my needs, but can't this delay be dropped altogether?

[edenhill]

This is now fixed in librdkafka commit confluentinc/librdkafka@8d9fdd2

$ time (echo "foo" | ./kafkacat -b 0 -P -t test -p 0)

real    0m0.052s
user    0m0.001s
sys 0m0.005s

Can you verify this on your end as well?

[ibalashov]

For some reason I still see the delay.
I tried rebuilding librdkafka and also getting one from https://github.com/edenhill/librdkafka/archive/master.tar.gz

Could it be something related to OS X?

[edenhill]

Huhm, are you sure kafkacat is using the updated librdkafka?
Is it linked dynamically or statically?
Try checking out a fresh kafkacat and run ./bootstrap.sh

[ibalashov]

Just did a fresh checkout and ./boostrap.sh on a linux installation elsewhere and now it works correctly.

Strangely, doing the same on OS X has no effect. Could you possibly spot any issues in the build log? https://gist.github.com/ibalashov/c74f8cb9217db3099362#file-kafkacat_buildlog_osx-txt

[edenhill]

Ah, SIGIO is ignored by default on OSX, but not Linux.
That's easily fixable.

But I'm not sure this is the right solution, a library typically shouldn't trigger signals and install signal handlers without the application's knowledge. Will have to think about it a bit more.

Is this stopping your development/deployment?

[ibalashov]

Glad to hear at least the cause is clear.
No worries, I needed this mainly for tests, and now I can run kafkacat from inside Linux VM if/when needed.
Thanks!

[edenhill]

I've made a better fix for this now (opt-in on the application side, kafkacat opts in) which should also work better on OSX.