no messages are sent to kafka

[sngular]

no messages are sent to kafka

[github-actions]

Thank you for collaborate with the project giving us feedback!!' You will help us a lot if you can attach and example AVRO and an example Test Plan in order to be able to reproduce the error. If that is the case. Cheers

[kszosze]

Hi Vitaliee,
I´ll check it later today and come you back.

Cheers

[kszosze]

Hi @vitalee78 ,

I took a look into your problem I saw couple of thinks.
a) an issue in the our generic serialiser which I fixed in the branch linked to this issue
b) a miss configuration in your JMX file. In order to add the Key into the message there was a flag name keyed.message which need to be enable ("YES" value) like the image attached

I also took a look into your test plan file and fixed there.
testPlan - Copy.jmx.txt

Just remove the ".txt" and you will be able to try it.

Cheers!!

[vitalee78]

Hi, Jose!
Thank you very much, I`ll definitely try today.

[vitalee78]

Hi, Jose!
if look through ./kafka-avro-console-consumer then a error

but on version 2.1.3 it worked in avro format

if through ./kafka-console-consumer then no errors

[kszosze]

HI Vitaliee,
my guess you are using the generic kafka sampler, that one only send the message as a normal Json so yo can read it with the normal kafka-console-consumer.. if you are expecting the binary payload that don´t do this transformation. May be you should try the kafka Sampler or the Confluent one which let you configure which serialiser class should be used.
That solve your problem?
Cheers!

[vitalee78]

Hi Jose!
I try Confluent, the message is not generated so the kafka is empty and the error is log Jmeter as follows

and I use serialiser class - GenericRecordSerializer for key and value.serializer

If Kafka Sampler, the message is generated and sent to kafka, but the error in kafka-avro-console-consumer...

and there is still a message in the log Jmeter - WARN, but maybe it`s not critical?

Help me understand what I'm doing wrong?

[kszosze]

HI @vitalee78
is strange, can you please attach your latest test plan and current schema? I'll try to debug it and help you.

Cheers

[vitalee78]

TestPlan_schema.zip

[kszosze]

Hi Vitaliee,
May is a matter of configuration.. I can send messages without no problem with generic and kafka, Confluent is failing because it cannot solve the schema name, but this is issue of my setup I guess.

can you please try with this test plan?

testPlan_KLoadGen.jmx.txt

I´m getting messages in the topic.....

[vitalee78]

Hi kszosze!
you are not tired of me yet? ))

yes it also works for me if I look in kafka-console-consumer
and if you look in kafka-avro-console-consumer, then an error.

Our application reads in the topic in avro format and therefore I check in kafka-avro-console-consumer
Also the app reads the key / value message into the GenericRecordSerializer
if in the key GenericRecordSerializer error

[kszosze]

Hi Vitaliee,
is not a problem help and not, I'm not tired of you.
What I'm seeing is you are using the same serialiser for key and value. That's the problem, Key is not an Avro document, so this serialiser should be use. Can you please change it back to the StringSerialiser one please?
org.apache.kafka.common.serialization.StringSerializer
I think that gonna work for you.
Cheers

[vitalee78]

yes, did the key in two versions - StringSerialiser works
And I realized that I misunderstood the task, talked to a colleague developer. Our application reads separately the key with its avro schema and the value with its avro schema
And I don’t understand, but what is the correct way to draw up a test plan and it is possible to do this in a plugin?
We need the key to be sent to avro too
schemas.zip

[kszosze]

Well, that's unusual, key usually are not codec in avro unless you where using the confluent system. Can you try using the io.confluent.kafka.serializers.KafkaAvroSerializer serialiser for the key?

[vitalee78]

yes, we are using the confluent system

I tried with ...KafkaAvroSerializer for the key

[kszosze]

If you are using the confluent then your better option is go with the Confluent Sampler, it uses both confluent serialiser by default.. you tried that rigth? and give you some topic related issue?

[vitalee78]

Yes, tried Confluent Sampler, also an error as in the last screenshot
and give me please some topic related issue

[kszosze]

Ok I'll take a look into it today with your example test plan and back to you asap.

[kszosze]

Ok I´m gonna need the key schema too in order to see why it´s not finding it the schema registry. Can you provide it also?
Cheers!

[vitalee78]

schemas.zip
key schema and value schema

[kszosze]

Hi Vitaliee,
after saw your Key Schema I realize we cannot handle it now. But we can implement it, it will take a couple days.
Are you interested in this feature so we will implement it?

Cheers

[vitalee78]

Hi kszosze!
Yes, we interested in this feature!
Thank you!

[kszosze]

Privet Vitalee,

There is a branch linked to this ticket which a first approach for your request.
Is a breaking change, so you are gonna need to rewrite/tweak your test plan.
I upload one as an example about how to use it with your provided schemas.
Wish this gonna work for you.
TestPlans.jmx.txt
Cheers!!

[vitalee78]

Hi kszosze!
I will try to do it today

Thank you!

[vitalee78]

Hi kszosze!
trying to subtract testplan, error

as I understood in the block

[kszosze]

Hi @vitalie,
are you using this branch https://github.com/corunet/kloadgen/tree/%2368_adding_avro_key_support ?
Can you please upload the JMX?
Cheers

[vitalee78]

sorry, I used old release
It works and now I can see our message, but in the topic the message sent to String?
because if you look in kafka-avro-console-consumer then the error

if you look in a standard consumer (kafka-console-consumer) it works fine

my JMX
TestPlans - Copy.jmx.txt

[kszosze]

Ok i'll take a look asap

[kszosze]

Ok i see.. Let's gonna change the Test plan for something like this...
ConfluentkafkaSampler as Java Request Sampler
and serializers should be io.confluent.kafka.serializers.KafkaAvroSerializer.

like this image

then you can see something like

using this command

kafka-avro-console-consumer --bootstrap-server kafka:9092 --topic matomo.matomo.matomo_log_action --group matomo

[vitalee78]

ok, I'll see it now

[vitalee78]

did, but I can't, there is an error on the jmeter side
message is not sent

[kszosze]

Which version of JMeter are you using? It sounds the plugin jar hasn't build properly. That class is added in the build process as a dependency. Can you please double check there is only one kloadgen-plugin-X.X.X.jar in the lib/ext folder

[kszosze]

Hi Vitaliee,
some new code had been merge in to master and bring it to this issue branch. There you will find a new Sampler, GenericBinaryKafkaSampler. Do you mind try this one?

Cheers!

[vitalee78]

yes, in the lib/ext folder is only one, kloadgen-plugin-3.1.0.jar

and I try new code

[vitalee78]

I tried using GenericBinaryKafkaSampler but again the error is like in the last screenshot
it looks like I'm doing something wrong

[kszosze]

Well. does is really odd. Can you please send me yout apache-jmeter.... folder to my email jegarcia at corunet dot com

[vitalee78]

Hi Jose!
There was a conflict with another .jar, I removed it and there is no error

and reproduced another error, applied GenericBinaryKafkaSampler, GenericConfluentKafkaSampler and GenericKafkaSampler
also in the message the key missing {idaction: 123}

[kszosze]

well, we are getting better ... I'm asuming those errors are with the latest jmx file I sent to you or the one you sent to me or one new mix of both? I need to know in order to solve you problem.

Cheers!!

[vitalee78]

with my JMX
TestPlans - Copy.jmx.txt

[kszosze]

I believe you are not using the latest version of that branch.. mainly because with just your jmx and only changing port of Schema Registry and Kafka brokers it give a good answer

[vitalee78]

I using this branch - https://github.com/corunet/kloadgen/tree/%2368_adding_avro_key_support, right?

[kszosze]

That's the right one. Just was a merge problem with latest changes in Master. I just fix and push them. Can you try it.
Sorry for that.

[vitalee78]

yes, it works right now!

though for some reason in the 'View Results Tree' report there is red text, but there are no errors in the kafka there are messages in avro

at the moment it suits us, thanks!

[kszosze]

Red means There is an error during the process. Please check the response, it should show the broker answer.

[kszosze]

Hi Vitaliee,
any news about the red thing or this branch let's you run your tests?
Cheers

[vitalee78]

Hi, Jose!
our testing is working fine now,
but with red text, I did not find the reason, during load I disable this report

[kszosze]

Strange, may be in the response data you can find some answers.
I'll close this issue then.
Cheers

[vitalee78]

in the response data - topic: name_topic, partition: 0, offset: 2358125
I understand there is no crime

[kszosze]

Forget it.. it was a tiny error in the response build. Is gonna be fixed in the next commit.
Sorry and thanks for the feedback!

[vitalee78]

ok,
that is, then I can take the build from the master?

[kszosze]

Once I'll do the merge.. for now all changes are over the branch

[vitalee78]

ok,
thank you for the work done, now we are using it for our tasks.
we have another point when generating data, but this is not critical, so maybe we will turn again