saabeilin/kafkian

An opinionated Kafka producer/consumer built on top of confluent-kafka-python/librdkafka

kafkian's Issues

[Feature Request] Ability to Disable Logging

I really love this module, but would love the ability to disable logging. There are times when I would rather not have message contents in our Kibana logs, plus the other output puts unnecessary noise in the logs. I love the logging during development, however.

How to stop consumer.

I know this is a silly question, but I don't know how to do that.
I try to check if message is None, but the consumer just stops consuming if there is no update, and message is the last message.

Could you please tell me how to do that?

for message in consumer: if message is not None: consumer.commit() else: consumer._close() break

How to get the schema from message which is avro type.

i want to upload the kafka message to bigquery , for that i need schema of the object, or directly pass avro object to biquery because avro object contain schema itself

for msg in consumer:
msg.schema

RecordNameStrategy and TopicRecordNameStrategy should not have suffix.

According to the following java code,
https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/subject/TopicNameStrategy.java

https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/subject/RecordNameStrategy.java

https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/subject/TopicRecordNameStrategy.java

I think,

• in case of RecordNameStrategy, a subject must be the <fully-qulified avro recordName>.
• in case of TopicRecordNameStrategy, a subject must be the <topicName>-<fully-qulified avro recordName>.

But,
https://github.com/saabeilin/kafkian/blob/master/kafkian/serde/serialization.py#L48
all strategy has -key or -value suffix (subject += '-key' if is_key else '-value'). These suffix should be applied only on TopicNameStrategy.

Anyway, I feel confluent-kafka-python is not user-friendly and does not support Subject Name Strategy well.
Thank you for your nice library!

[Feature request] Send statsd/dd-statsd metrics

Would be great to send [some] stats via statsd/dd-statsd. For example, backport from https://github.com/fyndiq/confluent_kafka_helpers/ - @simonwahlgren ?

README example gives error

What is the way to instantiate user in the README example? It gives error:

$ python3 produce.py 
Traceback (most recent call last):
  File "produce.py", line 48, in <module>
    user.uuid,
NameError: name 'user' is not defined