trendyol/confluent.kafka

confluent.kafka's Issues

Integrate with AppVeyor

We should integrate a distributed continuous integration service like AppVeyor.

Rename Project

Rename Project to libdgckafka.

Provide Configuration for AddKafkaConsumer

Currently, AddKafkaConsumer extension method is overloaded and accepting a lot of different parameters. One solution is builder pattern but I think it will be easier to provide a KafkaConfig class and expect it in AddKafkaConsumer method to provide extra configuration of kafka clusters, brokers, producers and consumers.

Fix Producing back to Main Topic

We shouldn't produce event to main topic in case of failure because other consumers in other domains can listen to our main topic.

• Produce to "retry" topic.
• Consume "retry" topic in specific periods.
• Put unconsumed messages to "error" topic.

Add Dockerfile

We should add Docker support.

Support for Custom Serializer/Deserializer

As of now,

• Primitive types is serialized/deserialized with big endian byte ordering.
• String type is serialized/deserialized with UTF8 charset.
• Custom types is serialized/deserialized in JSON format.

We can support for Serializer/Deserializer for types TKey and TValue for the type that derives from KafkaConsumer.

Provide Extensive Docs with Summary

Provide .NET docs and heavily comment the code for better understanding why we're doing what we're doing.

Enable Nullable Reference Types

Nullable reference types are very helpful in preventing bugs and NullReferenceExceptions from occurring.

Add Retry Consumer and Retry Topic

Producing failed messages back to main topic can cause many confusions in other consumers that are successfully parsing and processing messages. In addition, there will be multiple processing of messages in other consumer groups that are listening to the same topic.

Built-in support for Retry consumer will fix these issues. Producing unprocessed messages to retry topic and periodically consuming those messages is one strategy.

Test RetryConsumer

We need to run a test to see whether _retryConsumer works.

Run nuget publish Automatically

Setup some CI/CD pipeline to run nuget publish for every commit in master branch.

Add Unit Tests in Project.

Begin by adding a Unit Tests Project and adding tests for MessageExtensions.

Make KafkaConsumer Generic

Currently, KafkaConsumer deserializes messages in UTF8 string format.

We could support deserialization/serialization of complex types.

Example:

public class ProductConsumer : KafkaConsumer<Ignore, Product>
{
    // ...
}

Implement Builder Pattern in ServiceCollection

Currently we have many overloaded constructors in ServiceCollectionExtensions. We can provide ease of use with builder pattern or options parameter.

Example:

services.AddKafkaConsumer(options => {
    options.Topic("topic");
    options.RetryTopic("retry-topic");
    options.BootstrapServers("my-bootstrap-servers");
})

// Or like this:

services.AddKafkaConsumer()
    .Topic("topic")
    .RetryTopic("retry-topic")
    .MaxRetryCount(5);

Support Serialization/Deserialization of Custom Types

We should support both serialization/deserialization of custom types using JSON format. (Other formats can be provided in the future)

We should also support string serialization/deserialization in UTF8 format. (Other formats can be provided in the future)