tryfix/kstream

Apache Kafka Streams for Golang

KStream - Kafka Streams for Golang

KStream is a light-weight implementation of kafka streams. It is heavily inspired by Kafka-Streams(Java) library. KStream implements features like Internal Stores, Remote Stores, Local Tables, Global Tables and it guarantees Exactly Once Processing with its de-duplicator. It can process message as micro batches or one by one depending on the throughput required. KStream can handle good amount of throughput (50,000 messages pe second with micro batch enabled) in a fault tolerable manner with a very minimal amount of latency (2 ~ 5 ms)

Project uses two external libraries sarama for consumers and producers

Stream Components

Stream Topology

Stream Topology is a set of processor nodes typically starts with a source node and ends with a sink node

Stream Builder

Stream builder is responsible for building stream topologies into kafka streams with their dependencies like changelog topics, re-partition topics, etc...

KStream

KStream is a regular kafka stream which takes an input topic as an upstream and process the record stream to another kafka topic(downstream topic) It supports several functions like transforming, merging to another stream, joins with other streams etc.

KTable

@TODO

Global KTable

Global KTable also a KTable except for is each instance of the application has its own copy of all the partitions and it will be running on a separate thread so applications dose'nt have to worry about handling failures.

Store

Store is a pluggable local key-val store which is used by KTable, Global KTable and other oparations like joins, merges and removing duplicates.

Store Backend

Store is a pluggable local key-val store which is used by KTable, Global KTable and other oparations like joins, merges and removing duplicates.

RPC layer for Store

@TODO

Key discovery service

@TODO

kstream's Issues

Global Table sync stuck

Global Table sync gets stuck if the last message is a control record(https://kafka.apache.org/documentation/#controlbatch)

Add topic name to the log upon produce failure on invalid or non existing topics

cannot send message on (github.com/tryfix/kstream/producer.(*saramaProducer).Produce) due to kafka server: The request attempted to perform an operation on an invalid topic. on func github.com/gorilla/mux.(*Router).ServeHTTP

Filter Process return next false

When there is a filter process and filter func returns false, then other child streams in the parent's stream not running because filter process returns next = false.

Divide by zero panic in sync progress log

global_table_stream_instance.go L117 cause to panic if partition end is zero and log happens before partition end trigger the case.