New BPL library for Kafka

[brain4it]

New BPL library for Kafka

[raneq]

From the core apis, we don't need to implement all of them:

• Consumer: yes. To un/subscribe from/to topics and read from them.
• Producer: yes. To read from topics.
• Streams: maybe. A stream reads from a topic, transforms data, and writes to another topic.
• Connect: no. Kafka connectors are internal consumers (sink connectors) or producers (source connectors) loaded at Kafka Connectors, managed by the same Kafka platform.
• Admin: yes. To create or delete topics straight from brain4it.

[raneq]

Some notes on...

Clustering

• Kafka can be run in a cluster of nodes, where the message processor are called brokers.
• Each topic is split into a pre-defined number of partitions, and each partition is replicated with some replication factor. That means that not all brokers will have all of partitions
• For each (replicated) topic partition, there is exactly one broker that is the topic partition leader.
• Producers can only write to the topic-partition leader
• Consumers can only read from the topic-partition leader
• All Kafka nodes can answer a request for cluster state metadata about which nodes are alive and who are the leaders for each partition of each topic.
• Clients should know more than one cluster server (bootstrap nodes), but don't need to know about all of them beforehand, as all nodes know about all others in the cluster.
• Ideally, There are more topic partitions than brokers, and each broker is leader of a proportional amount of partitions, so that there isn't a single leader-of-all broker.
• Producers control which partition they publish messages to.
  WIP

[realor]

This library could have the following methods:

(set kid (kafka-create "producer|consumer" config))
(kafka-send kid topics message)
(kafka-poll kid topics millis)
(kafka-destroy kid)

Examples:

(set kid (kafka-create "producer" "localhost:9092"))
(kafka-send kid "containers" "hello world")
(kafka-send kid ("containers" "lights") message)
(kafka-destroy kid)

(set kid (kafka-create "consumer" "localhost:9092"))
(set msg (kafka-poll kid "containers" 10))
...
(kafka-destroy kid)

(set kid (kafka-create "consumer" ("servers" => ("10.1.20.87:9092" "10.1.20.88:9092") "key-deserializer" => "StringDeserializer" "value-deserializer" => "StringDeserializer" "group" => "test-group")))