ottbot/greta

A Kafka client for Clojure projects

greta

A modern and (fairly) low-level Kafka client for Clojure projects.

It implementments the API introduced in Kafka 0.8.3 as an Aleph TCP client.

Development targets Kafka 0.9.0+.

Usage

FIXME

Contributing

The tests assume that the greta-tests topic already exists, and that topics are not automatically created. This not default, so you must add auto.create.topics.enable = false to your broker configuration.

License

Copyright © 2015 Funding Circle

Distributed under the BSD 3-Clause License.

greta's Issues

A producer with RequiredAcks = 0 will break correlated requests

The broker doesn't send a response with when required acks is 0. I reckon this will cause problems with the correlated codec.

Support multiple apis on single connection?

Pretty straight forward to find the right codec via api-key for the requests (using gloss.core/header).

Harder to get the codec to use on the response, since only the correlation-id is returned, so some state is needed. Which begs the question if we should be correlating the correlation-id anyway..

The plan is see how using a connection per api works out, and change it if it becomes unwieldy.

Trade off is some duplication vs complexity.

Add TLS support w/ Kafka 0.9

This may "just work" by enabling ssl in the aleph tcp client.

Metadata API

• Send metadata request
• Read metadata response

Producer API

• Make produce request
• Read produce response

Deal with small max-bytes in fetch

If max bytes is small, say less than 80, kafka won't send enough of the response to get to the codecs that deal with truncated/partial messages.

We should deal with this better than letting a gloss exception (assert failure) raise, but not a huge priority.

Implement client API

After implementing the kafka API, we need some nice function to use it.

Consume API

• Make consume request
• Get messages from Kafka

Remove Eclipse Public license

📝 This project is currently proprietary and should have a commercial, non-distributable license attached.

Add codox

Only a single serde per-client is supported

Is this a problem? Should we pass more information to the serde (topic/partition info)? Right now it just assumes a simple bytes to string use-case, and would not enable to, say, support a confluent schema registry as-is.

A client could always create a client with the identity-serde and handle it as needed.

Offset API

• Make offset request
• Read offset response
• Make consumer group metadata request
• Read consumer metadata response
• Make offset commit request
• Read offset commit response
• Make a offset fetch request
• Read offset fetch response

KafkaSerde should only get map with key and value

We shouldn't rely on getting the attributes and magic byte back from the KafkaSerde -- just give it the key and value and only expect to get that back.