producer: ensure distributing messages with the same key to the same partition

[kafka-rust]

producer: ensure distributing messages with the same key to the same partition

[datariot]

This is a requirement for some work I'm about to do. I'm new to Rust (coming from Java/Scala + Kafka background) but I could take a stab at this unless you're already working on it.

[xitep]

hello @datariot, it wasn't a priority for me right now, but i could give it precedence since the changes will have impact on other issues i planed to resolve. unfortunately, this will include some public api changes, though not as big as in the last 0.2 release.

[datariot]

I feel partitioning by key is a pretty important part of a scalable Kafka project, so if it's going to impact other things it might be good to prioritize. Also it would selfishly help me ;-)

[xitep]

@datariot i just pushed two very experimental commits to branch consistent-partitioning allowing the partitioner to obtain the total partition count for a topic. this would allow you to implement consistent hashing in your own, custom partitioner. feel free to experiment with it. please note, this is really just a very first draft and i surely don't want to promise this API will be preserved (but it's going to be something like that.)

[xitep]

Hello @datariot,

so I took some the time to get this done. Can you review/try out branch consistent-partitioning? I'd be greatly interested in feedback.

So far, I've chosen SipHash since that is part of the std library, so no extra dependencies are required. However, I managed to provide the possibility for a user to define a different hash implementation through the pubilc API of the producer::DefaultPartitioner. This can give us room for experiments until we make a definite choice. So far I've looked, it seems xxhash might be a better alternative. (fnv is probably the fastest if one's keys are short - but that would be a bad default choice from our perspective, I think.) Do you have any opinions/suggestions on this?

Hope this works as expected. P.

[datariot]

Awesome @xitep! I'll try to set aside some time today or tomorrow to check this out.

[xitep]

@datariot have you had already a chance to play around with this so far? just curious.

[datariot]

@xitep yes, this is working as expected. I haven't put it under load so I don't have an idea on how SipHash performs. We usually partition on a String that is between 10-30 chars long.

[xitep]

@datariot many thanks for the confirmation. i'll merge the PR then. regarding SipHash i think it's fine enough as a default.

[xitep]

so it's in master now. however, in the end i decided to release it as part of 0.3.0, after implementing a few more issues. feel free to re-open in case of troubles. and many thanks so far.

[xitep]

Re-opening until this gets released with 0.3.0.

[xitep]

Closing this one; just released 0.3.0.