Tunable partition choosing strategy?

[helpshift]

Tunable partition choosing strategy?

[bosky101]

it is supported in a different branch, i'll merge into master asap.

thanks for reporting this.

[qrilka]

Perfect, does it contain some description or testcase to see how it could be use?

[qrilka]

If it takes some time to merge then what about sharing that branch?

[mitchellwrosen]

What's the status of this?

[qrilka]

I have emailed @bosky101 a couple of weeks ago about it but with no replies yet

[mitchellwrosen]

I have looked into this a little bit, and at first glance it looks like it would require a significant refactoring of the code. Currently, a "worker" (ekaf_fsm) represents a connection to a broker-topic-partition triple. There's a named process group for each topic, and workers are essentially just selected randomly using pg2:get_closest_pid/1. Does that sound right, @bosky101? And what about the feature branch that you alluded to above?

[bosky101]

Every topic has an ekaf_server worker that picks a worker which you rightly Sai's is ekaf_fsm.

Based on the strategy ekaf_server does:

• round robin
• or random
• sticky round robin until a batch is full

So adding a strategy - custom would suffice where you can pass a user defined partition picker.

Option 1
This partition picker will be passed I to every publish
Pros: different messages In the same topic can be routed differently
Cons: pass in the picker function to every publish

Option 2
Pass in topic partition picker as an env before ekaf starts
Cons: can't change later

Option 3
API to set / unset partition picker at a topic level

Option 4
You can also independently call ekaf_picker btw and publish via it. This moves the logic to the users of ekaf , and can be considered more advanced. And I can expose broker, worker states so that anyone can build a partition picker.

Thoughts ?

~B

On 11-Dec-2014, at 7:35 am, Mitchell Rosen [email protected] wrote:

I have looked into this a little bit, and at first glance it looks like it would require a significant refactoring of the code. Currently, a "worker" (ekaf_fsm) represents a connection to a broker-topic-partition triple. There's a named process group for each topic, and workers are essentially just selected randomly using pg2:get_closest_pid/1. Does that sound right, @bosky101? And what about the feature branch that you alluded to above?

—
Reply to this email directly or view it on GitHub.

[qrilka]

I think "native" Kafka approach (and which is supported e.g. by kafka-python) makes much sense: you set partinioner on per producer basis e.g. see https://github.com/mumrah/kafka-python/blob/master/kafka/producer/keyed.py#L30
As far as I understand currently there is no entity like producer in ekaf so that seem to be the reason why this question appears.
Having partitioner per topic seems OK for me but I don't have much experience with different Kafka usage scenarios so I'm not 100% sure.
@bosky101 should we assume your question as a sign that there is no branch with such feature yet?

[mitchellwrosen]

@bosky101
I think option 3 sounds the most reasonable, as the exact partition to write to should be exposed in ekaf just as it is in kafka. I'm not sure what you mean by option 4, can you give an example of manually using ekaf_picker to pick a worker at partition N? As far as I can tell, although workers are aware of the partition they're writing to, there's no mapping from partitions to workers. The best you can do is keep picking workers out of the topic's pg until one of them matches the requested partiton. Am I wrong about this?

Thanks,
Mitchell

[vglukhovskiy]

Any progress please on this ticket please?
We really need to have ability to control partitions. Any option would suffice.
Thanks

[RomanShestakov]

+1 - would be really really useful function, any plans to implement any time soon?

[bosky101]

Will add by November

Sent from my iPhone

On 24-Oct-2015, at 2:10 AM, RomanShestakov [email protected] wrote:

+1 - would be really really useful function, any plans to implement any time soon?

—
Reply to this email directly or view it on GitHub.

[gatesn]

+1 Will be really useful!

[RomanShestakov]

hi Bhasker - any progress with this ticket so far?

[joe-sinopoli]

this would be a great feature, what kind of eta are we looking at?

[jccampagne]

+1 - following thread/issue, this is something that would be very useful.
Thanks for the update and effort.
Looking forward to use this feature.

[bosky101]

Sorry for the delay, i'm done with my internal deadlines - will fix it this weekend.

Plan of action:

1. Expose current partitions, workers for a topic
2. If partition strategy is custom, then add a new app env that will call M:F/5 before picking a worker.
3. M:F/5 will be passed a dictionary of [{Topic, WorkerPids},...] and a CallbackPid
4. You must message pass {ok, PickedWorker} back to the CallbackPid (if the partition is down, that way you can still pick another partition/worker and pass to the callbackpid) or undefined

[RomanShestakov]

this is awesome, thanks a lot for update Bhasker! Really looking forward to it

[vglukhovskiy]

Great! Looking forward.

[bosky101]

@qrilka @mitchellwrosen @vglukhovskiy @RomanShestakov @Gatzy118 @jccampagne @thatJoeGuy16 Please test https://github.com/helpshift/ekaf/tree/feature/custom-partition-strategy out and let me know. Will merge into master when i have time to write tests.

[vglukhovskiy]

AWESOME! Will try tomorrow.
Many thanks

[RomanShestakov]

thank you very much - will check shortly!

[vglukhovskiy]

@bosky101,
We did manage to use your solution with a basic custom partition picker.
Many thanks one more time.

[RomanShestakov]

it works as advertised - many thanks for quick turnaround

[jccampagne]

@bosky101 thanks for the patch