Comments (25)
it is supported in a different branch, i'll merge into master asap.
thanks for reporting this.
from ekaf.
Perfect, does it contain some description or testcase to see how it could be use?
from ekaf.
If it takes some time to merge then what about sharing that branch?
from ekaf.
What's the status of this?
from ekaf.
I have emailed @bosky101 a couple of weeks ago about it but with no replies yet
from ekaf.
I have looked into this a little bit, and at first glance it looks like it would require a significant refactoring of the code. Currently, a "worker" (ekaf_fsm
) represents a connection to a broker-topic-partition
triple. There's a named process group for each topic
, and workers are essentially just selected randomly using pg2:get_closest_pid/1
. Does that sound right, @bosky101? And what about the feature branch that you alluded to above?
from ekaf.
Every topic has an ekaf_server worker that picks a worker which you rightly Sai's is ekaf_fsm.
Based on the strategy ekaf_server does:
- round robin
- or random
- sticky round robin until a batch is full
So adding a strategy - custom would suffice where you can pass a user defined partition picker.
Option 1
This partition picker will be passed I to every publish
Pros: different messages In the same topic can be routed differently
Cons: pass in the picker function to every publish
Option 2
Pass in topic partition picker as an env before ekaf starts
Cons: can't change later
Option 3
API to set / unset partition picker at a topic level
Option 4
You can also independently call ekaf_picker btw and publish via it. This moves the logic to the users of ekaf , and can be considered more advanced. And I can expose broker, worker states so that anyone can build a partition picker.
Thoughts ?
~B
On 11-Dec-2014, at 7:35 am, Mitchell Rosen [email protected] wrote:
I have looked into this a little bit, and at first glance it looks like it would require a significant refactoring of the code. Currently, a "worker" (ekaf_fsm) represents a connection to a broker-topic-partition triple. There's a named process group for each topic, and workers are essentially just selected randomly using pg2:get_closest_pid/1. Does that sound right, @bosky101? And what about the feature branch that you alluded to above?
—
Reply to this email directly or view it on GitHub.
from ekaf.
I think "native" Kafka approach (and which is supported e.g. by kafka-python
) makes much sense: you set partinioner on per producer basis e.g. see https://github.com/mumrah/kafka-python/blob/master/kafka/producer/keyed.py#L30
As far as I understand currently there is no entity like producer in ekaf
so that seem to be the reason why this question appears.
Having partitioner per topic seems OK for me but I don't have much experience with different Kafka usage scenarios so I'm not 100% sure.
@bosky101 should we assume your question as a sign that there is no branch with such feature yet?
from ekaf.
@bosky101
I think option 3 sounds the most reasonable, as the exact partition to write to should be exposed in ekaf
just as it is in kafka
. I'm not sure what you mean by option 4, can you give an example of manually using ekaf_picker
to pick a worker at partition N? As far as I can tell, although workers are aware of the partition they're writing to, there's no mapping from partitions to workers. The best you can do is keep picking workers out of the topic's pg
until one of them matches the requested partiton. Am I wrong about this?
Thanks,
Mitchell
from ekaf.
Any progress please on this ticket please?
We really need to have ability to control partitions. Any option would suffice.
Thanks
from ekaf.
+1 - would be really really useful function, any plans to implement any time soon?
from ekaf.
Will add by November
Sent from my iPhone
On 24-Oct-2015, at 2:10 AM, RomanShestakov [email protected] wrote:
+1 - would be really really useful function, any plans to implement any time soon?
—
Reply to this email directly or view it on GitHub.
from ekaf.
+1 Will be really useful!
from ekaf.
hi Bhasker - any progress with this ticket so far?
from ekaf.
this would be a great feature, what kind of eta are we looking at?
from ekaf.
+1 - following thread/issue, this is something that would be very useful.
Thanks for the update and effort.
Looking forward to use this feature.
from ekaf.
Sorry for the delay, i'm done with my internal deadlines - will fix it this weekend.
Plan of action:
- Expose current partitions, workers for a topic
- If partition strategy is custom, then add a new app env that will call M:F/5 before picking a worker.
- M:F/5 will be passed a dictionary of
[{Topic, WorkerPids},...]
and aCallbackPid
- You must message pass
{ok, PickedWorker}
back to theCallbackPid
(if the partition is down, that way you can still pick another partition/worker and pass to the callbackpid) orundefined
from ekaf.
this is awesome, thanks a lot for update Bhasker! Really looking forward to it
from ekaf.
Great! Looking forward.
from ekaf.
@qrilka @mitchellwrosen @vglukhovskiy @RomanShestakov @Gatzy118 @jccampagne @thatJoeGuy16 Please test https://github.com/helpshift/ekaf/tree/feature/custom-partition-strategy out and let me know. Will merge into master when i have time to write tests.
from ekaf.
AWESOME! Will try tomorrow.
Many thanks
from ekaf.
thank you very much - will check shortly!
from ekaf.
@bosky101,
We did manage to use your solution with a basic custom partition picker.
Many thanks one more time.
from ekaf.
it works as advertised - many thanks for quick turnaround
from ekaf.
@bosky101 thanks for the patch
from ekaf.
Related Issues (20)
- Move kafkamocker to test profile HOT 1
- ekaf tcp_closed HOT 1
- kafka close connection
- Unexpected behavior when calling process is actively receiving messages. HOT 9
- Partitioning strategy working with Kafka 0.10.0? HOT 1
- Can "kafka.bootstrap.server
- Can "kafka.bootstrap.server" be changed at runtime? HOT 1
- `ekaf_picker:pick_sync` does not consider custom strategy HOT 1
- ekaf app not restart when 6 erl node restart in turn HOT 1
- Bad behaviour when partitioning messages by key.
- In a distributed environment, ekaf_picker:pick(Topic) will fail if the node started first goes down HOT 3
- Multiple broker configuration HOT 3
- Async produce Message accumulation HOT 5
- how to set topic replication-factor in ekaf
- Kafka 2.0 support
- cause so many time_wait
- Why does Kafka build so many connections? HOT 5
- Random partition selection despite having custom selected HOT 1
- Kafka plug-in make error
- Kafka lose data
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ekaf.