Comments (5)
I did a little bit more digging and it seems like my workerManagers aren't closing due to the lock from start batch. My callback for WorkerFailedAttemptCallback
and WorkerFailureCallback
are both doNotCommitAndStop
.
I only have the 10 default workers and 14 partitions running (14 worker managers). It seems like what's happening is that the worker stops, but some partitions are running startBatch
, which waits on an available worker, but since my callback stops the worker, there are no workers and the call blocks indefinitely. I bumped up the number of workers to 30 in an attempt to free up more workers, but it seems like that's not enough. I'm not sure if adding more workers is the solution here. Why doesn't the Close()
call send a kill signal to the workers?
from go_kafka_client.
Another alternative is to have startBatch
only block for a certain amount of time and then quit. However, I can see how that wouldn't be the best solution.
from go_kafka_client.
I presume you want to close the consumer from the messagehandler. I also got these problems, and my fix
was to make a wrapper around the library; put a message processed
item in a channel from the messagehandler if you want to continue processing; and in new mainloop stop if no message processed
have been recieved on that channel after x seconds.
Not really nice, but for the most part it got rid of the fatals for me.
I agree this should be handled more graceful.
from go_kafka_client.
I'm calling Close()
from the consumer that comes out of kafka.NewConsumer(kafkaConfig)
. @baconalot What's the messagehanlder you're talking about?
from go_kafka_client.
it's a dead lock when startBatch and Close all try to hold the stopLock in worker manager
and at the same time Close's managerStop channel is waiting on startBatch to finish and select data from the managerStop channel.
I don't see why the wm.stopLock is necessary here.
from go_kafka_client.
Related Issues (20)
- 76% of time are `runtime.goexit` HOT 11
- go get -u github.com/stealthly/go_kafka_client throw error
- Manual commits HOT 1
- low level consumer throws invalid memory address on calling go_kafka_client.(*SaramaClient).Initialize HOT 1
- panic when i start use config.OffsetStorage = NewSiestaClient(config) HOT 8
- Documentation
- SiestaClient not commiting offsets with Go Kafka Client HOT 1
- Build out a framework for Go Kafka Client on Mesos HOT 3
- Panic thrown when Zookeeper Node Restarts HOT 3
- Godeps lists the wrong version of jimlawless/cfg HOT 1
- No API document? HOT 2
- BUG?: why not break after handle all request?(tryRemoveOldApiRequests) HOT 2
- when consumer connect to zk fail, it create a large number connect in short time.
- Can't find jimlawless/cfg HOT 1
- Install Error: HOT 1
- Create topic
- Unable to store consumer offsets in kafka HOT 1
- Failed awaiting on state barrier HOT 1
- Clarify relation to siesta
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go_kafka_client.