Comments (8)
Ok, mystery solved.. My task object had a non-unique id field derived from my input and I didn't explicitly configure a unique id so a large number of tasks were getting merged. Thanks for your help in troubleshooting.
from better-queue.
Node will exit if it does not have any callbacks waiting. Try hooking an event to drain or put a callback that waits for the last event to finish to prevent the process from exiting.
from better-queue.
Awesome, thanks for your help.
from better-queue.
I attached a listener for the drain event and it is firing only once, just before exiting. There are still 11.5M tasks remaining at that point. What is the best way to see how many remaining tasks are queued? My intuition is that tasks are getting lost somehow, as I am not using a persistent store, and I would expect memory issues with that many queued tasks.
from better-queue.
You can try calling queue.getStats()
to see the total. It would be a bug if it calls drain before it's completed all of the tasks unless tasks were getting completed faster than they are being pushed on. Do you have some code to reproduce this?
from better-queue.
Yeah, I am calling getStats on that drain event, but that is telling me how many have been completed, not how many are remaining, right? Any tips on getting better visibility into queue state would help a lot.
I do have code which I can share, but it currently involves parsing the english wikipedia article dump and posting it to elasticsearch. I can try to reduce it to something a bit more manageable, but it will take some time.
from better-queue.
One way you can get more visibility is by passing in your own store (or modifying the memory store) and adding a counter. You really only need to implement 5 functions or stub them out to see if it's working properly.
https://github.com/diamondio/better-queue#custom-store
from better-queue.
Ok, great. Let me give that a go and I'll report what I find.
from better-queue.
Related Issues (20)
- What is the suggested highest value for concurrent option? HOT 2
- task deleted and re-added on maxTimeout HOT 1
- TypeScript Support HOT 2
- maxRetries is not applied for tasks in the store
- 🐛 Large data in the store caused the app to crash
- How to update file progress?
- How to edit file progress?
- batchSize bigger than 1 => "TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined" HOT 1
- maxTimeout issue?
- Most recent version no longer works with PostgreSQL store?
- It is unclear in readme.md what time uint `getStats().average` is
- Why does `maxTimeout` have no effect on `getStats().average`? HOT 4
- in `on("task_failed", handler)` , when maxTimeout error is emitted, the `error` input must be an error object
- Add CI to GitHub repo HOT 1
- Oversight on task_progress event? HOT 5
- maxRetries is not honored when tasks with same id are pushed into the queue HOT 1
- Delete a task? HOT 3
- Task queue memory leak
- When retry, queue not processed in order HOT 3
- getRunningTask is not defined in the store interface HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from better-queue.