Comments (1)
@cypriss thank you for reporting this issue and I'm sorry it has taken so long to reply. Jobs has not been a main focus for me for quite some time now. I now understand why this is happening.
Jobs performance scales terribly with queue size. If you look at the Lua script for popping the next available jobs off the queue, there are two ZUNIONSTORE
commands and one ZINTERSTORE
command. The performance of these commands is sensitive to the size of the input sets (O(N)
) and the size of the resulting output set (O(M*log(M))
). When the queue is large, both the input and output sets have high cardinality and therefore the script to pop jobs off the queue runs incredibly slowly. I should have benchmarked Jobs with larger queue sizes in order to catch this issue sooner.
There are a few ways we could fix this, including:
- Keep all existing features and rewrite the commands in pop_next_jobs.lua so that it performs better with large queue sizes. I'm afraid the best we might be able to achieve here is
O(N)
performance instead ofO(N*log(N))
, which won't matter much considering other libraries are able to pop jobs in a way that scales independent of queue size. The problem is that Jobs currently allows arbitrary priority levels and stores them in a sorted set where the members are job ids and the scores are the corresponding priorities. - Don't allow arbitrary priority levels. We can change Jobs so that the number of priorities is a parameter set during initialization. If we do this, we can have a single set for each priority level and consume from them in order (highest priority to lowest priority) until we have consumed the desired number of jobs. This will make the performance of pop_next_jobs.lua dependent only on the number of priority levels and the number of jobs consumed (
O(P + J)
, I think). Users who don't need different priority levels can set it to 1 and expect similar performance to competing libraries.
I believe 2 is the best option but would involve breaking backwards compatibility. I can't make any guarantees about when I will have time to do this, but I would be more than happy to review a PR if someone wants to take this on.
from jobs.
Related Issues (20)
- Use go generate to convert lua script files to strings HOT 4
- Use an interface for jobs instead of a non-type-safe handler function (would break backwards-compatibility) HOT 4
- Support redis sentinel HOT 6
- Export job.freq HOT 4
- Reschedule with freq HOT 2
- FindById() doesn't return error for missing jobs HOT 7
- Add a changelog HOT 2
- Panic in xen container HOT 4
- Job runtime error messages should include a stack trace
- Error when calling Destroy() from within a job handler function HOT 8
- job not exec with panic HOT 8
- FindById and job identifiers/human names HOT 6
- Examples?
- Passing params / string name of job. HOT 2
- Endless loop? HOT 6
- Error Handling? HOT 5
- Application can't start due to error which message is 'jobs: In scanJob: Could not find Type with name = %s' HOT 1
- Support environmental prefix
- Job status is changed to executing but job has not started execution
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jobs.