Comments (10)
I think for critical applications reducing the timeout is a must anyway -- because even if celery was cleaning up the lock, the instance may still go down ungracefully and the lock won't be released fast enough.
from redbeat.
I believe upstream celery should be setting up the proper signal handlers and then calling the close() method. If that is all working properly then I don't think there is much redbeat can do. Reducing the loop interval would make the CPU and Redis busier, which may or may not be a concern. You could also adjust the redbeat_lock_timeout
. Changing the lock timeout either way is a workaround rather than a proper solution.
from redbeat.
I suspect there is an old lock that needs to be cleaned up, this PR seems like it might be related, https://github.com/sibson/redbeat/pull/84/files
from redbeat.
Thanks for the hint. What do you think of this PR? Not sure how i'd be able to get this into a production setup without having it be part of a PyPI package (would prefer to avoid pulling code from Github and using a development branch to provision the production setup).
But actually, i have the up to 25 minute delay before it says beat: Extending lock...
, so I'm not sure that this PR could impact that, because it only takes over at that point, at which the delay already happened if i understand correctly.
from redbeat.
hmm, I think you need is a non-blocking call to aquire, additional logging and retry on failure in https://github.com/sibson/redbeat/blob/master/redbeat/schedulers.py#L419-L426. The quick answer would be to redis-cli into the server and inspect the lock values to confirm it hasn't been cleaned up. The other piece to confirm is the cleanup code is being run when you shutdown the old version before deploy, https://github.com/sibson/redbeat/blob/master/redbeat/schedulers.py#L390-L393
from redbeat.
Yeah I'll check if the lock is set. I'm using Beanstalk with a multi-docker setup, not sure if Beanstalk does a graceful shutdown or just kills the celerybeat docker container. Seems to me it just kills it and the lock would be left over, which would be consistent with the delays of around 25m i've been seeing.
from redbeat.
Just confirmed that Elastic Beanstalk is not doing a graceful shutdown and the lock is not released when the instance is removed.
from redbeat.
What do you think could be approaches for redbeat not being shutdown gracefully, and having the lock left over?
from redbeat.
Would there be any downsides to reducing beat_max_loop_interval
to something like 20 seconds, so the default lock period would be 100 seconds?
from redbeat.
Update: even if I open a shell into the Docker container and kill the celerybeat process, it does not do a graceful shutdown, still leaves the lock in redis. btw, using latest redbeat 0.11.1
from redbeat.
Related Issues (20)
- Unable to Schedule Different Timezone per different task
- Celery 5.3.0rc1 not useable with redbeat anymore. HOT 3
- Redbeat doesn't work with celery 5.3.0 HOT 2
- Updated to 2.1.1 from 2.0.0 now it fails to run. HOT 6
- Retry did not work after (104, 'Connection reset by peer')
- The distributed lock set to None failed HOT 3
- Support for limit run option to avoid delay tasks that previous not finished
- tzinfo utcoffset taking None as parameter
- NoneType object has no attribute total_seconds
- RedisCluster is on EOL HOT 1
- [Question] Ensuring task is scheduled HOT 1
- Cannot import FixedOffset when running with celery>5.2.7
- New bugfix release HOT 2
- redis_url is not recognized HOT 3
- Conda packages for version higher than 2.0.0 HOT 1
- Usual way of defining periodic tasks in celery not working as expected
- High Availability?
- Different Application sharing same redis instance
- Tasks over 5 Minutes
- AttributeError: 'PersistentScheduler' object has no attribute 'lock_key'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redbeat.