Comments (6)
That actually sounds like quite a smart solution to me as it combines the limited resource usage with a timeout in an easy to use way. I can try to spin up a PR for that, too, but it won't be this week.
from clustermanagers.jl.
Perhaps we should put a sleep(0.1)
or something at the end of the loop so that it doesn't spin out of control?
EDIT: Reading comprehension fail, you already said that 🤦
from clustermanagers.jl.
One additional option might be to add in some logic so that if it's stalled for more than X seconds, it errors out. I have no sense of what a good default for X is, or if it should be user-configurable, but could open a separate issue for that if it's outside the scope of the PR you want to make. I'm fine with a simple addition of sleep if you want to spin that up
from clustermanagers.jl.
Opened a pull request for the sleep addition. Adding a timer would also be helpful I think but I don't believe there's a good "default" for it - depending on how busy a cluster is the usual waiting times might differ by a lot.
Another thing that is kind of related is that this loop also doesn't stop if the job gets cancelled before it has actually been started which of course would also more or less be handled by a timeout.
from clustermanagers.jl.
Fwiw, the lsf manager lets the user supply an iterator with retry delays. The default is an exponential backoff with a max number of attempts, but I tend to just use Iterators.cyclic(5)
to not have to deal with timeouts as the startup time at my place is extremely unpredictable.
Line 43 in 861b301
from clustermanagers.jl.
Alright, I'm gonna merge the PR for now, but looking forward to next one that superceeds it
from clustermanagers.jl.
Related Issues (20)
- Extra options on SGE HOT 5
- Error in `rmprocs` SGE HOT 1
- Ship telnet via jll? HOT 2
- addprocs(SGEManager) fails HOT 5
- SGE fails in rmprocs
- Singularity images does not work with SLURM HOT 5
- Error launching workers: no such file or directory HOT 5
- TagBot trigger issue HOT 8
- lsf_bpeek makes strong assumptions on iterator state of retry_delays
- Better handling of SLURM job submission timing
- Handling of busy LSF deamon HOT 4
- SLURM 10 nodes good, 16 nodes error HOT 3
- pbs error HOT 4
- LSF manager broken in Julia 1.8.1 HOT 2
- -o argument in addprocs_slurm leads to an error
- ClusterManagers can be run on top of dask clusters! HOT 2
- Elastic auto IP address function HOT 2
- Limiting number of cores per node on with LSF HOT 3
- Finalizer task switch bug
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clustermanagers.jl.