[SlurmManager] 100 % CPU usage while waiting for the job to get created

[David96]

[SlurmManager] 100 % CPU usage while waiting for the job to get created

[David96]

That actually sounds like quite a smart solution to me as it combines the limited resource usage with a timeout in an easy to use way. I can try to spin up a PR for that, too, but it won't be this week.

[kescobo]

Perhaps we should put a sleep(0.1) or something at the end of the loop so that it doesn't spin out of control?

EDIT: Reading comprehension fail, you already said that 🤦

[kescobo]

One additional option might be to add in some logic so that if it's stalled for more than X seconds, it errors out. I have no sense of what a good default for X is, or if it should be user-configurable, but could open a separate issue for that if it's outside the scope of the PR you want to make. I'm fine with a simple addition of sleep if you want to spin that up

[David96]

Opened a pull request for the sleep addition. Adding a timer would also be helpful I think but I don't believe there's a good "default" for it - depending on how busy a cluster is the usual waiting times might differ by a lot.
Another thing that is kind of related is that this loop also doesn't stop if the job gets cancelled before it has actually been started which of course would also more or less be handled by a timeout.

[DrChainsaw]

Fwiw, the lsf manager lets the user supply an iterator with retry delays. The default is an exponential backoff with a max number of attempts, but I tend to just use Iterators.cyclic(5) to not have to deal with timeouts as the startup time at my place is extremely unpredictable.

ClusterManagers.jl/src/lsf.jl

Line 43 in 861b301

for retry_delay in retry_delays

[kescobo]

Alright, I'm gonna merge the PR for now, but looking forward to next one that superceeds it