Feature Request: reset slurmctld for local/remote backends

[getzlab]

Feature Request: reset slurmctld for local/remote backends

[agraubert]

What if canine were to check for running jobs before performing a reset? If there are active jobs running, it could assume that everything is OK, but only perform the reset if the cluster is idle

[agraubert]

Added new backend option:

backend:
    slurm_conf_path: /path/to/slurm.conf

This option is accepted by all slurm backends, and if provided, will force a restart of slurmctld after the pipeline is staged, but before any jobs are dispatched

[hurrialice]

This could potentially be harmful - e.g. When you submit new canine jobs you do not want to disrupt the running ones. But yes, maybe give a flag when outputs still grow and let users do the hard reset themselves?

[julianhess]

If it's necessary to hard reset the cluster (e.g. due to corruption of the Slurm bitmap), the chances are very high that the currently running jobs will not properly complete anyway.

[julianhess]

P.S. just realized the proper way to run this is

slurmctld -c -f /path/to/slurm.conf && slurmctld reconfigure