Support for Slurm?

[netflix]

Support for Slurm?

[romain-intel]

@dgrahn : It would be able to work but there would definitely need to be some legwork to make it work :). If there is interest in working on a Slurm plugin (the first step to getting this to be compatible with Slurm), we can definitely help but probably would need some support from the community to test and what not.

[savingoyal]

@dgrahn We currently don't integrate with Slurm, but it's an interesting idea. @romain-intel Do you have any suggestions for Slurm alternatives for HPC workloads?

[romain-intel]

It's been a while and I had worked with Torque at the time but Slurm is definitely something we can consider. I suppose if you are mentioning Slurm, you intend on setting up multiple DSS-8440 in a cluster and would like to evaluate if Metaflow can potentially help with running large workloads on such a cluster. It could work but I am not sure it is quite ready for prime-time. You would still need to setup quite a few things to make sure that it all works properly (slurm, isolation of the accelerators if you wanted to share them among multiple flows, etc). Metaflow would also not necessarily benefit from some of the benefits of Slurm (request for multiple nodes at the same time for example since Metaflow "communicates" through a central location (S3 in the case of the AWS integration but you could imagine a shared file-system in a cluster-like setup).

On a single DSS-8440, you could use Metaflow without Slurm as well and make use of the multiple accelerators that way (since Metaflow would launch multiple processes on the same machine).

I am not sure of your exact use case but happy to discuss a little bit more. It's been a while since I worked in HPC but can hopefully still have a somewhat coherent conversation :).

[dgrahn]

@romain-intel It'll be one DSS-8440 and few lower-powered GPU machines that were previously available. It sounds like metaflow might not be the right technology for that use case at this point in time.

[Extremys]

Any progress? :)