[DISCUSSION] Multi-node MPI runs with Singularity or Charliecloud containers

[geoschem]

[DISCUSSION] Multi-node MPI runs with Singularity or Charliecloud containers

[JiaweiZhuang]

A new announcement today from the Singularity team:

• A New Open Source Project Integrates Singularity and Slurm via Kubernetes https://www.sylabs.io/2019/05/introducing-hpc-affinities-to-the-enterprise-a-new-open-source-project-integrates-singularity-and-slurm-via-kubernetes/

Maybe this can be used with Amazon EKS.

[JiaweiZhuang]

Interestingly, Alibaba cloud already supports RDMA-enabled containers:

• Using RDMA on Container Service for Kubernetes https://www.alibabacloud.com/blog/using-rdma-on-container-service-for-kubernetes_594462

They got a bandwidth of ~5500 MB/s from ib_read_bw. The network latency is not shown in the post, but I guess it should be ~1 us, close to an InfiniBand cluster. Not sure what's their container runtime, but probably not Docker.

Welcome anyone to test it out. I probably don't have time to test it on my own...

[JiaweiZhuang]

A very well-written paper just released today:

• Evaluation of Docker Containers for Scientific Workloads in the Cloud: https://arxiv.org/abs/1905.08415

They were able to use Docker over InfiniBand RDMA with very low overhead!

[JiaweiZhuang]

A new post today: Benchmarking MPI Applications in Singularity Containers on Traditional HPC and Cloud Infrastructures
Covers:

• Overview of recent HPC + cloud trend (as of 2019/05)
• OSU bandwidth/latency benchmark on local HPC and Azure
• MPICH vs OpenMPI compatibility issues for Singularity

[JiaweiZhuang]

Just notice a new post today on using Singularity + MPI on Azure: Accelerating HPC Containers on Azure

[lizziel]

@JiaweiZhuang Thanks for sharing. Has anyone attempted running GCHP on Azure? Regardless, I would think the equivalent advances for multi-node capability on other cloud platforms can't be far behind. I'm hopeful.

From the blog:

The latest release also includes a new MPI interface for multi-node job execution without having
to deal with potential configuration complications between Batch multi-instance tasks, the
selected MPI runtime, and the VM instance size selected. Batch Shipyard now provides an
easy-to-use schema for executing your MPI jobs with support for popular MPI frameworks such
as Open MPI, MPICH, MVAPICH, and Intel MPI. This new interface works seamlessly between
both Singularity and Docker containers, which when combined with Azure's HB/HC instances
with 100Gbit/s EDR InfiniBand, provides unparalleled performance for your distributed HPC
container applications in the cloud.

[JiaweiZhuang]

Singularity 3.3 adds a new doc on Singularity and MPI applications. The situation is similar to 2.x that you need to have a compatible MPI installation on the host:

The drawbacks are:
The MPI in the container must be compatible with the version of MPI available on the host.
The configuration of the MPI implementation in the container must be configured for optimal use of the hardware if performance is critical.

[JiaweiZhuang]

@lizziel There are definitely examples with multi-node Docker runs (e.g. AWS Batch Multi-node Parallel Jobs). They are just less explored, with limited documentations & benchmark results.

[lizziel]

@WilliamDowns and @LiamBindle of GCST are working on multi-node MPI runs with containers. I am closing out this discussion for legacy GCHP. Future discussions about multi-node MPI runs using containers may be created at new GCHP repository GCHPctm.