Open MPI fails with 480 processes on a single node

[jstrodtb]

Open MPI fails with 480 processes on a single node

[devreal]

Sounds like the file limits on that machine are too low. Try running ulimit -n 2048 to increase that limit.

See https://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux for details.

[github-actions]

It looks like this issue is expecting a response, but hasn't gotten one yet. If there are no responses in the next 2 weeks, we'll assume that the issue has been abandoned and will close it.

[jstrodtb]

@devreal the upper limit on files is 65536. Upon further testing, the failure happens at around np = 250. 65536 = 256^2, so that tracks (obviously, the system has other file handles open).

Is it possible that Open MPI is creating a direct connection between each process that lives on the same node? That would explain this np^2 behavior.

[ggouaillardet]

That can happen if communications use TCP, but that should not be the case by default.
Try

mpirun --mca pml ob1 --mca btl self,vader -np 768 ...

to force the shared memory component.

mpirun --mca pml_base_verbose 100 --mca btl_base_verbose 100 -np 768 ...

should tell you what is going on by default.
You will get more info if you configure Open MPI with --enable-debug