COSMA crash on Perlmutter when dealing with complex values

[yaoyi92]

COSMA crash on Perlmutter when dealing with complex values

[rasolca]

It is not a cmake problem. It is a problem with newer versions of Cray mpich.
MPI_CXX_FLOAT_COMPLEX, MPI_CXX_DOUBLE_COMPLEX and MPI_CXX_BOOL are not set (the MPI standard requires them even if the C++ bindings are not provided).
We opened a ticket about it some time ago but still no solution from HPE side.

[kabicm]

Thanks Yi for a detailed error report.

Can you try putting MPI_C_FLOAT_COMPLEX instead of MPI_CXX_FLOAT_COMPLEX, and similarly for the double type at these lines:

COSMA/src/cosma/mpi_mapper.hpp

Line 30 in 783803e

return MPI_CXX_FLOAT_COMPLEX;

COSMA/src/cosma/mpi_mapper.hpp

Line 35 in 783803e

return MPI_CXX_DOUBLE_COMPLEX;

[yaoyi92]

Yes, I can confirm the modification solves the problem. Thank you very much for the quick reply.

[kabicm]

Great to hear that! Although I am still confused why this is a problem. @rasolca do you know why MPI_C works and MPI_CXX doesn't?

@yaoyi92 keep in mind that you can also use COSMA with gpu-aware MPI or with NCCL backends, as described in the README, that should be much more performant! This should be the biggest change in this version.

[yaoyi92]

Thanks! I will check them out.

[kabicm]

@yaoyi92 if it's not a problem for you, can you also try leaving the MPI_CXX prefixes there, but modifying the cmake:

COSMA/CMakeLists.txt

Line 124 in 783803e

find_package(MPI REQUIRED)

to find_package(MPI COMPONENTS C CXX REQUIRED).

Maybe this was the problem?

[kabicm]

@rasolca do you propose then to put MPI_C_ in the code as a temporary solution?

[rasolca]

In general not, but it is needed for Cray-EX systems.

[kabicm]

Alright! As a temporary solution we modified this in commit 5e71fac until cray-mpich fixes it.