Error crash at the end of the job execution

[alazzaro]

Error crash at the end of the job execution

[alazzaro]

Confirmed, problem fixed.

[kabicm]

I am still struggling to reproduce this.

Can you please check if the following resolves the issue? (I want to see if it's related to synchronization)

export HIP_LAUNCH_BLOCKING=1 

When using gpu-aware MPI, is it necessary to call device_synchronize after MPI collective to ensure it has finished the execution and that the buffers can safely be used?

[kabicm]

Concretely, should this line be uncommented:

COSMA/src/cosma/gpu/gpu_aware_mpi_utils.cpp

Line 234 in 783803e

// gpu::runtime_api::device_synchronize();

[alazzaro]

I am still struggling to reproduce this.

Can you please check if the following resolves the issue? (I want to see if it's related to synchronization)

export HIP_LAUNCH_BLOCKING=1 

Tried it and it doesn't solve the problem. I must say, this is expected (see my comment below).

When using gpu-aware MPI, is it necessary to call device_synchronize after MPI collective to ensure it has finished the execution and that the buffers can safely be used?

No, it is implied in the blocking MPI call.

[alazzaro]

So, we did a bit of investigation on this problem.
Apparently, this is related to the order of destruction of the GPU arrays used in the memory pool versus the finalization of the GPU MPI libraries dependencies within the Cray-MPICH. COSMA does that at the very end, likely after MPI GPU is finalized... More investigation is needed from the MPI side.

[kabicm]

Thanks a lot @alazzaro for the feedback! Hope we can fix it soon! :)

[kabicm]

@alazzaro Hey Alfio, do we have some more insights about this? Could it still be that this is a COSMA bug?