small array of derived data type(in Fortran) can be sent by MPI_Isend and MPI_Irecv but it ran into errors when I augment the array

[Bellahra]

small array of derived data type(in Fortran) can be sent by MPI_Isend and MPI_Irecv but it ran into errors when I augment the array

[ggouaillardet]

Even if the issues reported by @bosilca are addressed, I do not think this can work:
since subarrays are passed to MPI_Isend() and MPI_Irecv(), temporary flattened arrays are allocated by the Fortran runtime and deallocated when these subroutines return, which typically occurs before the data is sent or received, and hence undefined behavior which can be a crash.

Bottom line, subarrays should not be used with non blocking communications for now.

Note the MPI standard defines the MPI_SUBARRAYS_SUPPORTED and MPI_ASYNC_PROTECTS_NONBLOCKING "macros" and they are currently both .false. under Open MPI.

[bosilca]

There are some major issues with this code, let me highlight two:

1. you posted your intent to communicate (MPI_Isend and MPI_Irecv) but you never checked if the communications completed (MPI_Wait* or MPI_Test*). Until they are completed you are not supposed to use (for the receiver) or alter (for the sender) the buffers used in nonblocking communications.
2. you are sending and receiving from the exact same buffer. What exactly do you expect to find inside, the original or the new values ? If you really want to replace the data you should use MPI_Sendrecv_replace.

Additional suggestions for improving this code:

1. Open MPI does not have support for Fortran array descriptions (aka. CFI_desc_t). This means that the Fortran compiler will have to flatten each subarray you are sending as a packed array. Your datatype will still work, but you need to keep in mind the performance implication of this additional internal data management
2. I would suggest you change your communication pattern to be MPI_Irecv followed by MPI_Isend, to make sure all communications are expected on the receiver side.

[Bellahra]

@bosilca Thank you very much for your kind and helpful reply. The original code works well after adding the MPI_Wait operations. I have a question about the difference between the combination of MPI_Isend & MPI_Irecv and MPI_Sendrecv_replace. In the above code, since different ranks own different values of one variable, the replacement operation is done by using the same buffer and then overwriting it with the MPI_Irecv operation. I am not clear about if there is any risk of doing so and the difference between this and using MPI_Sendrecv_replace. Looking forward to your reply.

[Bellahra]

@ggouaillardet Thank you for your suggestion. It really helped me understand the issue better.