Stuck in third send or recv when connecting two independent mpi applications with MPI_Comm_connect and MPI_Comm_accept with prte

[dariomnz]

Stuck in third send or recv when connecting two independent mpi applications with MPI_Comm_connect and MPI_Comm_accept with prte

[bosilca]

Your code works just fine on my setup. My main issue arise from the fact that prte DVM decide to only use the 'lo' interface which forces me to start all processes from the same host.

[dariomnz]

Can I get more feedback, such as how do I debug the app to find out why it's stuck?

[bosilca]

As a first step you should attach to the running processes with a debugger (gdb -p) and take a look at the stack trace to see where exactly the processes are blocked.

[dariomnz]

As I said before the program gets stuck in the third Send-Recv, you can see it in the gdb backtrace as you told me to debug.
gdb server:

gdb -p 593951
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 593951
[New LWP 593952]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fd8eed3546e in epoll_wait (epfd=3, events=0x564a1bcc7810, maxevents=32, timeout=0) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
30      ../sysdeps/unix/sysv/linux/epoll_wait.c: No such file or directory.
(gdb) bt
#0  0x00007fd8eed3546e in epoll_wait (epfd=3, events=0x564a1bcc7810, maxevents=32, timeout=0) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00007fd8ee988469 in ?? () from /lib/x86_64-linux-gnu/libevent_core-2.1.so.7
#2  0x00007fd8ee97e4a5 in event_base_loop () from /lib/x86_64-linux-gnu/libevent_core-2.1.so.7
#3  0x00007fd8eeb3bb53 in opal_progress_events () from /beegfs/home/dariomnz/bin/ompi5/lib/libopen-pal.so.80
#4  0x00007fd8eeb3bc25 in opal_progress () from /beegfs/home/dariomnz/bin/ompi5/lib/libopen-pal.so.80
#5  0x00007fd8ef06ff60 in mca_pml_ob1_recv () from /beegfs/home/dariomnz/bin/ompi5/lib/libmpi.so.40
#6  0x00007fd8eeeea9ef in PMPI_Recv () from /beegfs/home/dariomnz/bin/ompi5/lib/libmpi.so.40
#7  0x000056433f13e5fd in main (argc=1, argv=0x7ffefe82ca88) at server.c:45
(gdb) 

Client:

gdb -p 593961
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 593961
[New LWP 593962]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007faae290f46e in epoll_wait (epfd=3, events=0x555fdbb5b810, maxevents=32, timeout=0) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
30      ../sysdeps/unix/sysv/linux/epoll_wait.c: No such file or directory.
(gdb) bt
#0  0x00007faae290f46e in epoll_wait (epfd=3, events=0x555fdbb5b810, maxevents=32, timeout=0) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00007faae2562469 in ?? () from /lib/x86_64-linux-gnu/libevent_core-2.1.so.7
#2  0x00007faae25584a5 in event_base_loop () from /lib/x86_64-linux-gnu/libevent_core-2.1.so.7
#3  0x00007faae2715b53 in opal_progress_events () from /beegfs/home/dariomnz/bin/ompi5/lib/libopen-pal.so.80
#4  0x00007faae2715c25 in opal_progress () from /beegfs/home/dariomnz/bin/ompi5/lib/libopen-pal.so.80
#5  0x00007faae2c4e313 in mca_pml_ob1_send () from /beegfs/home/dariomnz/bin/ompi5/lib/libmpi.so.40
#6  0x00007faae2aca1f3 in PMPI_Send () from /beegfs/home/dariomnz/bin/ompi5/lib/libmpi.so.40
#7  0x0000562e06ef4500 in main (argc=2, argv=0x7ffdf534e6a8) at client.c:35
(gdb) 

Can I have more information on how to continue debugging to make it work?

[dariomnz]

After much trial and error the problem was that I needed to compile openmpi with slurm (--with-slurm=/opt/slurm), otherwise this behavior of getting stuck on the third send would happen.