dbcsr_example_3.cpp hangs on dbcsr::finalize_lib(group)

[cp2k]

dbcsr_example_3.cpp hangs on dbcsr::finalize_lib(group)

[alazzaro]

How many threads and MPI ranks?
Do you have a stack trace for that?

[alazzaro]

@dithillobothrium Could you look at this problem?

[alazzaro]

BTW, which compiler are you using?

[davydden]

How many threads and MPI ranks?

compiled without OpenMP, single MPI rank (purely serial).

Do you have a stack trace for that?

after dbcsr.h:17 it's

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00000001002317c7 libgfortran.5.dylib`_gfortrani_internal_unpack_8 + 295
    frame #1: 0x000000011d4c267f libdbcsr.dylib`__dbcsr_mpiwrap_MOD_mp_sum_lv at dbcsr_mpiwrap.F:9277
    frame #2: 0x000000011d4714de libdbcsr.dylib`__dbcsr_mm_sched_MOD_dbcsr_mm_sched_lib_finalize at dbcsr_mm_sched.F:632
    frame #3: 0x000000011d4698fc libdbcsr.dylib`__dbcsr_mm_MOD_dbcsr_multiply_lib_finalize at dbcsr_mm.F:186
    frame #4: 0x000000011d40fa0d libdbcsr.dylib`__dbcsr_lib_MOD_dbcsr_finalize_lib at dbcsr_lib.F:129

with frame #0: being:

(lldb) f 0
frame #0: 0x00000001002317c7 libgfortran.5.dylib`_gfortrani_internal_unpack_8 + 295
libgfortran.5.dylib`_gfortrani_internal_unpack_8:
->  0x1002317c7 <+295>: cmpq   %r13, %rax
    0x1002317ca <+298>: jne    0x100231858               ; <+440>
    0x1002317d0 <+304>: movq   $0x0, -0x68(%rsp)
    0x1002317d9 <+313>: cmpq   $0x1, %r10

which compiler are you using?

[email protected]

[dithillobothrium]

@dithillobothrium Could you look at this problem?

I can look

[alazzaro]

OK, it is hanging on an allreduce for a single rank, which is pretty weird...
I had a similar problem time ago with OpenMPI 3.1.2 on a linux machine and it turned our that it is something broken in OpenMPI 3.1.x. Indeed 3.0.2 works. Could you confirm that?

[dithillobothrium]

in my case it works with gcc 7.3.1 and openmpi 3.1.2 on a linux machine

[davydden]

Indeed 3.0.2 works. Could you confirm that?

unfortunately not, 3.0.2 gives the same issue.

[dev-zero]

@alazzaro just installed gfortran and openmpi on our tcopt1 (macos high sierra) via brew and building DBCSR now (using cmake -DUSE_OPENMP=OFF, just in case you want to try by yourself)

[dev-zero]

I can't reproduce it on macOS High Sierra with OpenMPI-3.1.3, clang-902.0.39.2 and gfortran-8.2.0

@davydden you can still reproduce it with openmpi-3.1.3?

[davydden]

you can still reproduce it with openmpi-3.1.3?

no, i can not. 3.1.3 works. Looks like it was indeed some issue with openmpi