jeffhammond/mukautuva

An MPI ABI compatibility layer

mukautuva's Issues

Launcher

We can create a MUK launcher that sets MPI_LIB and the appropriate implementation launcher together, based on MPI_PATH or some other combination of env var or launcher args.

Bonus points to add gdm, lldb, and valgrind support so I don't have to do this manually.

disconnect reconnect

Do I care? No.

mpirun.openmpi --oversubscribe -n 3    ./disconnect_reconnect3
. No Errors
[nuclear][[13406,1],0][../../../../../../opal/mca/btl/tcp/btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(17) failed: Connection reset by peer (104)
Unexpected output in disconnect_reconnect3: [nuclear][[13406,1],0][../../../../../../opal/mca/btl/tcp/btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(17) failed: Connection reset by peer (104)

MPICH test failures: datatypes

Works:

• dataalign2
• pairtype_size_extent
• struct_pack_mpi_bottom
• longdouble
• tmatchsize
• getpartelm
• struct_verydeep
• contig_zero_count
• struct_no_real_types
• typename
• concurrent_irecv
• zeroparms
• struct_pack
• slice_pack
• struct_zero_count
• get_elements_pairtype
• subarray_pack
• typename_oldapi
• simple_pack_external
• hindexed_zeros
• darray_pack
• vecblklen
• unusual_noncontigs
• large_type_sendrec
• large_type
• indexed_misc_oldapi
• lbub
• blockindexed_zero_count
• sendrecvt4
• sizedtypes
• simple_size_extent
• sendrecvt2
• type_large
• subarray
• typelb
• triangular_pack
• contigstruct
• unpack
• blockindexed_misc
• struct_unpack_mpi_bottom
• lbub_oldapi
• struct_derived_zeros
• typefree
• cxx_types
• slice_pack_external
• tfree
• contents
• struct_empty_el
• pairtype_pack
• hindexed_block
• lots_of_types
• dataalign
• transpose_pack_oldapi
• large_count
• simple_pack_external2
• zeroblks
• pack_external
• get_elements
• indexed_misc
• large_vec
• darray_cyclic
• transpose_pack
• tresized
• struct_ezhov
• simple_pack
• hvecblklen
• gaddress
• tresized2
• structpack2
• simple_size_extent_oldapi
• localpack
• typecommit
• simple_resized
• hindexed_block_contents
• simple_commit

MPICH test failure: idup_with_info

Starting ./idup_with_info
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Open-MPI
Open-MPI
Open-MPI
[nuclear:426822] *** Process received signal ***
[nuclear:426822] Signal: Segmentation fault (11)
[nuclear:426822] Signal code: Address not mapped (1)
[nuclear:426822] Failing at address: (nil)
[nuclear:426819] *** Process received signal ***
[nuclear:426819] Signal: Segmentation fault (11)
[nuclear:426819] Signal code: Address not mapped (1)
[nuclear:426819] Failing at address: (nil)
[nuclear:426820] *** Process received signal ***
[nuclear:426820] Signal: Segmentation fault (11)
[nuclear:426820] Signal code: Address not mapped (1)
[nuclear:426820] Failing at address: (nil)
[nuclear:426821] *** Process received signal ***
[nuclear:426821] Signal: Segmentation fault (11)
[nuclear:426821] Signal code: Address not mapped (1)
[nuclear:426821] Failing at address: (nil)
[nuclear:426820] [ 0] [nuclear:426821] [ 0] [nuclear:426822] [ 0] [nuclear:426819] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f8408242520]
[nuclear:426819] *** End of error message ***
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f0bf4842520]
[nuclear:426822] *** End of error message ***
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f9d2a042520]
[nuclear:426820] *** End of error message ***
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7efca0642520]
[nuclear:426821] *** End of error message ***

MPICH test failures: infogetstring

Not surprising.

Starting ./infogetstring
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:445106] *** Process received signal ***
[nuclear:445106] Signal: Segmentation fault (11)
[nuclear:445106] Signal code: Address not mapped (1)
[nuclear:445106] Failing at address: (nil)
[nuclear:445106] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f0607242520]
[nuclear:445106] *** End of error message ***

optimize array-of-request completions

cleanup_mapped_request does a check on .empty() for all the relevant maps every call. We can hoist this to optimize the common case.

    for (int i=0; i<count; i++) {
        if (!ignore) {
            MPI_Status_to_WRAP_Status(&impl_statuses[i], &array_of_statuses[i]);
        }
        if (impl_requests[i] == MPI_REQUEST_NULL) {
            cleanup_mapped_request(CONVERT_MPI_Request(array_of_requests[i]));
        }
        array_of_requests[i] = OUTPUT_MPI_Request(impl_requests[i]);
    }

need to support MPI_IN_PLACE in IALLTOALLW

This fails on the NULL.

    MPI_Ialltoallw(MPI_IN_PLACE, NULL, NULL, NULL, rbuf, rcounts, rdispls, types, comm, &req);
    MPI_Wait(&req, MPI_STATUS_IGNORE);

MPICH tests: isendrecv is busted

making isendrecv
Env includes
MPI_LIB=/opt/mpich/gcc/debug-ch4ucx/lib/libmpi.so mpirun.mpich -n 4    ./isendrecv
.Unexpected output in isendrecv: offset=0
Unexpected output in isendrecv: i,j=0,0
Unexpected output in isendrecv: i,j=1,1
Unexpected output in isendrecv: i,j=2,2
Unexpected output in isendrecv: i,j=3,3
Unexpected output in isendrecv: offset=0
Unexpected output in isendrecv: i,j=0,0
Unexpected output in isendrecv: offset=0
Unexpected output in isendrecv: i,j=0,0
Unexpected output in isendrecv: i,j=1,1
Unexpected output in isendrecv: i,j=1,1
Unexpected output in isendrecv: i,j=2,2
Unexpected output in isendrecv: i,j=3,3
Unexpected output in isendrecv: i,j=2,2
Unexpected output in isendrecv: i,j=3,3
Unexpected output in isendrecv: offset=1
Unexpected output in isendrecv: i,j=0,1
Unexpected output in isendrecv: i,j=1,2
Unexpected output in isendrecv: i,j=2,3
Unexpected output in isendrecv: i,j=3,0
Unexpected output in isendrecv: offset=1

comm spawn issues

1. It only works with Open-MPI as the default SO. MPI_LIB is ignored by spawned processes and thus things don't work.

2. This:

mpirun.openmpi --oversubscribe -n 1    ./spawninfo1
.--------------------------------------------------------------------------
Unexpected output in spawninfo1: --------------------------------------------------------------------------
mpirun.openmpi was unable to find the specified executable file, and therefore
Unexpected output in spawninfo1: mpirun.openmpi was unable to find the specified executable file, and therefore
did not launch the job.  This error was first reported for process
Unexpected output in spawninfo1: did not launch the job.  This error was first reported for process
rank 0; it may have occurred for other processes as well.
Unexpected output in spawninfo1: rank 0; it may have occurred for other processes as well.

3. This:

mpirun.openmpi --oversubscribe -n 2    ./spaiccreate
.[nuclear:4044987] *** An error occurred in MPI_Intercomm_create
Unexpected output in spaiccreate: [nuclear:4044987] *** An error occurred in MPI_Intercomm_create
[nuclear:4044987] *** reported by process [1230700545,0]
Unexpected output in spaiccreate: [nuclear:4044987] *** reported by process [1230700545,0]
[nuclear:4044987] *** on communicator MPI_COMM_WORLD
Unexpected output in spaiccreate: [nuclear:4044987] *** on communicator MPI_COMM_WORLD
[nuclear:4044987] *** MPI_ERR_COMM: invalid communicator
Unexpected output in spaiccreate: [nuclear:4044987] *** MPI_ERR_COMM: invalid communicator
[nuclear:4044987] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
Unexpected output in spaiccreate: [nuclear:4044987] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[nuclear:4044987] ***    and potentially your MPI job)
Unexpected output in spaiccreate: [nuclear:4044987] ***    and potentially your MPI job)
[nuclear:4044982] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal
Unexpected output in spaiccreate: [nuclear:4044982] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[nuclear:4044982] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
Unexpected output in spaiccreate: [nuclear:4044982] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
Program spaiccreate exited without No Errors

MPICH test failures: pt2pt/errinstatts

We need to check status.

Starting ./pt2pt/errinstatts
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Error class 6 (MPI_ERR_RANK: invalid rank)
Error returned from Irecv
Error class 6 (MPI_ERR_RANK: invalid rank)
Error returned from Irecv
Error class 6 (MPI_ERR_RANK: invalid rank)
Error returned from Recv
Error class 6 (MPI_ERR_RANK: invalid rank)
Error returned from Recv
Did not get ERR_IN_STATUS in Testsome (outcount = 0, should equal 2); class returned was 7
 Found 5 errors

address sanitizer is mad

This:

    void ** pmuk_mpi_message_no_proc = MUK_DLSYM(wrap_so_handle,"IMPL_MESSAGE_NO_PROC");
    muk_mpi_message_no_proc = *pmuk_mpi_message_no_proc;

...triggers this error:

=================================================================
==489331==ERROR: AddressSanitizer: global-buffer-overflow on address 0x7ffa935a8700 at pc 0x7ffa99acaf3d bp 0x7ffd31dd3d40 sp 0x7ffd31dd3d30
READ of size 8 at 0x7ffa935a8700 thread T0
    #0 0x7ffa99acaf3c in MUK_Alkaa /home/jhammond/mukautuva/libinit.c:507
    #1 0x7ffa99ad2384 in MPI_Init /home/jhammond/mukautuva/libinit.c:1443
    #2 0x5576e84cc61e in main /home/jhammond/mukautuva/testinit.c:29
    #3 0x7ffa98c29d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #4 0x7ffa98c29e3f in __libc_start_main_impl ../csu/libc-start.c:392
    #5 0x5576e84cc2a4 in _start (/home/jhammond/mukautuva/testinit.x+0x12a4)

0x7ffa935a8704 is located 0 bytes to the right of global variable 'IMPL_MESSAGE_NO_PROC' defined in 'impl-predefined.c:354:13' (0x7ffa935a8700) of size 4
SUMMARY: AddressSanitizer: global-buffer-overflow /home/jhammond/mukautuva/libinit.c:507 in MUK_Alkaa
Shadow bytes around the buggy address:
  0x0fffd26ad090: 00 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
  0x0fffd26ad0a0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
  0x0fffd26ad0b0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
  0x0fffd26ad0c0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
  0x0fffd26ad0d0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
=>0x0fffd26ad0e0:[04]f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
  0x0fffd26ad0f0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
  0x0fffd26ad100: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
  0x0fffd26ad110: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
  0x0fffd26ad120: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
  0x0fffd26ad130: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==489331==ABORTING

It goes away if i meke sure that IMPL_MESSAGE_NO_PROC is 8 bytes, e.g. like this:

// No Process Message Handle
#if 0
MPI_Message IMPL_MESSAGE_NO_PROC = MPI_MESSAGE_NO_PROC;
#else
typedef union MPI_Message_union
{
    MPI_Message handle;
    void *      dummy;
} MPI_Message_union;
MPI_Message_union IMPL_MESSAGE_NO_PROC = { .handle = MPI_MESSAGE_NO_PROC };
#endif

I can resolve this two ways:

1. implement the union thing immediately above, or a more advanced version of it.
2. move completely away from using pointers to handles for the predefined stuff.

2 is preferred but requires a massive rewrite.

OSU MPI_Compare_and_swap latency test fails

This is the only OSU test that doesn't work with Open-MPI right now.

# OSU MPI_Compare_and_swap latency Test v7.0
# Window creation: MPI_Win_allocate
# Synchronization: MPI_Win_flush
# Size          Latency (us)
[nuclear:83579] *** Process received signal ***
[nuclear:83579] Signal: Segmentation fault (11)
[nuclear:83579] Signal code: Address not mapped (1)
[nuclear:83579] Failing at address: 0x7f35b2bf8008
[nuclear:83579] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f7d01457520]
[nuclear:83579] [ 1] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(+0x6ee7)[0x7f7cfa6faee7]
[nuclear:83579] [ 2] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(mca_btl_vader_poll_handle_frag+0x95)[0x7f7cfa6fbae5]
[nuclear:83579] [ 3] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(+0x7f91)[0x7f7cfa6fbf91]
[nuclear:83579] [ 4] /lib/x86_64-linux-gnu/libopen-pal.so.40(opal_progress+0x34)[0x7f7d01195714]
[nuclear:83579] [ 5] /usr/lib/x86_64-linux-gnu/libmpi.so(ompi_request_default_wait+0x55)[0x7f7d0132c0e5]
[nuclear:83579] [ 6] /usr/lib/x86_64-linux-gnu/libmpi.so(ompi_coll_base_barrier_intra_recursivedoubling+0xe3)[0x7f7d01387d13]
[nuclear:83579] [ 7] /usr/lib/x86_64-linux-gnu/libmpi.so(MPI_Barrier+0xb2)[0x7f7d01340482]
[nuclear:83579] [ 8] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Barrier+0x14)[0x7f7cf9c178d4]
[nuclear:83579] [ 9] ././one-sided/osu_cas_latency(+0x3707)[0x558af31aa707]
[nuclear:83579] [10] ././one-sided/osu_cas_latency(+0x30da)[0x558af31aa0da]
[nuclear:83579] [11] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f7d0143ed90]
[nuclear:83579] [12] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f7d0143ee40]
[nuclear:83579] [13] ././one-sided/osu_cas_latency(+0x29c5)[0x558af31a99c5]
[nuclear:83579] *** End of error message ***

MPICH test failures: attributes

No surprise here.

Starting ./baseattr2
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Got invalid value -1 for HOST

Starting ./baseattrcomm
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
 Found 1 errors
Got invalid value -1 for IO

persistent collectives are broken

i don't care right now.

Fix debug stuff

Using MUK_Warning in the wrong place causes crazy stuff to happen. It is related to DL symbol resolution of PMPI_Comm_rank.

Fix code in debug.c to call WRAP or IMPL but never PMPI, since that can resolve multiple ways.

MPI_TYPE_GET_CONTENTS with nested user types is broken

From #11.

If we do e.g. vector of vectors, MPI_TYPE_GET_CONTENTS does not work.

generalized requests

Strategy:

Use extra_state argument to support trampoline.

Grequest_start malloc struct of user_state plus function pointers for WRAP callbacks.

Grequest callback trampolines get WRAP callbacks and user_state from extra_state and use them appropriately. Grequest query trampoline converts status forwards and backwards.

Grequest free and cancel trampolines need to free the extra_state wrapper. Set pointer to null to be safe.

MPI Functionality that is Always Available

The following means that one can't just initialize everything during MPI initialization. One needs load these on demand.

This requires modest changes to libinit.c but is not going to be implemented until more important things are working.

I implemented the first two correctly and the next two in a half-arsed way.

11.4.1 MPI Functionality that is Always Available

Some MPI functions may be invoked at any time, including prior to calling MPI_INIT or MPI_SESSION_INIT, and following MPI finalization, independent of whether the World Model, Sessions Model, or both are used. These functions can be called concurrently by multiple threads within an MPI Process. Table 11.1 lists the applicable MPI functions.

MPI_INITIALIZED
MPI_FINALIZED
MPI_GET_VERSION
MPI_GET_LIBRARY_VERSION
MPI_INFO_CREATE
MPI_INFO_CREATE_ENV
MPI_INFO_SET
MPI_INFO_DELETE
MPI_INFO_GET_STRING
MPI_INFO_GET_NKEYS
MPI_INFO_GET_NTHKEY
MPI_INFO_DUP
MPI_INFO_FREE
MPI_INFO_F2C
MPI_INFO_C2F
MPI_SESSION_CREATE_ERRHANDLER
MPI_SESSION_CALL_ERRHANDLER
MPI_ERRHANDLER_FREE
MPI_ERRHANDLER_F2C
MPI_ERRHANDLER_C2F
MPI_ERROR_STRING
MPI_ERROR_CLASS

functions that can be called before init and sessions

All of this is either unimplemented or XFAIL.

I need to reorganize the dlsym stuff to make this work.

`static_assert` called with only a single argument

static_assert with a single argument is not part of C++17, it requires a newer standard.

impl-predefined.c:19:58: warning: '_Static_assert' with no message is a C2x extension [-Wc2x-extensions]
_Static_assert( (int)MUK_UNDEFINED == (int)MPI_UNDEFINED );
                                                         ^
                                                         , ""

Build system

Sigh. I suppose we need one. Someone else can write it.

neighborhood alltoallw is busted

vectors of datatypes need to be size of degree on the graph not whole comm size

repro:

~/mukautuva/mpich-testsuite-4.1/coll$ make -j8 -k ; MPI_LIB=/usr/lib/x86_64-linux-gnu/libmpich.so /usr/bin/mpirun.mpich  -n 3 gdb -ex "set width 1000" -ex "thread apply all bt" -ex run -ex bt -ex "set confirm off" -ex quit --args ./neighb_alltoallw2

MPICH test failures: errors

Only the segfaults bother me. The other failures might just be Open-MPI not checking. Need to run again with MPICH underneath.

Starting ./topo/cartsmall
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:446987] *** Process received signal ***
[nuclear:446987] Signal: Segmentation fault (11)
[nuclear:446987] Signal code: Address not mapped (1)
[nuclear:446987] Failing at address: 0x10
[nuclear:446987] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f928e242520]
[nuclear:446987] [ 1] /usr/lib/x86_64-linux-gnu/libmpi.so(MPI_Cart_create+0x1e4)[0x7f928dd32fc4]
[nuclear:446987] [ 2] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Cart_create+0x66)[0x7f928775d984]
[nuclear:446987] [ 3] /home/jhammond/mukautuva/libmuk.so(MPI_Cart_create+0x4f)[0x7f928e4d2751]
[nuclear:446987] [ 4] ./topo/cartsmall(+0x2b92)[0x55cb440dcb92]
[nuclear:446987] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f928e229d90]
[nuclear:446987] [ 6] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f928e229e40]
[nuclear:446987] [ 7] ./topo/cartsmall(+0x2c25)[0x55cb440dcc25]
[nuclear:446987] *** End of error message ***

Starting ./coll/nb_rerr
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Did not detect invalid type/op pair (byte,max) in Allreduce
 Found 1 errors

Starting ./coll/ireducelength
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Did not detect mismatched length (short) on process 0
 Found 1 errors

Starting ./coll/nb_noalias
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Did not detect aliased arguments in MPI_Allreduce
Did not detect aliased arguments in MPI_Gather
Did not detect aliased arguments in MPI_Scatter
 Found 3 errors

Starting ./coll/rerr
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Did not detect invalid type/op pair (byte,max) in Allreduce
 Found 1 errors

Starting ./coll/noalias
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Did not detect aliased arguments in MPI_Allreduce
Did not detect aliased arguments in MPI_Gather
Did not detect aliased arguments in MPI_Scatter
 Found 3 errors

Starting ./coll/reducelength
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Did not detect mismatched length (short) on process 0
 Found 1 errors

Starting ./coll/noalias2
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
 Found 13 errors

Starting ./coll/reduce_local
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:449224] *** Process received signal ***
[nuclear:449224] Signal: Segmentation fault (11)
[nuclear:449224] Signal code: Address not mapped (1)
[nuclear:449224] Failing at address: 0x1
[nuclear:449224] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7effadc42520]
[nuclear:449224] [ 1] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_op_avx.so(+0x1a01f)[0x7effac11a01f]
[nuclear:449224] [ 2] /usr/lib/x86_64-linux-gnu/libmpi.so(mca_coll_base_reduce_local+0x12b)[0x7effad7782db]
[nuclear:449224] [ 3] /usr/lib/x86_64-linux-gnu/libmpi.so(MPI_Reduce_local+0x171)[0x7effad74cc91]
[nuclear:449224] [ 4] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Reduce_local+0x6d)[0x7effa71c2a32]
[nuclear:449224] [ 5] /home/jhammond/mukautuva/libmuk.so(MPI_Reduce_local+0x45)[0x7effadf343d0]
[nuclear:449224] [ 6] ./coll/reduce_local(+0x2c0e)[0x55d2b0585c0e]
[nuclear:449224] [ 7] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7effadc29d90]
[nuclear:449224] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7effadc29e40]
[nuclear:449224] [ 9] ./coll/reduce_local(+0x2de5)[0x55d2b0585de5]
[nuclear:449224] *** End of error message ***

Starting ./coll/noalias3
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
 Found 13 errors

Starting ./datatype/type_lb_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:449774] *** Process received signal ***
[nuclear:449774] Signal: Segmentation fault (11)
[nuclear:449774] Signal code: Address not mapped (1)
[nuclear:449774] Failing at address: (nil)
[nuclear:449774] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f2caf442520]
[nuclear:449774] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Type_lb+0x38)[0x7f2cac14583e]
[nuclear:449774] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Type_lb+0x2e)[0x7f2caf6bfb80]
[nuclear:449774] [ 3] ./datatype/type_lb_nullarg(+0x3bc5)[0x563a771a7bc5]
[nuclear:449774] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f2caf429d90]
[nuclear:449774] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f2caf429e40]
[nuclear:449774] [ 6] ./datatype/type_lb_nullarg(+0x3c35)[0x563a771a7c35]
[nuclear:449774] *** End of error message ***

Starting ./datatype/type_contiguous_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:449865] *** Process received signal ***
[nuclear:449865] Signal: Segmentation fault (11)
[nuclear:449865] Signal code: Address not mapped (1)
[nuclear:449865] Failing at address: (nil)
[nuclear:449865] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fe32d842520]
[nuclear:449865] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Type_contiguous+0x28)[0x7fe326a36b7e]
[nuclear:449865] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Type_contiguous+0x34)[0x7fe32dba6e01]
[nuclear:449865] [ 3] ./datatype/type_contiguous_nullarg(+0x2b71)[0x55cea9c23b71]
[nuclear:449865] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fe32d829d90]
[nuclear:449865] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fe32d829e40]
[nuclear:449865] [ 6] ./datatype/type_contiguous_nullarg(+0x2bd5)[0x55cea9c23bd5]
[nuclear:449865] *** End of error message ***

Starting ./datatype/type_extent_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:450039] *** Process received signal ***
[nuclear:450039] Signal: Segmentation fault (11)
[nuclear:450039] Signal code: Address not mapped (1)
[nuclear:450039] Failing at address: (nil)
[nuclear:450039] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fa317642520]
[nuclear:450039] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Type_extent+0x38)[0x7fa3143b3374]
[nuclear:450039] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Type_extent+0x2e)[0x7fa3179275d4]
[nuclear:450039] [ 3] ./datatype/type_extent_nullarg(+0x2b6c)[0x557ff2662b6c]
[nuclear:450039] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fa317629d90]
[nuclear:450039] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fa317629e40]
[nuclear:450039] [ 6] ./datatype/type_extent_nullarg(+0x2bd5)[0x557ff2662bd5]
[nuclear:450039] *** End of error message ***
Starting ./datatype/type_get_extent_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:450207] *** Process received signal ***
[nuclear:450207] Signal: Segmentation fault (11)
[nuclear:450207] Signal code: Address not mapped (1)
[nuclear:450207] Failing at address: (nil)
[nuclear:450207] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f9851442520]
[nuclear:450207] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Type_get_extent+0x59)[0x7f984a1f1581]
[nuclear:450207] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Type_get_extent+0x37)[0x7f985178185a]
[nuclear:450207] [ 3] ./datatype/type_get_extent_nullarg(+0x3bca)[0x561169c70bca]
[nuclear:450207] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f9851429d90]
[nuclear:450207] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f9851429e40]
[nuclear:450207] [ 6] ./datatype/type_get_extent_nullarg(+0x3c35)[0x561169c70c35]
[nuclear:450207] *** End of error message ***
Starting ./datatype/type_get_true_extent_x_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
 No Errors
Starting ./datatype/type_vector_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:450389] *** Process received signal ***
[nuclear:450389] Signal: Segmentation fault (11)
[nuclear:450389] Signal code: Address not mapped (1)
[nuclear:450389] Failing at address: (nil)
[nuclear:450389] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f3f16e42520]
[nuclear:450389] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Type_vector+0x2e)[0x7f3f0ff1dac2]
[nuclear:450389] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Type_vector+0x40)[0x7f3f1708fd8e]
[nuclear:450389] [ 3] ./datatype/type_vector_nullarg(+0x2b7c)[0x5654f0f94b7c]
[nuclear:450389] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f3f16e29d90]
[nuclear:450389] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f3f16e29e40]
[nuclear:450389] [ 6] ./datatype/type_vector_nullarg(+0x2be5)[0x5654f0f94be5]
[nuclear:450389] *** End of error message ***
Starting ./datatype/type_ub_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:450561] *** Process received signal ***
[nuclear:450561] Signal: Segmentation fault (11)
[nuclear:450561] Signal code: Address not mapped (1)
[nuclear:450561] Failing at address: (nil)
[nuclear:450561] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f97c1242520]
[nuclear:450561] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Type_ub+0x38)[0x7f97b9f08a74]
[nuclear:450561] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Type_ub+0x2e)[0x7f97c147cd4c]
[nuclear:450561] [ 3] ./datatype/type_ub_nullarg(+0x3bc5)[0x55780fac9bc5]
[nuclear:450561] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f97c1229d90]
[nuclear:450561] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f97c1229e40]
[nuclear:450561] [ 6] ./datatype/type_ub_nullarg(+0x3c35)[0x55780fac9c35]
[nuclear:450561] *** End of error message ***
Starting ./group/group_translate_ranks_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
 Found 1 errors
Starting ./group/group_rank_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
 No Errors
Starting ./group/group_intersection_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:450744] *** Process received signal ***
[nuclear:450744] Signal: Segmentation fault (11)
[nuclear:450744] Signal code: Address not mapped (1)
[nuclear:450744] Failing at address: (nil)
[nuclear:450744] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f6f31042520]
[nuclear:450744] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Group_intersection+0x29)[0x7f6f29cf7c81]
[nuclear:450744] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Group_intersection+0x37)[0x7f6f3126acde]
[nuclear:450744] [ 3] ./group/group_intersection_nullarg(+0x2bc1)[0x5644380f3bc1]
[nuclear:450744] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f6f31029d90]
[nuclear:450744] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f6f31029e40]
[nuclear:450744] [ 6] ./group/group_intersection_nullarg(+0x2c45)[0x5644380f3c45]
[nuclear:450744] *** End of error message ***
Starting ./group/gerr
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Did not detect invalid handle (comm) in group_incl
 Found 1 errors
Starting ./group/group_difference_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:451003] *** Process received signal ***
[nuclear:451003] Signal: Segmentation fault (11)
[nuclear:451003] Signal code: Address not mapped (1)
[nuclear:451003] Failing at address: (nil)
[nuclear:451003] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f81b6e42520]
[nuclear:451003] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Group_difference+0x29)[0x7f81b4061b17]
[nuclear:451003] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Group_difference+0x37)[0x7f81b71d6bbd]
[nuclear:451003] [ 3] ./group/group_difference_nullarg(+0x2be5)[0x55befe3d1be5]
[nuclear:451003] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f81b6e29d90]
[nuclear:451003] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f81b6e29e40]
[nuclear:451003] [ 6] ./group/group_difference_nullarg(+0x2c75)[0x55befe3d1c75]
[nuclear:451003] *** End of error message ***
Starting ./group/group_size_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
 No Errors
Starting ./group/group_union_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:451181] *** Process received signal ***
[nuclear:451181] Signal: Segmentation fault (11)
[nuclear:451181] Signal code: Address not mapped (1)
[nuclear:451181] Failing at address: (nil)
[nuclear:451181] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f1034e42520]
[nuclear:451181] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Group_union+0x29)[0x7f102e05ae84]
[nuclear:451181] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Group_union+0x37)[0x7f10351d3e36]
[nuclear:451181] [ 3] ./group/group_union_nullarg(+0x2be5)[0x55b9d18dbbe5]
[nuclear:451181] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f1034e29d90]
[nuclear:451181] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f1034e29e40]
[nuclear:451181] [ 6] ./group/group_union_nullarg(+0x2c75)[0x55b9d18dbc75]
[nuclear:451181] *** End of error message ***
Starting ./group/group_range_incl_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:451352] *** Process received signal ***
[nuclear:451352] Signal: Segmentation fault (11)
[nuclear:451352] Signal code: Address not mapped (1)
[nuclear:451352] Failing at address: (nil)
[nuclear:451352] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f328ce42520]
[nuclear:451352] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Group_range_incl+0x2c)[0x7f32860b8d4a]
[nuclear:451352] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Group_range_incl+0x3a)[0x7f328d22bd56]
[nuclear:451352] [ 3] ./group/group_range_incl_nullarg(+0x2bc1)[0x55abd92babc1]
[nuclear:451352] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f328ce29d90]
[nuclear:451352] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f328ce29e40]
[nuclear:451352] [ 6] ./group/group_range_incl_nullarg(+0x2c45)[0x55abd92bac45]
[nuclear:451352] *** End of error message ***
Starting ./group/group_range_excl_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:451444] *** Process received signal ***
[nuclear:451444] Signal: Segmentation fault (11)
[nuclear:451444] Signal code: Address not mapped (1)
[nuclear:451444] Failing at address: (nil)
[nuclear:451444] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f12fba42520]
[nuclear:451444] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Group_range_excl+0x2c)[0x7f12f8764ce7]
[nuclear:451444] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Group_range_excl+0x3a)[0x7f12fbce4d1a]
[nuclear:451444] [ 3] ./group/group_range_excl_nullarg(+0x2be1)[0x55a051341be1]
[nuclear:451444] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f12fba29d90]
[nuclear:451444] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f12fba29e40]
[nuclear:451444] [ 6] ./group/group_range_excl_nullarg(+0x2c65)[0x55a051341c65]
[nuclear:451444] *** End of error message ***
Starting ./comm/comm_size_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
 No Errors
Starting ./comm/too_many_icomms
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
 No Errors
Starting ./comm/subcomm_abort2
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Starting ./comm/too_many_comms3
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
 No Errors
Starting ./comm/intercomm_create_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:451890] *** Process received signal ***
[nuclear:451890] Signal: Segmentation fault (11)
[nuclear:451890] Signal code: Address not mapped (1)
[nuclear:451890] Failing at address: (nil)
[nuclear:451890] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f40d1042520]
[nuclear:451890] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Intercomm_create+0x36)[0x7f40c9d21323]
[nuclear:451890] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Intercomm_create+0x4d)[0x7f40d129b0c5]
[nuclear:451890] [ 3] ./comm/intercomm_create_nullarg(+0x2bc5)[0x561f08c12bc5]
[nuclear:451890] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f40d1029d90]
[nuclear:451890] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f40d1029e40]
[nuclear:451890] [ 6] ./comm/intercomm_create_nullarg(+0x2c55)[0x561f08c12c55]
[nuclear:451890] *** End of error message ***
Starting ./comm/cfree
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:451976] *** Process received signal ***
[nuclear:451976] Signal: Segmentation fault (11)
[nuclear:451976] Signal code: Address not mapped (1)
[nuclear:451976] Failing at address: 0x563d3712f2a2
[nuclear:451976] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fb46b042520]
[nuclear:451976] [ 1] /usr/lib/x86_64-linux-gnu/libmpi.so(MPI_Barrier+0x43)[0x7fb46af2d413]
[nuclear:451976] [ 2] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Barrier+0x26)[0x7fb46827253f]
[nuclear:451976] [ 3] /home/jhammond/mukautuva/libmuk.so(MPI_Barrier+0x23)[0x7fb46b3ec203]
[nuclear:451976] [ 4] ./comm/cfree(+0x2b91)[0x563854340b91]
[nuclear:451976] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fb46b029d90]
[nuclear:451976] [ 6] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fb46b029e40]
[nuclear:451976] [ 7] ./comm/cfree(+0x2c45)[0x563854340c45]
[nuclear:451976] *** End of error message ***
Starting ./comm/comm_create_group_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:452141] *** Process received signal ***
[nuclear:452141] Signal: Segmentation fault (11)
[nuclear:452141] Signal code: Address not mapped (1)
[nuclear:452141] Failing at address: (nil)
[nuclear:452141] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f69fbe42520]
[nuclear:452141] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Comm_create_group+0x2c)[0x7f69f8aeaef1]
[nuclear:452141] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Comm_create_group+0x3a)[0x7f69fc063b39]
[nuclear:452141] [ 3] ./comm/comm_create_group_nullarg(+0x2b7c)[0x556095338b7c]
[nuclear:452141] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f69fbe29d90]
[nuclear:452141] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f69fbe29e40]
[nuclear:452141] [ 6] ./comm/comm_create_group_nullarg(+0x2bf5)[0x556095338bf5]
[nuclear:452141] *** End of error message ***
Starting ./comm/comm_split_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:452228] *** Process received signal ***
[nuclear:452228] Signal: Segmentation fault (11)
[nuclear:452228] Signal code: Address not mapped (1)
[nuclear:452228] Failing at address: (nil)
[nuclear:452228] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fee5d442520]
[nuclear:452228] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Comm_split+0x2b)[0x7fee564e069f]
[nuclear:452228] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Comm_split+0x38)[0x7fee5d65c070]
[nuclear:452228] [ 3] ./comm/comm_split_nullarg(+0x2b77)[0x55619b7eab77]
[nuclear:452228] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fee5d429d90]
[nuclear:452228] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fee5d429e40]
[nuclear:452228] [ 6] ./comm/comm_split_nullarg(+0x2be5)[0x55619b7eabe5]
[nuclear:452228] *** End of error message ***
Starting ./comm/too_many_icomms2
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
 No Errors
Starting ./comm/comm_get_info_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:452413] *** Process received signal ***
[nuclear:452413] Signal: Segmentation fault (11)
[nuclear:452413] Signal code: Address not mapped (1)
[nuclear:452413] Failing at address: (nil)
[nuclear:452413] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7ff867c42520]
[nuclear:452413] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Comm_get_info+0x25)[0x7ff864e3a226]
[nuclear:452413] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Comm_get_info+0x2e)[0x7ff867fb0d72]
[nuclear:452413] [ 3] ./comm/comm_get_info_nullarg(+0x2b7e)[0x55b6bc84fb7e]
[nuclear:452413] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7ff867c29d90]
[nuclear:452413] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7ff867c29e40]
[nuclear:452413] [ 6] ./comm/comm_get_info_nullarg(+0x2bf5)[0x55b6bc84fbf5]
[nuclear:452413] *** End of error message ***
Starting ./comm/comm_create_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:452586] *** Process received signal ***
[nuclear:452586] Signal: Segmentation fault (11)
[nuclear:452586] Signal code: Address not mapped (1)
[nuclear:452586] Failing at address: (nil)
[nuclear:452586] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fa01ee42520]
[nuclear:452586] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Comm_create+0x29)[0x7fa017f3ada9]
[nuclear:452586] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Comm_create+0x37)[0x7fa01f0b8a60]
[nuclear:452586] [ 3] ./comm/comm_create_nullarg(+0x2b77)[0x561b26d61b77]
[nuclear:452586] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fa01ee29d90]
[nuclear:452586] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fa01ee29e40]
[nuclear:452586] [ 6] ./comm/comm_create_nullarg(+0x2bf5)[0x561b26d61bf5]
[nuclear:452586] *** End of error message ***
Starting ./comm/intercomm_abort
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Starting ./comm/comm_group_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:452769] *** Process received signal ***
[nuclear:452769] Signal: Segmentation fault (11)
[nuclear:452769] Signal code: Address not mapped (1)
[nuclear:452769] Failing at address: (nil)
[nuclear:452769] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fdca6842520]
[nuclear:452769] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Comm_group+0x25)[0x7fdc9f961327]
[nuclear:452769] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Comm_group+0x2e)[0x7fdca6adae12]
[nuclear:452769] [ 3] ./comm/comm_group_nullarg(+0x2b82)[0x5577838b3b82]
[nuclear:452769] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fdca6829d90]
[nuclear:452769] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fdca6829e40]
[nuclear:452769] [ 6] ./comm/comm_group_nullarg(+0x2c05)[0x5577838b3c05]
[nuclear:452769] *** End of error message ***
Starting ./comm/userdup
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Comm_create_keyval: errcode = 13, class = 13, msg = MPI_ERR_ARG: invalid argument of some other kind
Starting ./comm/subcomm_abort
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Starting ./comm/ccreate1
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Failed to create odd group: : errcode = 6, class = 6, msg = MPI_ERR_RANK: invalid rank
Starting ./comm/manysplit
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
 No Errors
Starting ./comm/too_many_comms2
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
double free or corruption (out)
[nuclear:453292] *** Process received signal ***
[nuclear:453292] Signal: Aborted (6)
[nuclear:453292] Signal code:  (-6)
[nuclear:453292] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f9f9ee42520]
[nuclear:453292] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f9f9ee96a7c]
[nuclear:453292] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f9f9ee42476]
[nuclear:453292] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f9f9ee287f3]
[nuclear:453292] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x896f6)[0x7f9f9ee896f6]
[nuclear:453292] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0xa0d7c)[0x7f9f9eea0d7c]
[nuclear:453292] [ 6] /lib/x86_64-linux-gnu/libc.so.6(+0xa2ef0)[0x7f9f9eea2ef0]
[nuclear:453292] [ 7] /lib/x86_64-linux-gnu/libc.so.6(free+0x73)[0x7f9f9eea54d3]
[nuclear:453292] [ 8] /home/jhammond/mukautuva/ompi-wrap.so(+0x1c37d)[0x7f9f97fe537d]
[nuclear:453292] [ 9] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Comm_free+0x35)[0x7f9f97fe7107]
[nuclear:453292] [10] /home/jhammond/mukautuva/libmuk.so(MPI_Comm_free+0x23)[0x7f9f9f18bc9f]
[nuclear:453292] [11] ./comm/too_many_comms2(+0x2c50)[0x55806f2aec50]
[nuclear:453292] [12] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f9f9ee29d90]
[nuclear:453292] [13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f9f9ee29e40]
[nuclear:453292] [14] ./comm/too_many_comms2(+0x2ce5)[0x55806f2aece5]
[nuclear:453292] *** End of error message ***
Starting ./comm/comm_split_type_nullarg
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
[nuclear:453382] *** Process received signal ***
[nuclear:453382] Signal: Segmentation fault (11)
[nuclear:453382] Signal code: Address not mapped (1)
[nuclear:453382] Failing at address: (nil)
[nuclear:453382] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fd020242520]
[nuclear:453382] [ 1] /home/jhammond/mukautuva/ompi-wrap.so(WRAP_Comm_split_type+0x43)[0x7fd01cf2e724]
[nuclear:453382] [ 2] /home/jhammond/mukautuva/libmuk.so(MPI_Comm_split_type+0x43)[0x7fd0204a60c7]
[nuclear:453382] [ 3] ./comm/comm_split_type_nullarg(+0x2be0)[0x5592b8401be0]
[nuclear:453382] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fd020229d90]
[nuclear:453382] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fd020229e40]
[nuclear:453382] [ 6] ./comm/comm_split_type_nullarg(+0x2c75)[0x5592b8401c75]
[nuclear:453382] *** End of error message ***

MPICH RMA test failures

wincall - errhandler related

fkeyvalwin - keyval with non-default fns

MPI_Cart_create does not work

~/mukautuva/mpich-testsuite-4.1/coll$ MPI_LIB=/usr/lib/x86_64-linux-gnu/libmpich.so mpirun.mpich -n 2 ./ring_neighbor_alltoall
soname = /usr/lib/x86_64-linux-gnu/libmpich.so
soname = /usr/lib/x86_64-linux-gnu/libmpich.so
MPICH
MPICH
Abort(670642949) on node 0 (rank 0 in comm 0): Fatal error in internal_Cart_create: Invalid communicator, error stack:
internal_Cart_create(102): MPI_Cart_create(comm=0x53026058, ndims=1, dims=0x7ffcfb7c5e70, periods=0x7ffcfb7c5e74, reorder=1, comm_cart=0x556253f6cb50) failed
internal_Cart_create(46).: Invalid communicator
Abort(871969541) on node 1 (rank 1 in comm 0): Fatal error in internal_Cart_create: Invalid communicator, error stack:
internal_Cart_create(102): MPI_Cart_create(comm=0xa260e058, ndims=1, dims=0x7ffc278484a0, periods=0x7ffc278484a4, reorder=1, comm_cart=0x55d7a2a25b50) failed
internal_Cart_create(46).: Invalid communicator

OSU MPI Multi-process Latency Test and RDMAV_FORK_SAFE

I don't think this is my problem but I am recording it for posterity.

soname = /usr/lib/x86_64-linux-gnu/libmpich.so
soname = /usr/lib/x86_64-linux-gnu/libmpich.so
MPICH
MPICH
# Number of forked processes in sender: 2
# Number of forked processes in receiver: 2
# OSU MPI Multi-process Latency Test v7.0
# Size          Latency (us)
A process has executed an operation involving a call
to the fork() system call to create a child process.

As a result, the libfabric EFA provider is operating in
a condition that could result in memory corruption or
other system errors.

For the libfabric EFA provider to work safely when fork()
is called, you will need to set the following environment
variable:
          RDMAV_FORK_SAFE

However, setting this environment variable can result in
signficant performance impact to your application due to
increased cost of memory registration.

You may want to check with your application vendor to see
if an application-level alternative (of not using fork)
exists.

Your job will now abort.

MPICH test failures: ic1

Starting ./ic1
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
soname = /usr/lib/x86_64-linux-gnu/libmpi.so
Open-MPI
Open-MPI
[nuclear:442794] *** Process received signal ***
[nuclear:442794] Signal: Segmentation fault (11)
[nuclear:442794] Signal code: Address not mapped (1)
[nuclear:442794] Failing at address: (nil)
[nuclear:442794] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f9382842520]
[nuclear:442795] *** Process received signal ***
[nuclear:442795] Signal: Segmentation fault (11)
[nuclear:442795] Signal code: Address not mapped (1)
[nuclear:442795] Failing at address: (nil)
[nuclear:442795] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fdaaf242520]
[nuclear:442795] *** End of error message ***
[nuclear:442794] *** End of error message ***