sandia-openshmem/sos

Sandia OpenSHMEM is an implementation of the OpenSHMEM specification over multiple Networking APIs, including Portals 4, the Open Fabric Interface (OFI), and UCX. Please click on the Wiki tab for help with building and using SOS.

Sandia OpenSHMEM

About

Sandia OpenSHMEM is an implementation of the OpenSHMEM specification over Portals 4.0, the Open Fabrics Interface (OFI), and XPMEM.

Please refer to the tests-sos repository to download only the unit tests and the performance test suite that are included with Sandia OpenSHMEM.

This README file is formatted with Markdown. The human-readable README file is located here.

The following sections briefly describe how to build Sandia OpenSHMEM and the available configuration parameters and runtime environment variables. More detailed information can be found in the SOS Wiki pages.

Building

The Sandia OpenSHMEM implementation utilizes the GNU Autoconf/Automake/Libtool tools to generate a configure script. If the configure file is not present (e.g. after downloading the repository for the first time), generate it by running:

  $ ./autogen.sh

Once the configure file exists, run:

  $ ./configure <options>
  $ make
  $ make check
  $ make install

The make check step is not strictly necessary, but is a good idea. Make check utilizes the TEST_RUNNER and NPROCS make variables, which can be used to override defaults, e.g. make check NPROCS=4 or make check TEST_RUNNER="mpiexec -n 2 -ppn 1 -hosts compute1,compute2".

Sandia OpenSHMEM must be configured to use either the Portals 4 or OFI network transport, but not both. It can optionally be configured to use XPMEM or CMA to optimize communication between PEs within the same shared memory domain.

Customizing the build

Please refer to the full README file located here to see the available configure options, runtime environment variables, OFI transport options, and debugging options.

sos's Issues

Update Documentation

The README file is stale and needs some updates, including:

1. Update project name to Sandia OpenSHMEM
2. Update build instructions for OFI
3. Update build instructions for PMI
4. Update running instructions for PMI
5. Update testing instructions to document TEST_RUNNER and NPROCS make variables
6. Review LICENSE file to ensure it is current
7. Do we want to add a contributors file?
8. Update NEWS file, following GNU NEWS formatting guidelines
9. Update version information to 1.2.0
10. Update Fortran bindings info to describe long/short header option

Update test/feature_tests and test/performance_tests from openshmem.org/tests-uh

1. Push local changes upstream to the openshmem.org repository
2. Pull downstream updated examples to update Sandia SHMEM test suite

We should also add a test/README file to document which directories are downstream of another repository.

assert outgoing transactions are within max transfer limits

Basic error checking: PE argument

Check that PE argument is valid

symmetric address input check

--enable-error-checking configure option should also include input checking for symmetric address verification

Need environment management system

Currently, the environment variables used in the code are checked (and worse, 
listed in sma_info) in an ad-hoc manner.  This is probably not the best path 
forward; we really need some type of management infrastructure to make sure 
that sma_info is always up to date.

Original issue reported on code.google.com by [email protected] on 21 Nov 2013 at 1:44

Building OFI without Portals

Presently, an installation of Portals 4 is required to build, even when Portals is not enabled. This is a reminder to ensure that the OFI transport can be built without an installation of Portals.

LibEV used in CI testing should be fetched from originator website

libev.schmorp.de is currently down, so Travis CI uses a copy of libev on a googledocs page. Once the main libev website is up we should migrate to using the latest version of libev directly from the official source.

libshmem_pmi should not be named as such or need symbolic link from libshmem

PRK should not have to know that SOS was build with PMI included...

[jrhammon@esgmonster github-official]$ make allshmem
cd SHMEM/Synch_p2p;         make p2p       "DEFAULT_OPT_FLAGS   = -O3"
make[1]: Entering directory `/home/jrhammon/Work/INTEL/PCL/ESG/PRK/github-official/SHMEM/Synch_p2p'
/opt/shmem/sandia/intel/bin/oshcc -I/opt/shmem/sandia/intel/include -O3 -DSHMEM     -I../../include -c p2p.c
/opt/shmem/sandia/intel/bin/oshcc -I/opt/shmem/sandia/intel/include -O3 -DSHMEM     -I../../include -c ../../common/wtime.c
/opt/shmem/sandia/intel/bin/oshcc -I/opt/shmem/sandia/intel/include -O3 -DSHMEM     -I../../include -c ../../common/SHMEM_bail_out.c
/opt/shmem/sandia/intel/bin/oshcc -I/opt/shmem/sandia/intel/include  -o p2p -L/opt/shmem/sandia/intel/lib  -O3 -DSHMEM p2p.o wtime.o SHMEM_bail_out.o  -lshmem
ld: cannot find -lshmem
make[1]: *** [p2p] Error 1
make[1]: Leaving directory `/home/jrhammon/Work/INTEL/PCL/ESG/PRK/github-official/SHMEM/Synch_p2p'
make: *** [allshmem] Error 2

[jrhammon@esgmonster github-official]$ ll /opt/shmem/sandia/intel/lib
total 2.6M
-rw-r--r-- 1 root root 114K Dec 22 09:10 libshmem_pmi.a
-rwxr-xr-x 1 root root  958 Dec 22 09:10 libshmem_pmi.la
lrwxrwxrwx 1 root root   21 Dec 22 09:10 libshmem_pmi.so -> libshmem_pmi.so.0.0.0
lrwxrwxrwx 1 root root   21 Dec 22 09:10 libshmem_pmi.so.0 -> libshmem_pmi.so.0.0.0
-rwxr-xr-x 1 root root  77K Dec 22 09:10 libshmem_pmi.so.0.0.0
-rw-r--r-- 1 root root 1.6M Dec 22 09:10 libsma.a
-rwxr-xr-x 1 root root  996 Dec 22 09:10 libsma.la
lrwxrwxrwx 1 root root   15 Dec 22 09:10 libsma.so -> libsma.so.0.0.0
lrwxrwxrwx 1 root root   15 Dec 22 09:10 libsma.so.0 -> libsma.so.0.0.0
-rwxr-xr-x 1 root root 821K Dec 22 09:10 libsma.so.0.0.0

Unit tests fix: pWrk incorrectly sized

Check that pWrk arrays are correctly sized. They are supposed to be of size:

MAX((nreduce / 2 + 1), SHMEM_REDUCE_MIN_WRKDATA_SIZE)

Total data ordering optimizations missing

Optimizations for networks which provide the TOTAL_DATA_ORDERING feature have 
slowly bitrotted and were removed in r288.

Original issue reported on code.google.com by [email protected] on 4 Sep 2013 at 3:01

No resource exhaustion prevention on eq

With the use of bounce buffers for medium messages, its now likely that enough 
eq entries can be delivered (even once #4 is fixed) to overflow the eq.  Either 
we need to size the eq such that a multi-threaded application can not inject 
enough messages to overflow the eq (might be difficult if the volatile size is 
extremely low) or resource control the eq so that it never overflows by rate 
limiting puts.

Original issue reported on code.google.com by [email protected] on 27 Jul 2012 at 4:49

cq usage in OFI transport

cq is no longer needed with openSHMEM 1.3 spec which exposes non-blocking API. The options are to either use the cq for counter error handling or remove the queue all together (this isn't yet exposed in OFI).

Update deprecated constants in tests

Replace _ SHMEM_ constants with new SHMEM_ constants.

bounce buffer optimization disabled

The bounce buffer optimization for medium sized messages is currently disabled 
(which means everything > volatile size uses the long message protocol).  The 
bounce buffer protocol is broken because we never listen for EQ entries for the 
bounce buffered messages, eventually resulting in the queue being overrun 
(which will cause an error somewhere down the line).

There are a couple of potential solutions to draining the queue, including 
polling for EQs after the put, but that raises the problem of what to do with 
the counts for long messages (which we can see due to collectives, which leave 
a number of puts outstanding and then wait for them all).  Since it occurs to 
us that the current method is in no way thread safe, a better strategy all 
around might be needed.

Original issue reported on code.google.com by [email protected] on 27 Jul 2012 at 4:27

C++ Support

We need to add to our headers:

ifdef __cplusplus

extern "C"
{

endif /* __cplusplus */

We also need to verify that header and type dependencies are being handled so C++ code compiles and links correctly against the C header.

adjacent_32bit_amo test fails

On OS X with two processes, the adjacent_32bit_amo test segfaults inside the 
progress engine.

Original issue reported on code.google.com by [email protected] on 9 Dec 2012 at 10:38

Update deprecated API usage in tests

Many tests call deprecated API routines. Update to use new API or mark tests as deprecated if they are testing deprecated functionality (e.g. cache API).

Profiling build broken on Mac

On Mac, configuring with --enable-profiling results in build failures in the test suite:

Undefined symbols for architecture x86_64:
  "__my_pe", referenced from:
      _main in circular_shift.o
  "__num_pes", referenced from:
      _main in circular_shift.o
  "_shmem_barrier_all", referenced from:
      _main in circular_shift.o
  "_shmem_int_get", referenced from:
      _main in circular_shift.o
  "_start_pes", referenced from:
      _main in circular_shift.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Undefined symbols for architecture x86_64:
  "__my_pe", referenced from:
      _main in hello.o
  "__num_pes", referenced from:
      _main in hello.o
  "_start_pes", referenced from:
      _main in hello.o
ld: symbol(s) not found for architecture x86_64
make[2]: *** [circular_shift] Error 1
make[2]: *** Waiting for unfinished jobs....
Undefined symbols for architecture x86_64:
  "__my_pe", referenced from:
      _main in accessible_ping.o
  "__num_pes", referenced from:
      _main in accessible_ping.o
  "_shmem_pe_accessible", referenced from:
      _main in accessible_ping.o
  "_start_pes", referenced from:
      _main in accessible_ping.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [hello] Error 1
make[2]: *** [accessible_ping] Error 1
Undefined symbols for architecture x86_64:
  "__my_pe", referenced from:
      _main in micro_unit_shmem.o
  "__num_pes", referenced from:
      _main in micro_unit_shmem.o
  "_shmem_fence", referenced from:
      _putfence in micro_unit_shmem.o
      _gettest in micro_unit_shmem.o
      _atomic_add in micro_unit_shmem.o
      _atomic_inc in micro_unit_shmem.o
      _fetchatomic_add in micro_unit_shmem.o
      _fetchatomic_inc in micro_unit_shmem.o
  "_shmem_int_add", referenced from:
      _atomic_add in micro_unit_shmem.o
  "_shmem_int_cswap", referenced from:
      _cswaptest in micro_unit_shmem.o
  "_shmem_int_fadd", referenced from:
      _fetchatomic_add in micro_unit_shmem.o
  "_shmem_int_finc", referenced from:
      _fetchatomic_inc in micro_unit_shmem.o
  "_shmem_int_g", referenced from:
      _gettest in micro_unit_shmem.o
  "_shmem_int_inc", referenced from:
      _atomic_inc in micro_unit_shmem.o
  "_shmem_int_p", referenced from:
      _putfence in micro_unit_shmem.o
      _gettest in micro_unit_shmem.o
      _swaptest in micro_unit_shmem.o
      _cswaptest in micro_unit_shmem.o
  "_shmem_int_swap", referenced from:
      _swaptest in micro_unit_shmem.o
  "_shmem_int_wait_until", referenced from:
      _wait_until in micro_unit_shmem.o
  "_start_pes", referenced from:
      _main in micro_unit_shmem.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Undefined symbols for architecture x86_64:
  "__my_pe", referenced from:
      _main in max_reduction.o
  "__num_pes", referenced from:
      _main in max_reduction.o
  "_shmem_barrier_all", referenced from:
      _main in max_reduction.o
  "_shmem_long_max_to_all", referenced from:
      _main in max_reduction.o
  "_start_pes", referenced from:
      _main in max_reduction.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Update SMA_INFO output

The SMA_INFO output in src/init.c is missing some environment variables. For example, CMA variables should be output when CMA is included in the build.

Global Exit Busted

The current shmem_global_exit() implementation is not up to scratch. PEs sitting in shmem_barrier when shmem_global_exit is called end up hanging or erroring out without the desired exit code.

libtoolize: command not found

From https://travis-ci.org/jeffhammond/PRK/jobs/99387697:

+cd /Users/travis/PRK-deps
+git clone --depth 10 https://github.com/regrant/sandia-shmem.git
Cloning into 'sandia-shmem'...
remote: Counting objects: 252, done.
remote: Compressing objects: 100% (193/193), done.
remote: Total 252 (delta 99), reused 116 (delta 58), pack-reused 0
Receiving objects: 100% (252/252), 267.67 KiB | 0 bytes/s, done.
Resolving deltas: 100% (99/99), done.
Checking connectivity... done.
+cd sandia-shmem
+./autogen.sh
+test -d ./config
+aclocal -I config
+libtoolize --force --copy
./autogen.sh: line 6: libtoolize: command not found
+autoheader
+automake --foreign --add-missing --copy

This is addressed on StackOverflow.

A patch is in-progress.

Env Var Consistency Checks

Add initialization-time checking to to verify that parameters (e.g. env vars that select collective algorithms) were provided with the same value at every PE. Alternatively, these could be read at PE 0 and broadcast to all others. This is probably part of a larger ticket to overhaul how we handle runtime parameters.

Symbol hiding macros for internal variables / functions

Currently, all non-static internal symbols are named shmem_internal_*, but are 
public functions in the sma library.  They should really be marked as private 
on linkers that support public/private function marking.

Original issue reported on code.google.com by [email protected] on 21 Nov 2013 at 1:41

bcast test hangs

  When the requested sizes of ni_req_limits.max_waw_ordered_size and ni_req_limits.max_volatile_size [input to PtlNIInit()] are boosted to any value > 512, at least 3 SHMEM tests (make check PROCS=4) [big_reduction, bcast & fcollect64) will reliably hang; as in never exit.
When ni_req_limits.max_waw_ordered_size and ni_req_limits.max_volatile_size == 
512, the 3 offending tests pass with no problems.
Recent changes in portals-shmem/src/runtime.c, addressing the put performance 
issues, cause hangs using Portals4.svn 1825 or 1808.
Could you verify this behavior?
I'll get you a stack trace, although I'm sure it will only show a wait for a 
Portals event.

Original issue reported on code.google.com by [email protected] on 8 May 2012 at 10:04

compiler wrapper needs Pthreads for static linking

If I try to build a static binary, oshcc fails due to not knowing where Pthreads is...

/opt/shmem/sandia/intel/bin/oshcc -static -I/opt/shmem/sandia/intel/include  -o p2p -L/opt/shmem/sandia/intel/lib  -O3 -DSHMEM p2p.o wtime.o SHMEM_bail_out.o  -lshmem
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_ep_getinfo':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:456: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/opt/shmem/sandia/intel/lib/libsma.a(simple_pmi.o): In function `PMII_Connect_to_pm':
/home/jrhammon/Work/PGAS/SHMEM/sandia-shmem/shmem_pmi/simple_pmi.c:885: warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/opt/libfabric//lib/libfabric.a(src_libfabric_la-fabric.o): In function `fi_ini':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../src/fabric.c:369: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../src/fabric.c:348: undefined reference to `pthread_mutex_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../src/fabric.c:404: undefined reference to `pthread_mutex_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_dom_add_to_list':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:79: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_dom_check_list':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:101: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:103: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:103: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_dom_remove_from_list':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:109: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_dom_list_head':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:119: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:126: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_fab_add_to_list':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:132: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_fabric':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:341: undefined reference to `pthread_spin_init'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_fab_check_list':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:154: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:156: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:156: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_fab_remove_from_list':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:162: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_fabric_close':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:296: undefined reference to `pthread_spin_destroy'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_fab_list_head':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:172: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:179: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_fabric_check_service':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:377: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:379: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:379: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_fabric_add_service':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:392: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_fabric_remove_service':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:400: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `fi_sockets_ini':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:624: undefined reference to `pthread_spin_init'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `fi_sockets_fini':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:588: undefined reference to `pthread_spin_destroy'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_dom_add_to_list':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:81: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_dom_remove_from_list':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:113: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_fab_add_to_list':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:134: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_fab_remove_from_list':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:166: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o): In function `sock_fabric_add_service':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:394: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_fabric.o):/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_fabric.c:406: more undefined references to `pthread_spin_unlock' follow
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep.o): In function `sock_rx_ctx_cancel':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:440: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:474: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep.o): In function `sock_ep_close':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:607: undefined reference to `pthread_mutex_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:619: undefined reference to `pthread_mutex_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:632: undefined reference to `pthread_spin_destroy'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:657: undefined reference to `pthread_spin_destroy'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:626: undefined reference to `pthread_join'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:615: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:617: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:609: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:611: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:597: undefined reference to `pthread_join'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep.o): In function `sock_tx_size_left':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:552: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:554: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep.o): In function `sock_ctx_bind_cq':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:172: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:174: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:159: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep.o): In function `sock_ctx_bind_cntr':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:240: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:242: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:224: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep.o): In function `sock_ep_bind':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:740: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:743: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:810: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:813: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:865: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:867: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:856: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:858: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep.o): In function `sock_alloc_endpoint':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:1498: undefined reference to `pthread_spin_init'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:1548: undefined reference to `pthread_spin_init'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep.o): In function `sock_ep_get_conn':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:1609: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:1615: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep.c:1615: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ctx.o): In function `sock_rx_ctx_alloc':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ctx.c:61: undefined reference to `pthread_spin_init'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ctx.o): In function `sock_tx_context_alloc':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ctx.c:99: undefined reference to `pthread_spin_init'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ctx.c:100: undefined reference to `pthread_spin_init'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ctx.c:101: undefined reference to `pthread_spin_init'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ctx.o): In function `sock_rx_ctx_free':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ctx.c:72: undefined reference to `pthread_spin_destroy'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ctx.o): In function `sock_tx_ctx_free':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ctx.c:146: undefined reference to `pthread_spin_destroy'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ctx.c:147: undefined reference to `pthread_spin_destroy'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ctx.c:148: undefined reference to `pthread_spin_destroy'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ctx.o): In function `sock_tx_ctx_start':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ctx.c:156: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ctx.o): In function `sock_tx_ctx_commit':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ctx.c:168: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ctx.o): In function `sock_tx_ctx_abort':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ctx.c:174: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_progress_rx_ep':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2431: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2448: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_process_rx_conn_msg':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1619: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1623: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_process_rx_send':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1543: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1554: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1465: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1494: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1577: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1580: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1478: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_rx_peek_recv':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1244: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1273: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_rx_claim_recv':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1288: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:1338: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_signal':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2362: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2369: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_poll_add':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2374: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_add_tx_ctx':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2382: undefined reference to `pthread_mutex_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_add_rx_ctx':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2391: undefined reference to `pthread_mutex_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_remove_tx_ctx':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2400: undefined reference to `pthread_mutex_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_remove_rx_ctx':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2407: undefined reference to `pthread_mutex_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_progress_rx_ctx':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2459: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2461: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2463: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2493: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_progress_tx_ctx':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2529: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2531: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2536: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2561: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2536: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2536: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_thread_set_affinity':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2631: undefined reference to `pthread_self'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2659: undefined reference to `pthread_setaffinity_np'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_progress_thread':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2691: undefined reference to `pthread_mutex_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2714: undefined reference to `pthread_mutex_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2719: undefined reference to `pthread_mutex_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_poll':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2579: undefined reference to `pthread_mutex_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2600: undefined reference to `pthread_mutex_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2605: undefined reference to `pthread_mutex_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2611: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2618: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_init':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2757: undefined reference to `pthread_spin_init'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2758: undefined reference to `pthread_spin_init'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2759: undefined reference to `pthread_mutex_init'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2775: undefined reference to `pthread_create'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2790: undefined reference to `pthread_spin_destroy'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_finalize':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2810: undefined reference to `pthread_spin_destroy'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2811: undefined reference to `pthread_spin_destroy'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2812: undefined reference to `pthread_mutex_destroy'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2801: undefined reference to `pthread_join'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_poll_add':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2377: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2377: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_add_tx_ctx':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2385: undefined reference to `pthread_mutex_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_add_rx_ctx':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2394: undefined reference to `pthread_mutex_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_remove_tx_ctx':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2402: undefined reference to `pthread_mutex_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_progress.o): In function `sock_pe_remove_rx_ctx':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_progress.c:2409: undefined reference to `pthread_mutex_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_conn.o): In function `sock_conn_map_init':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_conn.c:104: undefined reference to `pthread_spin_init'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_conn.o): In function `sock_conn_map_insert':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_conn.c:160: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_conn.c:162: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_conn.o): In function `_sock_conn_listen':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_conn.c:261: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_conn.c:263: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_conn.o): In function `sock_conn_listen':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_conn.c:360: undefined reference to `pthread_create'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_conn.o): In function `sock_ep_connect':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_conn.c:392: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_conn.c:394: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_conn.c:478: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_conn.c:486: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_conn.o): In function `sock_conn_map_destroy':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_conn.c:137: undefined reference to `pthread_spin_destroy'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_msg.o): In function `sock_ep_recvmsg':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_msg.c:113: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_msg.c:115: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_msg.c:137: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_msg.c:139: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_msg.o): In function `sock_ep_trecvmsg':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_msg.c:446: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_msg.c:448: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_msg.c:470: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_msg.c:473: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_trigger.o): In function `sock_queue_rma_op':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_trigger.c:82: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_trigger.c:84: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_trigger.o): In function `sock_queue_msg_op':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_trigger.c:121: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_trigger.c:123: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_trigger.o): In function `sock_queue_tmsg_op':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_trigger.c:160: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_trigger.c:162: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_trigger.o): In function `sock_queue_atomic_op':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_trigger.c:214: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_trigger.c:216: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-common.o): In function `fi_wait_cond':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../src/common.c:59: undefined reference to `clock_gettime'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../src/common.c:62: undefined reference to `pthread_cond_timedwait'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../src/common.c:57: undefined reference to `pthread_cond_wait'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_av.o): In function `sock_av_close':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_av.c:411: undefined reference to `shm_unlink'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_av.o): In function `sock_av_open':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_av.c:514: undefined reference to `shm_open'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_av.c:542: undefined reference to `shm_unlink'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_dom.o): In function `sock_mr_close':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_dom.c:197: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_dom.c:202: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_dom.o): In function `sock_dom_close':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_dom.c:169: undefined reference to `pthread_spin_destroy'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_dom.o): In function `sock_regattr':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_dom.c:330: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_dom.c:341: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_dom.c:359: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_dom.o): In function `sock_domain':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_dom.c:491: undefined reference to `pthread_spin_init'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_dom.c:491: undefined reference to `pthread_spin_init'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_eq.o): In function `sock_eq_fi_close':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_eq.c:275: undefined reference to `pthread_spin_destroy'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_eq.o): In function `sock_eq_sread':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_eq.c:95: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_eq.c:114: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_eq.c:114: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_eq.o): In function `sock_eq_readerr':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_eq.c:136: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_eq.c:162: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_eq.c:162: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_eq.o): In function `sock_eq_report_event':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_eq.c:180: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_eq.c:184: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_eq.o): In function `sock_eq_report_error':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_eq.c:225: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_eq.c:230: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_eq.o): In function `sock_eq_open':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_eq.c:388: undefined reference to `pthread_spin_init'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cq.o): In function `sock_cq_progress':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:61: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cq.o): In function `sock_cq_close':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:375: undefined reference to `pthread_spin_destroy'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:376: undefined reference to `pthread_spin_destroy'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cq.o): In function `sock_cq_sreadfrom':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:292: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:286: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:292: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:292: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:301: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:308: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cq.o): In function `sock_cq_readerr':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:341: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:348: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cq.o): In function `_sock_cq_write':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:114: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:146: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cq.o): In function `sock_cq_open':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:542: undefined reference to `pthread_spin_init'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:583: undefined reference to `pthread_spin_init'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cq.o): In function `sock_cq_report_error':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:604: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:630: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cq.o): In function `sock_cq_check_size_ok':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:637: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:641: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cq.o): In function `sock_cq_progress':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cq.c:73: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cntr.o): In function `sock_cntr_close':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:330: undefined reference to `pthread_mutex_destroy'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:331: undefined reference to `pthread_spin_destroy'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:332: undefined reference to `pthread_spin_destroy'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:334: undefined reference to `pthread_cond_destroy'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cntr.o): In function `sock_cntr_progress':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:63: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cntr.o): In function `sock_cntr_check_trigger_list':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:86: undefined reference to `pthread_spin_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cntr.o): In function `sock_cntr_wait':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:225: undefined reference to `pthread_mutex_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:271: undefined reference to `pthread_mutex_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:266: undefined reference to `pthread_mutex_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:246: undefined reference to `pthread_mutex_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:259: undefined reference to `pthread_mutex_lock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cntr.o): In function `sock_cntr_set':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:208: undefined reference to `pthread_mutex_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:211: undefined reference to `pthread_cond_signal'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:212: undefined reference to `pthread_mutex_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cntr.o): In function `sock_cntr_add':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:194: undefined reference to `pthread_mutex_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:197: undefined reference to `pthread_cond_signal'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:198: undefined reference to `pthread_mutex_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cntr.o): In function `sock_cntr_inc':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:169: undefined reference to `pthread_mutex_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:172: undefined reference to `pthread_cond_signal'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:173: undefined reference to `pthread_mutex_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cntr.o): In function `sock_cntr_err_inc':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:180: undefined reference to `pthread_mutex_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:184: undefined reference to `pthread_cond_signal'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:185: undefined reference to `pthread_mutex_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cntr.o): In function `sock_cntr_open':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:410: undefined reference to `pthread_cond_init'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:458: undefined reference to `pthread_mutex_init'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:459: undefined reference to `pthread_spin_init'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:471: undefined reference to `pthread_spin_init'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cntr.o): In function `sock_cntr_progress':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:75: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_cntr.o): In function `sock_cntr_check_trigger_list':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_cntr.c:156: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_poll.o): In function `sock_poll_poll':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_poll.c:101: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_poll.c:130: undefined reference to `pthread_spin_unlock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_poll.c:124: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_poll.c:113: undefined reference to `pthread_mutex_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_poll.c:119: undefined reference to `pthread_mutex_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_wait.o): In function `sock_wait_init':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_wait.c:96: undefined reference to `pthread_mutex_init'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_wait.c:97: undefined reference to `pthread_cond_init'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_wait.o): In function `sock_wait_signal':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_wait.c:199: undefined reference to `pthread_cond_signal'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep_msg.o): In function `sock_ep_cm_enqueue_msg':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep_msg.c:414: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep_msg.c:416: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep_msg.o): In function `sock_ep_cm_flush_msg':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep_msg.c:475: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep_msg.c:501: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep_msg.o): In function `sock_pep_fi_close':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep_msg.c:975: undefined reference to `pthread_join'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep_msg.c:981: undefined reference to `pthread_spin_destroy'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep_msg.o): In function `sock_pep_start_listener_thread':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep_msg.c:1143: undefined reference to `pthread_create'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep_msg.o): In function `sock_ep_cm_handle_ack':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep_msg.c:531: undefined reference to `pthread_spin_lock'
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep_msg.c:574: undefined reference to `pthread_spin_unlock'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep_msg.o): In function `sock_msg_ep':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep_msg.c:933: undefined reference to `pthread_create'
/opt/libfabric//lib/libfabric.a(src_libfabric_la-sock_ep_msg.o): In function `sock_msg_passive_ep':
/home/jrhammon/Work/INTEL/SFI/libfabric/build/../prov/sockets/src/sock_ep_msg.c:1294: undefined reference to `pthread_spin_init'
/opt/gcc/5.3.0/lib/gcc/x86_64-unknown-linux-gnu/5.3.0//libgcc_eh.a(unwind-dw2-fde-dip.o): In function `__gthread_mutex_lock':
/tmp/gcc-5.3.0/x86_64-unknown-linux-gnu/libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock'
/tmp/gcc-5.3.0/x86_64-unknown-linux-gnu/libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock'
/tmp/gcc-5.3.0/x86_64-unknown-linux-gnu/libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock'
/tmp/gcc-5.3.0/x86_64-unknown-linux-gnu/libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock'
/opt/gcc/5.3.0/lib/gcc/x86_64-unknown-linux-gnu/5.3.0//libgcc_eh.a(unwind-dw2-fde-dip.o): In function `__gthread_mutex_unlock':
/tmp/gcc-5.3.0/x86_64-unknown-linux-gnu/libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock'
/opt/gcc/5.3.0/lib/gcc/x86_64-unknown-linux-gnu/5.3.0//libgcc_eh.a(unwind-dw2-fde-dip.o): In function `__gthread_mutex_lock':
/tmp/gcc-5.3.0/x86_64-unknown-linux-gnu/libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock'
/opt/gcc/5.3.0/lib/gcc/x86_64-unknown-linux-gnu/5.3.0//libgcc_eh.a(unwind-dw2-fde-dip.o): In function `__gthread_mutex_unlock':
/tmp/gcc-5.3.0/x86_64-unknown-linux-gnu/libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock'
/tmp/gcc-5.3.0/x86_64-unknown-linux-gnu/libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock'
/tmp/gcc-5.3.0/x86_64-unknown-linux-gnu/libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock'
/tmp/gcc-5.3.0/x86_64-unknown-linux-gnu/libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock'
/tmp/gcc-5.3.0/x86_64-unknown-linux-gnu/libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock'
make[1]: *** [p2p] Error 1
make[1]: Leaving directory `/home/jrhammon/Work/INTEL/PCL/ESG/PRK/github-official/SHMEM/Synch_p2p'
make: *** [allshmem] Error 2

impossible to register IB memory on XPMEM shared memory

It is impossible to register with IB some memory that has been attached with 
XPMEM.
On that test case, with centos 6.2, ibv_reg_mr fails with errno=EFAULT

Original issue reported on code.google.com by [email protected] on 14 Jun 2012 at 8:26

Update Fortran Bindings for OpenSHMEM 1.2

Fortran bindings are incomplete.

Replace abort with exit?

When exiting in error, should we call exit() instead of abort()? Abort generates SIGABRT and on some systems captures a core dump.

CMA and Portals

CMA currently makes a Portals call in shmem_internal_put_nb, causing a build failure when CMA is used with the OFI transport. Is calling the transport the right solution or should CMA also work in a transport-less shared memory build?

PMI configure testing

This is fixed by configuring with --enable-pmi-simple. If that cannot be the default, then configure should check for pmi.h and enable that option if it is not found.

Here's the issue I saw originally:

./configure --with-ofi=/opt/libfabric/  --enable-remote-virtual-addressing --disable-fortran --prefix=/opt/shmem/sandia && make -j32

leads to this

[jrhammon@esgmonster sandia-shmem]$ make
Making all in mpp
make[1]: Entering directory `/home/jrhammon/Work/PGAS/SHMEM/sandia-shmem/mpp'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/jrhammon/Work/PGAS/SHMEM/sandia-shmem/mpp'
Making all in src
make[1]: Entering directory `/home/jrhammon/Work/PGAS/SHMEM/sandia-shmem/src'
make  all-am
make[2]: Entering directory `/home/jrhammon/Work/PGAS/SHMEM/sandia-shmem/src'
  CC       runtime-pmi.lo
runtime-pmi.c:26:17: fatal error: pmi.h: No such file or directory
compilation terminated.
make[2]: *** [runtime-pmi.lo] Error 1
make[2]: Leaving directory `/home/jrhammon/Work/PGAS/SHMEM/sandia-shmem/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/jrhammon/Work/PGAS/SHMEM/sandia-shmem/src'
make: *** [all-recursive] Error 1

Basic error checking: Local/Remote Buffer

Check that local/remote buffer argument is non-NULL when length > 0. If length == 0, functions should return immediately.

Support VPATH Builds

Presently, we don't support VPATH builds, as reported below:

jhammond@cori04:~/SHMEM/sandia-shmem/build-gcc-ofi> ../configure CC=icc FC=ifort              --with-pmi=/opt/cray/pmi/default              --with-xpmem=/opt/cray/xpmem/default               --with-ofi=$HOME/OFI/install-ofi-gcc-gni-cori              --enable-remote-virtual-addressing              --enable-static              --prefix=$HOME/SHMEM/install-sandia-shmem-ofi-xpmem-icc              LDFLAGS="-L/opt/cray/ugni/default/lib64 -lugni \
                      -L/opt/cray/alps/default/lib64 -lalps -lalpslli -lalpsutil \
                      -ldl -lrt" && make  
...
Compilers:
  C:              icc
  Fortran:        ifort

Transports:
  Portals 4:      no
  OFI:            yes
  XPMEM:          no
  CMA:            no

Making all in mpp
make[1]: Entering directory `/global/u1/j/jhammond/SHMEM/sandia-shmem/build-gcc-ofi/mpp'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/global/u1/j/jhammond/SHMEM/sandia-shmem/build-gcc-ofi/mpp'
Making all in src
make[1]: Entering directory `/global/u1/j/jhammond/SHMEM/sandia-shmem/build-gcc-ofi/src'
(CDPATH="${ZSH_VERSION+.}:" && cd ../.. && /bin/sh /global/homes/j/jhammond/SHMEM/sandia-shmem/config/missing autoheader)
rm -f stamp-h1
touch ../../src/config.h.in
cd .. && /bin/sh ./config.status src/config.h
config.status: creating src/config.h
make  all-am
make[2]: Entering directory `/global/u1/j/jhammond/SHMEM/sandia-shmem/build-gcc-ofi/src'
  CC       shmem_free_list.lo
../../src/shmem_free_list.c(18): catastrophic error: cannot open source file "shmem.h"
  #include "shmem.h"
                    ^

compilation aborted for ../../src/shmem_free_list.c (code 4)
make[2]: *** [shmem_free_list.lo] Error 1
make[2]: Leaving directory `/global/u1/j/jhammond/SHMEM/sandia-shmem/build-gcc-ofi/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/global/u1/j/jhammond/SHMEM/sandia-shmem/build-gcc-ofi/src'
make: *** [all-recursive] Error 1
jhammond@cori04:~/SHMEM/sandia-shmem/build-gcc-ofi> fid . 
jhammond@cori04:~/SHMEM/sandia-shmem/build-gcc-ofi> find .. -name shmem.h
../build-gcc-ofi/mpp/shmem.h

Update test/examples from openshmem.org/openshmem-examples

1. Push local changes upstream to the openshmem.org repository
2. Pull downstream updated examples to update Sandia SHMEM test suite

implement shmem_internal_put

I see heavy use of shmem_internal_put_nb followed by shmem_internal_put_wait throughout the code. Why not add shmem_internal_put that blocks to eliminate all these unnecessary lines of code?

I recognize at this point that this improvement constitutes more work than the status quo. If you are interested, I will work on PR.

collect broken

Stan is reporting that they're seeing random failures in collect benchmarks.

When I was looking at the code for potential optimizations, I noticed that a 
rank sends its data to rank 0, then sends a completion/size notice to myrank + 
1.  Highest rank sends a completion word to rank 0.  This is unsafe, as there's 
no ordering between the two sends.  I'm not sure if this is the problem, but it 
certainly needs to be fixed.

Original issue reported on code.google.com by [email protected] on 14 Oct 2013 at 9:06

shmem_heat_image does not scale beyond 6 PEs

What steps will reproduce the problem?
1. cd .../portals-shmem/test/openshmem/Examples
2. make check PROCS=8
3.

What is the expected output? What do you see instead?
Expect:
yod -f ../../hostfile -np 8 ./shmem_heat_image 10 10
Solving heat conduction task on 10 by 10 grid by 8 processors
Iteration 0
Iteration 1000
Iteration 2000
Iteration 3000
Iteration 4000
Elapsed time: 0.23 sec

Instead:
Cannot distribute rows, too many processors

Suggested fix:
Change shmem_heat_image.c for argc == 1 from 10 to _num_pes()
   if (1 == argc) {
-      totalmx = mx = 10;
-      my = 10;
+      my = totalmx = mx = n_of_nodes;

Original issue reported on code.google.com by [email protected] on 25 Jan 2013 at 9:53

Utility functions for collectives: put-with-flag, send, recv, etc.

The following pattern (in src/collectives.c) is essentially send-recv, and it appears frequently in the collectives code. Why not reduce the chance of programmer errors and implement shmem_internal_send and shmem_internal_recv? That would make it much easier to adapt algorithms from e.g. MPICH or OpenMPI as well.

        /* send data to peer */
        shmem_internal_put_nb((char*) target + curr_offset, (char*) target + curr_offset,
                              distance * len, real_peer, &completion);
        shmem_internal_put_wait(&completion);
        shmem_internal_fence();

        /* mark completion for this round */
        shmem_internal_atomic_small(&pSync_bytes[i], &one, sizeof(int8_t),
                                    real_peer,
                                    SHM_INTERNAL_SUM, SHM_INTERNAL_SIGNED_BYTE);

        SHMEM_WAIT_UNTIL(&pSync_bytes[i], SHMEM_CMP_NE, 0);

This pattern is used approximately 37 times in the collectives code:

jrhammon-mac01:sandia-shmem jrhammon$ git grep -c SHMEM_WAIT src/collectives.c 
src/collectives.c:37

Examples from OpenSHMEM spec

Convert the examples from the spec into test cases, if they aren't already.

oops when using XPMEM with threads

Run xpmem-th-server, then on another terminal run xpmem-th-client with
the two parameters given by the server --> oops



$ ./xpmem-th-server
Server parameters: 8589958440 16384


$ ./xpmem-th-client  8589958440 16384



The problem seems to be that xpmem_fault_handler is entered by both
threads at the same time. The first remap_pfn_range succeds, but the
second, which is called with the same parameters fails.

Original issue reported on code.google.com by [email protected] on 14 Jun 2012 at 8:24

Add support for SMA_DEBUG env var

If SMA_DEBUG is set, the implementation should print out additional debugging information.
Presently, this is not supported.

shmem_2dheat test does not scale

What steps will reproduce the problem?
1. yod -f hostfile -np 16 shmem_2dheat -h 20 -w 20 -m 2
2. test hangs
3. defaults '-w 20 -h 20' insufficient grid sizing in order to make forward 
progress.

What is the expected output? What do you see instead?
test completion instead of test hang.

-  if (ROOT == my_rank && argc < 2)
+  if (argc < 2)
     {
-      printf ("Using defaults: -h 20 -w 20 -m 2\n");
+      HEIGHT = _num_pes() + 8;
+      WIDTH = _num_pes() + 8;
+      if (ROOT == my_rank)
+        printf ("Using defaults: -h %d -w %d -m 2\n",HEIGHT,WIDTH);
     }

Suggested patch is a temp fix (+8)as this is not a complete solution. I suspect 
there is an algorthm bug in exceeding grid bounds, although after spending a 
couple hours of code reading I'm not seeing it.
Main point is this tmp fix scales to 69 PEs instead of 16.

Original issue reported on code.google.com by [email protected] on 26 Jan 2013 at 12:49

Test suite failures with NPROCS=1

Multiple tests generate errors or hang when NPROCS=1. This is especially bad, since the default number of processes is 1 when the user doesn't set the NPROCS environment variable.

Potential shmem_test_lock race

The UH test feature_tests/C/test_shmem_lock.c occasionally hangs after printing:

Test for set, and clear lock: Passed

There could be a race in the ensuing test of shmem_test_lock(). So far, I've only observed this hang in the Travis environment.

correct size for inject

It looks like there is an assumption that 'long double' is the largest size type. This is not actually guaranteed by the C language standard. For determining things like the inject size, we should be checking to find the largest size, like largest_type_size = max(sizeof(long double), sizeof(double complex)). There may also be a need to consider the Fortran types (I don't recall if the user can do things like set the size of real to be 128-bit). Those sizes are captured at configure time in src/config.h.

CMA_GET/PUT_MAX Backwards?

Is this backwards? Should we be using the network for small transfers and CMA for larger transfers that amortize the system call overhead?

CMA_PUT_MAX (default: 8192)
'--with-cma', shmem put lengths <= CMA_PUT_MAX use process_vm_writev();
otherwise use Portals4 transport put.

missing shfree() calls in shmem_2dheat.c

What steps will reproduce the problem?
1. test/openshmem/examples/shmem_2dheat.c
2. Missing shfree() calls which eventually exceed SMA_SYMMETRIC_SIZE default 
allocation, resulting in shmalloc() failure.

End of following functions need
jacobi()
gauss_seidel()
sor()

+    shfree(U_Send_Buffer);
+    shfree(U_Curr_Below);
+    shfree(U_Curr_Above);

Original issue reported on code.google.com by [email protected] on 26 Jan 2013 at 12:38

Basic error checking: PE subset arguments in collectives

Check that PE active set arguments to collectives span a valid range of PEs.

Can wait/wait_until block on network activity?

The current implementations of shmem_wait() and shmem_wait_until() block on network activity. We implement put/AMO using shared memory (e.g. via XPMEM/CMA) and we also allow direct pointer access (i.e., shmem_ptr() in XPMEM builds) from other PEs. Given that the update may not come from the network, the current implementation appears to be incorrect.

PMI Support

Current PMI support is a mess; it relies on being able to pull PMI headers/library from the Portals 4 code. Find a way to make this better. Could be adding a launcher to Sandia SHMEM that can be selected at compile time, or modifying the build to use PMI from the system MPI library.