ompi-x/libomp

libomp's Issues

looks like OMP nthreads is off-by-one

I just ran a test against the latest gv_pmix branch and noticed that the omp_get_thread_num() returns one less than expected. For example, if NumCpus=12 and NumRanks=2, would expect to have 6 threads but instead we get 5 threads.

  [PID:1037] DBG: LINE:1780 nthreads=5, _n_local_cpus=12, _n_local_ranks=2
  [PID:1036] DBG: LINE:1780 nthreads=5, _n_local_cpus=12, _n_local_ranks=2
 ...<snip>...
  [mpi_omp_hello.c:main:48] In parallel loop: calling omp_get_thread_num()...
  [mpi_omp_hello.c:main:50] omp_get_thread_num() succeeded
  Number of threads = 5

Looks like this is related to kmp_runtime.cpp#L1779.

binding may be floating to all places

Adding this as issue to track status while testing. Using gv_pmix branch at da5ea2c (after PR #4 was merged).

I enabled debug prints/instrumentation to __kmp_affinity_set_init_mask() to show the binding of OMP threads to places. I see that some threads show all places instead of a specific place X.

Here's the command I used on machine with 12 cores (libomp/moc test show NCPUs=12).

NOTE: I lowered the KA_TRACE() lines to debug level=5 in code, otherwise use KMP_A_DEBUG=101.

  beaker:$ env KMP_A_DEBUG=6 mpirun \
                   -x OMP_PLACES=threads \
                  -x OMP_CPU_AFFINITY=TRUE \
                  -x OMP_PROC_BIND=spread \ 
                  --map-by core -np 2 \
                 ./mpi_omp_hello.dbg

Here's the relevant output sorted for easier reading and with the PID info:

  __kmp_affinity_set_init_mask: binding T#0 (PID:13615) to place 0
  __kmp_affinity_set_init_mask: binding T#1 (PID:13615) to all places
  __kmp_affinity_set_init_mask: binding T#2 (PID:13615) to place 1
  __kmp_affinity_set_init_mask: binding T#3 (PID:13615) to place 1
  __kmp_affinity_set_init_mask: binding T#4 (PID:13615) to place 1

  __kmp_affinity_set_init_mask: binding T#0 (PID:13614) to place 0
  __kmp_affinity_set_init_mask: binding T#1 (PID:13614) to all places
  __kmp_affinity_set_init_mask: binding T#2 (PID:13614) to all places
  __kmp_affinity_set_init_mask: binding T#3 (PID:13614) to place 0
  __kmp_affinity_set_init_mask: binding T#4 (PID:13614) to place 0

Implement hooks for the creation of places

Implement the hooks for the creation of places so we can do more advanced partitioning of available resources when more than 1 MPI rank is running on the node.
We already identified the __kmp_partition_places() function (runtime/src/kmp_partition.cpp) that seems to initialize places based on the binding strategy; it might be a good place to start.

Also, the goal would ultimately be to control where and how Open MP places are created while sharing resources between MPI ranks.

Implement hooks for controlling BINDING_STRATEGIES