binding may be floating to all places

[ompi-x]

binding may be floating to all places

[gvallee]

Test on the Thomas' desktop: 6 cores, 12 HT. I will include the output of the test on my desktop asap, I believe I do not have that problem but there is clearly a bug somewhere.

[gvallee]

The test is running only 2 MPI ranks but the work I did was assuming 1 MPI rank per core and then spread OpenMP threads on the HT of that core. It is still a use case we want to support but it is not yet working.

[naughtont3]

Quick follow-up test on BEAKER desktop (lstopo shows: 6 cores, each with 2 hwthreads).

beaker:$ env KMP_A_DEBUG=6 mpirun  -x OMP_PLACES=threads -x OMP_CPU_AFFINITY=TRUE -x OMP_PROC_BIND=spread  --map-by core \
             -np 2     ./mpi_omp_hello.dbg >& LOG.np2
beaker:$ grep binding LOG.np2 | sort -k4
__kmp_affinity_set_init_mask: binding T#1 (PID:56312) to all places
__kmp_affinity_set_init_mask: binding T#3 (PID:56312) to all places
__kmp_affinity_set_init_mask: binding T#0 (PID:56312) to place 0
__kmp_affinity_set_init_mask: binding T#2 (PID:56312) to place 0
__kmp_affinity_set_init_mask: binding T#4 (PID:56312) to place 0
__kmp_affinity_set_init_mask: binding T#1 (PID:56313) to all places
__kmp_affinity_set_init_mask: binding T#0 (PID:56313) to place 0
__kmp_affinity_set_init_mask: binding T#2 (PID:56313) to place 1
__kmp_affinity_set_init_mask: binding T#3 (PID:56313) to place 1
__kmp_affinity_set_init_mask: binding T#4 (PID:56313) to place 1
beaker:$ env KMP_A_DEBUG=6 mpirun  -x OMP_PLACES=threads -x OMP_CPU_AFFINITY=TRUE -x OMP_PROC_BIND=spread  --map-by core \
               -np 2     ./mpi_omp_hello.dbg >& LOG.np2.2
beaker:$ grep binding LOG.np2.2 | sort -k4
__kmp_affinity_set_init_mask: binding T#1 (PID:56336) to all places
__kmp_affinity_set_init_mask: binding T#2 (PID:56336) to all places
__kmp_affinity_set_init_mask: binding T#0 (PID:56336) to place 0
__kmp_affinity_set_init_mask: binding T#3 (PID:56336) to place 0
__kmp_affinity_set_init_mask: binding T#4 (PID:56336) to place 0
__kmp_affinity_set_init_mask: binding T#1 (PID:56337) to all places
__kmp_affinity_set_init_mask: binding T#3 (PID:56337) to all places
__kmp_affinity_set_init_mask: binding T#0 (PID:56337) to place 0
__kmp_affinity_set_init_mask: binding T#2 (PID:56337) to place 1
__kmp_affinity_set_init_mask: binding T#4 (PID:56337) to place 1
beaker:$ env KMP_A_DEBUG=6 mpirun  -x OMP_PLACES=threads -x OMP_CPU_AFFINITY=TRUE -x OMP_PROC_BIND=spread  --map-by core \
            -np 6     ./mpi_omp_hello.dbg >& LOG.np6
beaker:$ grep binding LOG.np6 | sort -k4
__kmp_affinity_set_init_mask: binding T#0 (PID:56360) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56361) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56362) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56363) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56364) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56365) to place 0
beaker:$ env KMP_A_DEBUG=6 mpirun  -x OMP_PLACES=threads -x OMP_CPU_AFFINITY=TRUE -x OMP_PROC_BIND=spread  --map-by core \
               -np 6     ./mpi_omp_hello.dbg >& LOG.np6.2
beaker:$ grep binding LOG.np6.2 | sort -k4
__kmp_affinity_set_init_mask: binding T#0 (PID:56387) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56388) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56389) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56390) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56391) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56392) to place 0
beaker:$

[naughtont3]

The --np 6 case creates the 6 places, one for each MPI rank per core, as expected.

[naughtont3]

lstopo-beaker.pdf