kokkos/hpcbind

Binding utilities used for MPI, OpenMP and GPUs

THIS REPOSITORY HAS BEEN DEPRECATED

Use bin/hpcbind from kokkos/kokkos instead

hpcbind's Issues

Need review with testbed and production environment stakeholders

Make OMP_NESTED false by default

Since Kokkos 'usually' does not use this feature, it makes sense to disable it by default, so that a distract user does not leave it on in OpenMP by mistake. I'm not 100% sure, but there may be some performance hit in leaving it on. Either way, it makes more sense to have a feature not commonly used off, unless the user specifically asks for it.

Long setup time due to hwloc. Add store/load cpuset configuration

hpcbind can take several seconds to inspect the architecture and get the final cpuset desired. For a script that launches several simulations with the same cpuset configuration, this can cause a significant overhead (especially if simulations themselves are "fast").

I would suggest adding the capability to store the HPCBIND_* variables to file, and load them on a subsequent run. The idea is that loading the env variables from file should be faster than continuously calling hwloc-ls.

Nodes without GPUs may have nvidia-smi installed leading to incorrect GPU settings

Some nodes may have nvidia-smi installed without having GPUs or GPU driver installed.
In this case nvidia-smi will be found (sets HPCBIND_VISIBLE_GPUS) and the
command used to find the number of GPUs outputs:

"NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running."

and this registers as 2 lines and hence sets NUM_GPUS to 2 (because it is 2 lines long).
This may be harmless...

GPUs are not assigned correctly

I have CUDA_VISIBLE_DEVICES=0,1,2,3 in the shell. Running:

mpiexec -np 4  hpcbind --distribute=4 --output-prefix=test --output-mode=all --lstopo -- ./test

shows basically the same GPU setting for each test.hpcbind.[0-3]:

[HPCBIND]
HPCBIND_HAS_HWLOC=1
HPCBIND_HAS_NVIDIA=1
HPCBIND_HWLOC_CPUSET=0x00001001
HPCBIND_HWLOC_DISTRIBUTE=4
HPCBIND_HWLOC_DISTRIBUTE_PARTITION=0
HPCBIND_HWLOC_PARENT_CPUSET=0x00555555
HPCBIND_HWLOC_PROC_BIND=all
HPCBIND_HWLOC_VERSION=2.2.0
HPCBIND_NUM_CORES=1
HPCBIND_NUM_NUMAS=1
HPCBIND_NUM_PUS=2
HPCBIND_NUM_SOCKETS=1
HPCBIND_NVIDIA_ENABLE_GPU_MAPPING=1
HPCBIND_NVIDIA_VISIBLE_GPUS=0,1,2,3
HPCBIND_OPENMP_RATIO=1/1
HPCBIND_OPENMP_VERSION=4.0
HPCBIND_QUEUE_MAPPING=0
HPCBIND_QUEUE_NAME=openmpi
HPCBIND_QUEUE_RANK=0
HPCBIND_QUEUE_SIZE=4
[HWLOC]
[CUDA]
CUDA_HOME=/home/aznb/spack/var/spack/environments/trilinos-cudacc61/.spack-env/view
CUDA_LAUNCH_BLOCKING=1
CUDA_VISIBLE_DEVICES=0,1,2,3
[OPENMP]
OMP_NESTED=false
OMP_NUM_THREADS=2
OMP_PLACES=threads
OMP_PROC_BIND=spread
[GOMP] (gcc, g++, and gfortran)
[KMP] (icc, icpc, and ifort)
[XLSMPOPTS] (xlc, xlc++, and xlf)
[BINDINGS]
Machine (128GB total)
  Package L#0
    NUMANode L#0 (P#0 64GB)
    L3 L#0 (15MB) + L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
      PU L#0 (P#0)
      PU L#1 (P#12)
  Package L#1
    NUMANode L#1 (P#1 64GB)

and my app './test' report the same GPU id for all processes.
However, if I run

mpiexec -np 4  hpcbind --distribute=4 --output-prefix=test --output-mode=all --lstopo -- ./test --kokkos-num-devices=4

then each process reports a unique GPU id

Deprecated --openmp-percent is actually broken

If I specify openmp-percent instead of openmp-ratio, I get the error message

/home/lbertag/bin/hpcbind: line 240: 100: command not found

It looks like the syntax for dividing 100 by openmp-percent value is wrong. But perhaps openmp-percent can be removed altogether?