GPU pinning

[mtazzari]

GPU pinning

[fredRos]

We decided to do leave this to the user's control via

void galario_use_gpu(int device_id);

[mtazzari]

The issue is still closed, but I 'd like to post this for reference.

I see a funny behaviour on my desktop regarding GPU pinning. The GPU ID at runtime appears different from that read in nvidia-smi. The reason boils down to the fact that:

nvidia-smi enumerates in PCI order while, by default, the CUDA driver and runtime APIs do not.

(from https://stackoverflow.com/questions/26123252/inconsistency-of-ids-between-nvidia-smi-l-and-cudevicegetname)

This is the output of nvidia-smi:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 730	   Off  | 0000:01:00.0     N/A |		  N/A |
| N/A   46C    P0    N/A /  N/A |     86MiB /  2000MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1060   Off  | 0000:02:00.0     Off |		  N/A |
|  0%   55C    P0    25W / 140W |      1MiB /  6072MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

where it seems that gpu 1 is GTX 1060, i.e. the GPGPU-dedicated one, and gpu 0 is GT 730, i.e. the default system graphics card where I don't want to run the CUDA code.

The issue is that ngpus = g_double.ngpus() sets ngpus=2, which is correct, but g_double.use_gpu(max(0, ngpus-1))=use_gpu(max(0, 1))=use_gpu(1) makes the script use GT 730 instead of GTX 1060.

The reason of the discrepancy is stated above in quotes. nvidia-smi docs suggest:

"It is recommended that users desiring consistencyuse either UUDI or PCI bus ID, since device enumeration ordering is not guaranteed to be consistent"

This behaviour can be overridden by setting CUDA environment variables, as described here:
http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars

I would leave this just as reference in the case we need to improve the GPU pinning implementation. At the moment, the user can quickly check which GPU is using with

watch -n0.1 nvidia-smi