Explicitly scaling up LocalCUDACluster() creates Dask workers that don't use all available GPUs.

[rapidsai]

Explicitly scaling up LocalCUDACluster() creates Dask workers that don't use all available GPUs.

[mrocklin]

There isn’t a great answer to your question currently. But if you wanted to do things manually you could skip LocalCUDACluster and create dask workers manually

CUDA_VISIBLE_DEVICES=0 dask-worker ... # use the dask-worker command line arguments as you like
CUDA_VISIBLE_DEVICES=1 dask-worker ... # use the dask-worker command line arguments as you like
dask-scheduler ...

Currently there is a bit of work to make things like LocalCUDACluster more configurable happening here: dask/distributed#2675

In general though, I have no idea what’s going to happen when mixing many processes and GPUs. Maybe this is ok?

[chinmaychandak]

So, I created workers manually. Now, I am able to use both GPUs; however, the throughput is lower than using a single GPU, because some workers go into an unresponsive event loop (and some even crash and fail to restart) either due to GIL issues or moving around large chunks of data.

Not sure why this isn't the case when using a single GPU. Any ideas as to where the problem might be?

[mrocklin]

Unfortunately not. It could be many things. Without more information I doubt I'll be able to help. Often at this stage the recommendation is to try to reduce down your situation to a minimal reproducible example.

[pentschev]

The problem reported seems largely due to communication bandwidth, which is being worked with UCX-Py, possibly similar to #216. Since this has gone stale anyway, I'm closing. Feel free to reopen if you can provide more details and an example.