If I am training on a SINGLE GPU, should this "--dist-backend 'gloo'" argument be added to the command?

[HassanBinHaroon]

If I am training on a SINGLE GPU, should this "--dist-backend 'gloo'" argument be added to the command?

[Jaiaid]

@HassanBinHaroon
yes
it should be --dist-backend gloo (without the quote). You do not need to give quote for cmd line args
but if in your system nccl version <2.5. It should be fine to use nccl

[HassanBinHaroon]

@HassanBinHaroon yes it should be --dist-backend gloo (without the quote). You do not need to give quote for cmd line args but if in your system nccl version <2.5. It should be fine to use nccl

@Jaiaid Thanks!

[Jaiaid]

@HassanBinHaroon
yes they should come with pytorch
yes, it should otherwise throw an error from NCCL (something like internal check error), to avoid that you have to do --dist-backend gloo
BTW have you tried actually using NCCL to run your code?

[HassanBinHaroon]

@Jaiaid Yes, I have tried running code --dist-backend nccl. It logs the user warning that (I think) you recently added and code executes smoothly BTW.

[Jaiaid]

Thank you for the information.

[HassanBinHaroon]

@Jaiaid I am using just SINGLE GPU for training. I am not forcefully adjusting the rank and It's -1 by default. The command that I have been using is "python main.py -b 512 --dist-backend gloo -a resnet18 imagenet/"

[HassanBinHaroon]

@HassanBinHaroon If you use "python main.py -b 512 --dist-backend nccl -a resnet18 imagenet/" does it run smoothly?

@Jaiaid Yes, it absolutely runs smoothly.

[HassanBinHaroon]

@HassanBinHaroon yes it should be --dist-backend gloo (without the quote). You do not need to give quote for cmd line args but if in your system nccl version <2.5. It should be fine to use nccl

@Jaiaid Does the nccl and gloo comes with PyTorch?
If yes, then how to check their availability and version?

I am checking the availability using the following commands. Is it the right procedure?

"import torch.distributed as dist

print(dist.is_available()) # Should print True if distributed is available
print(dist.is_nccl_available()) # Should print True if NCCL is available
print(dist.is_gloo_available()) # Should print True if Gloo is available
"
Moreover, I am checking the nccl version using the following command, Again, please enlighten is it correct procedure?

"print(torch.cuda.nccl.version)"

The output is (2,19,3)

Does it mean that I have to be compulsorily using --dist-backend gloo?

Please elaborate, Thanks!

[Jaiaid]

@HassanBinHaroon
If you are using one rank in one GPU than nccl backend should be fine but if there are multiple rank using a single GPU than it will be an issue. I guess I should improve the message.
In your case, are you using multiple rank/process in a single GPU machine?

[Jaiaid]

@HassanBinHaroon
If you use "python main.py -b 512 --dist-backend nccl -a resnet18 imagenet/" does it run smoothly?