NMNIST Training Notebook CUDA Error

[lava-nc]

NMNIST Training Notebook CUDA Error

[bamsumit]

Hi @VishalPathak-GTRI wanted to check if you are able to run all the unittests.

It looks like you don't have cuda enabled machne. If you don't have GPU capable machine, you would want to set the code to use CPU instead.

device = torch.device('cpu')

in block 4 of the notebook.

[VishalPathak-GTRI]

So I actually am on a SLURM based GPU manager using CentOS; I have confirmed torch installation for GPU usage with torch.cuda.is_available, and with torch.cuda.get_device_name(0). However despite this I have issues with the notebook. There seems to be another post that I also replied on that is having similar issues.

[bamsumit]

Try this piece of code in python interpreter in your virtual environment and see if you get the error:

import torch
torch.round(torch.arange(10) / 2)
device = torch.device('cuda')
torch.round(torch.arange(10).to(device) / 2)

[VishalPathak-GTRI]

Alright, so I'm still having some issues but I think I can provide more information in order to get some advice:

That image is of the unittests for lava passing successfully.

That image is of the unittests for lava-dl failing..

This is the output of the commands that you had requested I perform.

What would you suggest?

[bamsumit]

Okay, you have a problem using GPU from torch. Looks like there is a cuda version mismatch between your torch installation and Nvidia drivers.

Let's see the output of the following commands

python -c "import torch; print(torch.__version__)"
nvidia-smi | grep CUDA
nvcc --version

[VishalPathak-GTRI]

Happy to oblige, but I did just figure something out first; I am accessing GPUs off a remote cluster. I have been consistently working on a TeslaK20Xm and I tried to grab a GeForceGTX1080Ti and the error that I linked above has disappeared.

The above image shows the accuracy of the unittesting. Strangely, this the errors all state issues with a module not being found? Additionally, the below image shows the error in jupyter itself:

This error occurs immediately upon executing the first cell under "Training Loop" in the NMNIST tutorial.

As far as the outputs of the commands you requested the below image should suffice:

[bamsumit]

Okay there is quite a bit of mismatch in your cuda versions

• Your torch installation is for cuda 10.2
• Your nvidia driver is for cuda 11.2
  • This is the reason you are getting no kernel image available as there is no runtime for cuda 10.2 in torch
• Your nvcc compiler is for cuda 10
  • You will encounter problems when you use lava-dl compiled codes

Bottom line: the cuda versions of all three must be the same

[franzhd]

Hello, i managed to make cuda the same version as nvcc as i show in the screen.

But the test still give me the same error "CUDA error: no kernel image is available for execution on the device" and the same test coverage.

how can i resolve this problem? thanks for your help.
P.S. i tried also to update the torch version of the virtual env to the latest one but it still don't work.

[ahmetakman]

You can look at the #55 for some similar issue and investigate my case.
There are two things I would like to mention. First I think there is no PyTorch version for CUDA 11.6 the latest one is for 11.3.
Secondly I strongly disagree with @bamsumit . Nvidia driver cuda runtime version is only about compatibility need not to be same with torch cuda version and nvcc compiler version. However installed pytorch cuda version and nvcc compiler should be same and nvidia driver runtime cuda version should be equal or higher than them.
I have struggled with a similar issue past weeks and my suggestion would be holding nvidia driver like that and installing cuda 11.3 and latest stable pytorch with 11.3 .
cheers,

[bamsumit]

@VishalPathak-GTRI as I said you want to have your cuda versions for torch, nvcc and nvidia drivers at the same version if you can. You can sometimes make things work with different versions, but that's not always guaranteed. You can dig into nvidia cuda compatibility documentation for details.

As @ahmetakman pointed out, there is no official torch build for cuda versions later than Cuda 11.3 so I am guessing your torch is 1.11.0+cu113. So you at least

• change your nvcc to cuda 11.3

I have working configurations with nvidia driver at cuda 11.6. So you can try not to change update your driver and see if it works.

[bamsumit]

@VishalPathak-GTRI is this issue resolved?