[WSL-2] lava-dl installation: BUILD FAILED - RuntimeError: CUDA error: no kernel image is available for execution on the device

[lava-nc]

[WSL-2] lava-dl installation: BUILD FAILED - RuntimeError: CUDA error: no kernel image is available for execution on the device

[bamsumit]

Hi @GoHeFa, it looks like your PyTorch installation does not support RTX3090 (cuda compute capability 8.6, i.e. sm 86). This is from your error message

NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

I am assuming you installed the latest PyTorch version. The solution is to build your PyTorch locally.

Before you do that, let's confirm the following things

• Are you able to use GPU with your torch?

import torch
x = torch.rand(100)
x = x.to(torch.device('cuda')

• What are the output of following commands?

python -c "import torch; print(torch.__version__)"
nvidia-smi | grep CUDA
nvcc --version | grep release
gcc --version

• I assume you are using insider build of windows to access GPU from WSL2. How did you verify that cuda is working?

[GoHeFa]

@bamsumit First of all, thanks for your response.
I rechecked the torch version compatibility with another project (pure python/PyTorch with GPU usage) I am working on.
With torch == 1.8.1 and torchvision == 0.9.1 successfully installed (with pip3), during runtime I get the error:

NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

After successfully installing the latest PyTorch version via: pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html I can successfully run my code without any errors or warnings.

Now to your tests:

• Are you able to use GPU with your torch?

import torch
x = torch.rand(100)
x = x.to(torch.device('cuda'))

Works fine without any errors with the latest PyTorch version installed (with the pip3 command depicted above). While with torch == 1.8.1 and torchvision == 0.9.1 I receive the error described above.

• What are the output of following commands?
  (Output received with the latest PyTorch version installed.)

(base) user@machine:~/miniconda3/lava-dl$ python -c "import torch; print(torch.__version__)"
1.10.0+cu113
(base) user@machine:~/miniconda3/lava-dl$ nvidia-smi | grep CUDA
| NVIDIA-SMI 510.00       Driver Version: 510.06       CUDA Version: 11.6     |
(base) user@machine:~/miniconda3/lava-dl$ nvcc --version | grep release
Cuda compilation tools, release 11.5, V11.5.119
(base) user@machine:~/miniconda3/lava-dl$ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

• I assume you are using insider build of windows to access GPU from WSL2. How did you verify that cuda is working?
  I am not using the insider built of Windows. My current Microsoft Windows 10 version is Microsoft Windows [Version 10.0.19044.1387].
  Furthermore, entering wsl -l -v into a PowerShell gives me:

  NAME            STATE           VERSION
* Ubuntu-20.04    Running         2

To verify that my CUDA installation is working I only did: (Honestly, hence describing it as "working" is not appropriate.)

nvidia-smi | grep CUDA
nvcc --version | grep release

Also, entering nvidia-smi in my Power Shell or in a WSL shell yields the expected output.

[bamsumit]

So are you able to run lava-dl with torch 1.10.0+cu113 or not?

[GoHeFa]

Unfortunately I am noch able to build lava-dl with torch 1.10.0+cu113 (while normal PyTorch GPU execution works fine).

(base) user@machine:~/miniconda3/lava-dl$ pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
Looking in links: https://download.pytorch.org/whl/cu113/torch_stable.html
Requirement already satisfied: torch==1.10.0+cu113 in /home/user/miniconda3/lib/python3.8/site-packages (1.10.0+cu113)
Requirement already satisfied: torchvision==0.11.1+cu113 in /home/user/miniconda3/lib/python3.8/site-packages (0.11.1+cu113)
Requirement already satisfied: torchaudio==0.10.0+cu113 in /home/user/miniconda3/lib/python3.8/site-packages (0.10.0+cu113)
Requirement already satisfied: typing-extensions in /home/user/miniconda3/lib/python3.8/site-packages (from torch==1.10.0+cu113) (4.0.0)
Requirement already satisfied: pillow!=8.3.0,>=5.3.0 in /home/user/miniconda3/lib/python3.8/site-packages (from torchvision==0.11.1+cu113) (8.4.0)
Requirement already satisfied: numpy in /home/user/miniconda3/lib/python3.8/site-packages (from torchvision==0.11.1+cu113) (1.21.2)
(base) user@machine:~/miniconda3/lava-dl$ pyb -E unit
PyBuilder version 0.13.3
Build started at 2021-12-08 18:13:13
------------------------------------------------------------
[INFO]  Installing or updating plugin "pypi:pybuilder_bandit, module name 'pybuilder_bandit'"
[INFO]  Processing plugin packages 'pybuilder_bandit' to be installed with {}
[INFO]  Activated environments: unit
[INFO]  Building lava-dl version 0.1.1
[INFO]  Executing build in /home/user/miniconda3/lava-dl
[INFO]  Going to execute tasks: analyze, publish
[INFO]  Processing plugin packages 'flake8~=3.7' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'pypandoc~=1.4' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'setuptools>=38.6.0' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'sphinx_rtd_theme' to be installed with {}
[INFO]  Processing plugin packages 'sphinx_tabs' to be installed with {}
[INFO]  Processing plugin packages 'twine>=1.15.0' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'unittest-xml-reporting~=3.0.4' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'wheel>=0.34.0' to be installed with {'upgrade': True}
[INFO]  Creating target 'build' VEnv in '/home/user/miniconda3/lava-dl/target/venv/build/cpython-3.8.12.final.0'
[INFO]  Processing dependency packages 'requirements.txt' to be installed with {}
[INFO]  Creating target 'test' VEnv in '/home/user/miniconda3/lava-dl/target/venv/test/cpython-3.8.12.final.0'
[INFO]  Processing dependency packages 'requirements.txt' to be installed with {}
[INFO]  Executing flake8 on project sources.
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /home/user/miniconda3/lava-dl/tests/lava
/home/user/miniconda3/lava-dl/target/venv/build/cpython-3.8.12.final.0/lib/python3.8/site-packages/torch/cuda/__init__.py:104: UserWarning:
NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
------------------------------------------------------------
BUILD FAILED - RuntimeError: CUDA error: no kernel image is available for execution on the device (/home/user/miniconda3/lava-dl/tests/lava/lib/dl/slayer/neuron/test_adrf_iz.py:45)
------------------------------------------------------------
Build finished at 2021-12-08 18:13:19
Build took 6 seconds (6421 ms)

[bamsumit]

I would try re-installing nvidia driver and nvcc compiler for cuda 11.3 and try again (currently you have them at 11.6 and 11.5). If that does not work, I am afraid you will have to build pytorch from source locally.

[mgkwill]

I would try re-installing nvidia driver and nvcc compiler for cuda 11.3 and try again (currently you have them at 11.6 and 11.5). If that does not work, I am afraid you will have to build pytorch from source locally.

Hi @GoHeFa did @bamsumit's suggestion fix the issue?

[GoHeFa]

@mgkwill Unfortunately, it's currently not possible for me to test the suggestion, hence I can't tell you.

[VishalPathak-GTRI]

Has anyone else had this issue? I'm having the exact same problem at the moment, here are some of the requested outputs:

[bamsumit]

@VishalPathak-GTRI are you also running it in WSL-2?

First thing I would check is to check if you can compile and run cuda code using nvcc.

[uslumt]

Hello everyone, I'm able to train pytorch model on gpu using cuda however with the same set up on LAVA blocks gives an error.
torch 1.10.1+cu111
torch-encoding 1.2.1
torchaudio 0.10.1+rocm4.1
torchsummary 1.5.1
torchvision 0.11.2+cu111

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85

NVIDIA-SMI 495.29.05 Driver Version: 495.29.05 CUDA Version: 11.5

error :
nvcc fatal : Unsupported gpu architecture 'compute_80'
ninja: build stopped: subcommand failed.

[bamsumit]

@uslumt , can you try updating your nvcc compiler to v 11.5?