GPU Not Available using PyTorch in azureml_py38_PT_and_TF kernel on DSVM

[azure]

GPU Not Available using PyTorch in azureml_py38_PT_and_TF kernel on DSVM

[leestott]

What are you trying to do

I'm trying to install pytorch with GPU acceleration.

What command are using > What DSVM version are you running and OS
Ubuntu 20.04 DSVM on a Standard NV32as v4 (East US 2)

Error encountered
Typing in python :

import torch
torch.cuda.is_available() # evaluates to False

Expected behaviour
should evaluate to true (it did on the nvidia machines of the NS family,
but they are all unavailable). So I started troubleshooting :

$ lspci
0299:00:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
[AMD/ATI] Vega 10 [Radeon Instinct MI25 MxGPU]

and there is no /dev/kfd and lsmod does not show any amd related modules
: the amdgpu drivers are not loaded. In recent kernel versions they
should be bundled by default, but if they are not, one can install them
using these instructions :
https://amdgpu-install.readthedocs.io/en/latest/install-prereq.html#installing-the-installer-package

which boils down to downloading the right package and using the
following commands :

sudo apt --fix-broken install ./amdgpu-install_21.50.2.50002-1_all.deb
sudo apt update
sudo amdgpu-install --no-32
sudo modprobe amdgpu

I then get the following error message indefinitely and the last command
hangs :

...
[drm:amdgpu_virt_read_pf2vf_data [amdgpu]] ERROR invalid pf2vf message
[drm:amdgpu_virt_read_pf2vf_data [amdgpu]] ERROR invalid pf2vf message
[drm:amdgpu_virt_read_pf2vf_data [amdgpu]] ERROR invalid pf2vf message
[drm:amdgpu_virt_read_pf2vf_data [amdgpu]] ERROR invalid pf2vf message
[drm:amdgpu_virt_read_pf2vf_data [amdgpu]] ERROR invalid pf2vf message
[drm:amdgpu_virt_read_pf2vf_data [amdgpu]] ERROR invalid pf2vf message
[drm:amdgpu_virt_read_pf2vf_data [amdgpu]] ERROR invalid pf2vf message
...

[riow1983]

Hi,
I tried again to create my VM from the latest version of DSVM image (detailed below) and found the issue above has been solved. I'll close this issue. Thanks.

"imageReference": {
      "communityGalleryImageId": null,
      "exactVersion": "22.05.11",
      "id": null,
      "offer": "ubuntu-2004",
      "publisher": "microsoft-dsvm",
      "sharedGalleryImageId": null,
      "sku": "2004-gen2",
      "version": "latest"
    },