Problem with conv4 on gpu

[andreasdominik]

Problem with conv4 on gpu

[andreasdominik]

... maybe I need to say in addition:
normal multiplication of CuArrays in the gpu works fine; i.e. cuda is functional.

[andreasdominik]

O.K. - CUDA.cached_memory() is only available on driver >= 11.3
That's it.
We try to update - and get rid of tensorflow that cannot handle the new driver versions^^
Srry for opening and closing the issue - but so it's alt least documented for the rest of the world...

[andreasdominik]

closed but it becomes more weird:
I found a docker container on the server with Cuda 11.0 - and it works:

using Knet
smi = `nvidia-smi`; run(smi)
x = rand(Float32, 24,24,3,4) |> CuArray
w = param(5,5,3,8)
size(conv4(w,x))

Wed Jun  1 13:28:34 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.191.01   Driver Version: 450.191.01   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  Off  | 00000000:03:00.0 Off |                    0 |
| N/A   29C    P0    34W / 250W |  19301MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

(20, 20, 8, 4)

[andreasdominik]

indeed - the longer I try to find out what's going on here, I get more confused; maybe it is better to re-open it..