RuntimeError: CUDA out of memory. Tried to allocate 121.04 GiB (GPU 0; 47.54 GiB total capacity; 2.57 GiB already allocated; 13.33 GiB free; 32.50 GiB reserved in total by PyTorch)

[WesLee88524]

RuntimeError: CUDA out of memory. Tried to allocate 121.04 GiB (GPU 0; 47.54 GiB total capacity; 2.57 GiB already allocated; 13.33 GiB free; 32.50 GiB reserved in total by PyTorch)

[ocetintas]

Hi, you are simply running out of memory in your GPU. Maybe try to reduce the batch size with --num_batch, or run SUSHI with a fewer number of frames e.g. 128. This should drop the performance slightly but you should still easily reach state of the art results.

[WesLee88524]

I change the number of frames to 128,it worked!

But I am still wondering why I have 40GB of memory but it appears the error of out of memory.

[WesLee88524]

Hi, what kind of GPU you used to train? How long will it trains last? I have trained it on a RTX A6000 GPU more than 2days ,is it right?

[ocetintas]

We used an RTX8000. Depending on the parameters, training lasts between 12 hours to 2 days.