RuntimeError: DataLoader worker (pid(s) 27447) exited unexpectedly

[skelemoa]

RuntimeError: DataLoader worker (pid(s) 27447) exited unexpectedly

[pranaygupta36]

The arguments in lines are batch size, num_workers for the dataloader. Maybe you can try reducing the num_workers from 8 and see if it works.

[divyanshusharma1709]

Hi @WurmD
It is possible that you might be running out of memory around the 1200th epoch. How much GPU memory do you have available?

[WurmD]

8GB

Usually when that's the case I see an actual RuntimeError: CUDA error: out of memory

I'll go monitor the GPU RAM usage now

EDIT: Hmm, lack of GPU RAM doesn't seem to be the issue,
It's not breaking now due to the same RuntimeError, and GPU RAM seems to remain steady at 795MiB / 8119MiB during the whole cycle (up to epoch 1700), and on the 2nd cycle it ups to 817MiB / 8119MiB