GPU OOM when pretraining

[hi-zhenyu]

GPU OOM when pretraining

[zdou0830]

Hello, you may need to tune per_gpu_batch_size based on your GPUs.

I set precision to 32 because using mixed-precision training can sometimes cause the NaN issue.

[hi-zhenyu]

Hi, thanks for your reply.

In the paper, you said it takes around 8 days for METER-SwinBASE and METER-CLIP-ViTBASE−16 for training 100k steps on 8 A100 GPUs. However, in my experiments, it obviously needs more time for training, especially the per_gpu_batch_size need to be reduced due to the OOM problem (<32).

So could you please give me some advice to speed up training like the paper statement (~8 days with 8 A100 GPUs).

[zdou0830]

Hi, I think you can try reducing the GPU memory usage by setting the precision to 16, using a smaller max_text_len, etc. Training the models for ~50k steps should also give you a decent performance.