Question about the training speed

[jiahuiyu]

Question about the training speed

[JiahuiYu]

I have tried 1080 Ti or P100 for training, the speed is around 0.4 batches/sec.
Regarding your case, several options are available:

1. Use more GPUs for parallel training, a simple modification is required. Please refer to #14
2. Use low-resolution images will significantly increase speed. E.g., change IMG_SHAPES: [256, 256, 3] to [64, 64, 3] in inpaint.yml.

[zhiweige]

okay, thanks for your suggestions. I will try it.

[tabsun]

I have changed Trainer to MultiGPUTrainer and use 4 x 1080 Ti for training, but the training speed seems slower than single GPU:(
|----------------------------| 0.10%, 4953/4947597 sec. train epoch 1, iter 10/10000, loss 1.013579, 0.00 batches
|----------------------------| 0.20%, 10009/5046506 sec. train epoch 1, iter 20/10000, loss 0.890207, 0.00 batche
|----------------------------| 0.30%, 15481/5455784 sec. train epoch 1, iter 30/10000, loss 0.796239, 0.00 batche
|----------------------------| 0.40%, 20848/5345027 sec. train epoch 1, iter 40/10000, loss 0.757341, 0.00 batche
|----------------------------| 0.50%, 26577/5700523 sec. train epoch 1, iter 50/10000, loss 0.715168, 0.00 batche
|----------------------------| 0.60%, 32215/5603806 sec. train epoch 1, iter 60/10000, loss 0.686893, 0.00 batche
|----------------------------| 0.70%, 37648/5395326 sec. train epoch 1, iter 70/10000, loss 0.657164, 0.00 batche
|----------------------------| 0.80%, 43417/5722711 sec. train epoch 1, iter 80/10000, loss 0.638888, 0.00 batches/sec.

Is there any point I have ignored? When using single GPU, it can run a little faster at 0.01 batches/sec.
Thank you

[JiahuiYu]

@tabsun Hi, thanks for your interest in our work first.

There must be some problems here, and here are several potential bugs:

1. Your disk is very slow, which makes the data loading part blocked.
2. Check your GPU utilization and make sure that all of them are running on different GPUs.

[tabsun]

@JiahuiYu Thanks for your advices. I have tried to move my training data into the same disk as my code. The speed has no change. What do you mean by GPU utilization ? I have checked all 4 GPU memory has been occupied, is this enough? Maybe I should deep into the training code to verify what has blocked the process.

[JiahuiYu]

@tabsun You can view your GPU utilization by nvidia-smi. In most cases, the GPU utilization should be above 90%.