2 GPU error when training

[yulunzhang]

2 GPU error when training

[yulunzhang]

Hi,
I re-ran the code with multi-GPU (2, 3, or 4 GPUs) and they all passed. I used anaconda to install pytorch0.4.0. You can try with this instead of Docker.
However, I didn't use Docker to train or test the code. It seems there're some problems in /proc/mounts. If you still want to use Docker to run the code, you may refer to those links to solve your problem.
https://stackoverflow.com/questions/46138549/docker-openmpi-and-unexpected-end-of-proc-mounts-line
http://www.cloudypoint.com/Tutorials/discussion/docker-solved-docker-openmpi-and-unexpected-end-of-procmounts-line/

[KindleHe]

Thanks for your good suggestions！I will try to figure it out. I have started training 2x model with 1 P40 GPU，but I found 1000 epochs needs about 20 days to finish，could you please tell more details about traning time and corresponding gpu numbers？ By the way，could I reproduce the paper’ results with you training command without any other training tricks？（Your work is really impressive，I will dedicate to super-resolution studying and hope to generate great works as yours）

[yulunzhang]

Hi,
I use one Titan Xp for each model. It takes about 6 days to reach 1000 epochs. It seems that multi-gpu training cannot boost the training obviously. Each training image is converted to .npy file to save training time. I also moved the training data to SSD, which can also boost the training.
No extra tricks were used, the results in the paper can be reproduced with my provided training scripts. Some people have reproduced the results with the code.

[KindleHe]

I see you offer some script for data converting and processing, but as a new pytorch learner, I do quite not understand how the training images being converted to .npy files?

[yulunzhang]

Hi,
Please see RCAN_TrainCode/code/data/srdata.py lines 30 to 50.

[KindleHe]

Thanks, it means that I just need to python main.py --model RCAN --save RCAN_BIX2_G10R20P48 --scale 2 --n_resgroups 10 --n_resblocks 20 --n_feats 64 --reset --chop --save_results --print_model --patch_size 96 command and the code will automatically run to the line 30 to 50 to finish the .npy generating process, am I right?

[yulunzhang]

Yes, you are right.

[swz30]

Hi Yulun,
You mentioned in the above comments that "multi-gpu training cannot boost the training obviously". Why is that?

Also, could you please tell how did you end up choosing RG=10 and RCAB=20 for your model? Can I reduce any blocks to train the model faster without compromising accuracy?

Thanks.