andyj1/pytorch-multigpu

Multi GPU Training Code for Deep Learning with PyTorch

pytorch-multigpu

Multi GPU Training Code for Deep Learning with PyTorch. Train PyramidNet for CIFAR10 classification task. This code is for comparing several ways of multi-GPU training.

Requirement

• Python 3
• PyTorch 1.0.0+
• TorchVision
• TensorboardX

Usage

single gpu

cd single_gpu
python train.py 

DataParallel

cd data_parallel
python train.py --gpu_devices 0 1 2 3 --batch_size 768

DistributedDataParallel

cd dist_parallel
python train.py --gpu_device 0 1 2 3 --batch_size 768

Performance

single gpu

• batch size: 240
• batch time: 6s
• training time: 22 min
• gpu util: 99 %
• gpu memory: 10 G

DataParallel(4 k80)

• batch size: 768
• batch time: 5s
• training time: 5 min
• gpu util: 99 %
• gpu memory: 10 G * 4