MNIST slowdown when MultiprocessIterator is used

[keisukefukuda]

MNIST slowdown when MultiprocessIterator is used

[keisukefukuda]

Without multiprocessing.set_start_method call, the performance is somewhat middle of them.

$ mpiexec -n 1 python train_mnist.py -e 10 -g -i m
==========================================
Num process (COMM_WORLD): 1
Using GPUs
Using hierarchical communicator
Num unit: 1000
Num Minibatch-size: 100
Num epoch: 10
==========================================
epoch       main/loss   validation/main/loss  main/accuracy  validation/main/accuracy  elapsed_time
1           0.195358    0.097223              0.941534       0.968                     26.5627
2           0.0735971   0.0708732             0.976983       0.9774                    52.2765
3           0.0475518   0.0635201             0.985315       0.9802                    77.8528
4           0.0357444   0.0772405             0.988282       0.9778                    103.57
5           0.0284912   0.0697874             0.990615       0.9797                    129.152
6           0.0229091   0.0946116             0.992781       0.9771                    154.747
7           0.0222558   0.0911476             0.992665       0.9771                    180.222
8           0.0178596   0.0967532             0.993715       0.9778                    206.053
9           0.0173879   0.0904732             0.994665       0.9805                    231.692
10          0.012329    0.0826058             0.996299       0.9806                    257.238