distribute training with horovod cost more time every epoch

[facebookresearch]

distribute training with horovod cost more time every epoch

[KaimingHe]

The results in the paper are based on TPU training.

Using horovod is beyond the scope of this repo.

[HobbitLong]

bandwidth: 10Gbps machine: 8*v100 nodes: 8 dateset: imagenet

If training with resnet50, the training time is same as paper(about 3min every epoch), but when training with vit_base_patch16, the training time is 10min(the peak receive network is 7Gbps). After use method in link, the training time is 6min(the peak receive network is 4.4Gbps) I wonder if the bandwidth is bottleneck for the mae training. What bandwidth do you use in the paper?

@guangdongliang, I run into similar situation and investigated it a bit. I believe network bandwidth is a bottleneck here. The larger the network, the heavier the communication delay between nodes. You may consider using a cluster with infiniband, or try Google cloud with Google virtual NIC.