question about "samples_per_bn"

[valencebond]

question about "samples_per_bn"

[yxgeee]

When samples_per_bn is smaller than samples_per_gpu, sync bn won't work so it is disabled. And note that the actual samples_per_bn should be at least the same as samples_per_gpu (samples_per_bn=N x samples_per_gpu, N>=1), so the actual samples_per_bn will be the same as samples_per_gpu by default if samples_per_bn<samples_per_gpu.

In source-domain pre-training, the best setup is to train with a batch size of 64 and global sync bn (samples_per_bn=64). And in target-domain fine-tuning on 4 GPUs, the best setup is to train with a batch size of 64 and no sync bn (samples_per_bn=16, 16x4=64). However, when conducting experiments on 8 GPUs (8 batch_size on each GPU), sync bn will be activated since two GPUs need to sync their BNs to perform samples_per_bn=16.

[valencebond]

hi，@yxgeee, thanks for your detailed explanation, but why in target-domain fine-tuning, the best setup is to train with no sync bn. i am not familiar with domain adaptation, but this is against common practice in general training CNN.

[yxgeee]

Such an optimal setup (16 samples for each BN and 64 samples in a mini-batch) was found empirically, as the re-ID dataset is sensitive to the number of batch size.