Feedback on the issue of ema coefficient.

[PeiqinZhuang]

Feedback on the issue of ema coefficient.

[huang-ziyuan]

Thanks for the feedback. Could you specify which model/config are you referring to?

[PeiqinZhuang]

Please refer to the following link. (

TAdaConv/configs/pool/base.yaml

Line 65 in 75b7839

ENABLE: false

) As mentioned, all configure files are based on the base configure file. However, the ema mode is turned off in the base configure file. If we want to execute the setting of tadaformer_b16_ssv2_16f, it is supposed to explicitly turn on the ema mode. BTW, the ema decay coefficient is different from what you reported in your work, e.g. 0.99996 v.s. 0.9996. Maybe you can check it later.

[huang-ziyuan]

Yes, if you wish to turn on EMA, you will have to manually enable ema and set the ema decay factor in your config file to run. We indeed used EMA during training TAdaFormer, but we actually observe negligible performance differences for the ema models and the original model. Therefore, turning on ema is not a must for training TAdaFormers.

[PeiqinZhuang]

As mentioned in your paper, turning on the ema mode may prevent the model from over-fitting problems. I am curious how important it is for training.

[huang-ziyuan]

It affects the top-1 accuracy by 0.2~0.4 for large models.