Strategies to avoid overfitting

[megvii-model]

Strategies to avoid overfitting

[nmaac]

Sorry, I did not save the schedule, I was changing lr manually each time accuracy stopped improving. I started from 1e-3 and dropped it to 1e-6 at the very end...

Seems to be tedious and time-consuming.. Just use the simple linear decay schedule :)

[nmaac]

Hi @1243France , if you meet overfitting problem, please try:

• more augmentation and regularization;
• use less FReLU (example) which also shows great improvement and avoid overfitting problem as well.

[nekitmm]

@1243France, I was able to reproduce the model in pytorch. Please see the repo:

https://github.com/nekitmm/FunnelAct_Pytorch

Training Acc@1 was about 81.3%, so looks like you overfitted your model.

[1243France]

@1243France, I was able to reproduce the model in pytorch. Please see the repo:

https://github.com/nekitmm/FunnelAct_Pytorch

Training Acc@1 was about 81.3%, so looks like you overfitted your model.

Hi @nekitmm , could you offer your lr_scheduler settings? Thanks for your help.

[nekitmm]

Sorry, I did not save the schedule, I was changing lr manually each time accuracy stopped improving. I started from 1e-3 and dropped it to 1e-6 at the very end...

[nmaac]

Hi @1243France , if you meet overfitting problem, please try:

• more augmentation and regularization;
• use less FReLU (example) which also shows great improvement and avoid overfitting problem as well.

@1243France , I reviewed your code briefly, please note as we presented in both the paper and code, we do not apply frelu to the last stage to avoid overfitting. If you do not use this strategy, please follow the strategy I suggested above.