Trouble reproducing efficientnet_b0 training

[huggingface]

Trouble reproducing efficientnet_b0 training

[rwightman]

• 12 min per epoch sounds fast compared to my 2-3 GPU setups :) but yes, compared to ResNets they are suprisingly slow given how much smaller they are
• So far, my best results usually come in after 300-400 epochs with this model family and this training scheme, if you start seeing the validation loss and error definitively stop moving downwards (or start moving upwards) before this your hparams could use adjustment
• Being 3-4 pts from the end result at 80-100 epochs isn't out of the ordinary, the last 2/3 to 3/4 of the training is a pretty slow and boring grind
• I would raise your batch size, try and get into the 128-256 per GPU range. Synchronized BN is not enable by default (and I wouldn't bother enabling it) so the BN stats are only from one of your GPU, not the total.
• Enable AMP mixed precision training (--amp and install APEX), this will also speed things up on a volta or turing card and allow you to target a larger batch / GPU if you can't hit the range above running at FP32.
• your ema decay should be similar to mine with your GPU batch of 32, if you get it up to 128-256 per-GPU, try 0.999 to 0.998 range
• turn the color jitter way down as mentioned in other issues about this ... I've used .05-.07, but if you wanted to follow the original, they didn't appear to have any color augmentation enabled until their recent 'AutoAugment' training scheme was used
• make sure the #kwargs['drop_connect_rate'] = 0.2 line in the gen_efficientnet.py for the model you're using is uncommented, I have no cmd line arg for this yet since it's specific to these models
• you don't need so many workers per GPU (-j 8), if you've got lots and lots of cores, sure, but otherwise 4-6 is usually fine

I'm going to close this issue as we don't need multiple hparam threads open, but please update this issue or another if you find something that works

[DrJimFan]

This is very helpful!! I'll try them out and let you know if it still can't reproduce. Thanks again.