Batchnorm vs dropout

[jcjohnson]

Batchnorm vs dropout

[jcjohnson]

I didn't do any super controlled experiments, but deeper models with batchnorm do seem to train faster.

[AlekzNet]

I made some comparisons for the 3x128 model (16MB corpus, seq_length=400, batch_size=200, 200 iteration per epoch):

The peaks at iteration 400 are related to the learning rate decay (0.92) for the first 3 test runs, I removed the decay for all subsequent runs.

The bottom line - the batch normalization DOES help. E.g. without the BN, the highest possible LR was 0.01, (0.05 with the BN).

Fun! ;)

[jcjohnson]

Those are some beautiful curves - thanks for sharing! It's interesting that the models without BN show a plateau at the beginning - that suggests to me that maybe the initialization is not correct.