specAugment policy and schedules

[rwth-i6]

specAugment policy and schedules

[albertz]

We already have variations of that, where we also play around with scheduling of SpecAugment.
E.g. see Switchboard base2.conv2l.specaug4a.
@papar22 and @ZhouW321 also have some more variations, which we will upload soon to the repo.

Note that the random_mask in that config already runs multiple times (which is stochastically sampled). That are the options min_num and max_num. If you want that the mask is always runs exactly 3 times, just set min_num=3, max_num=3.

Yes sure, you can play around with learning rate warm-up as well. My experience however is that increasing usually will not help.

Reducing the LR decay helps when you want to increase your overall training time, i.e. train more epochs. And training longer usually helps. When you look at this original SpecAugment paper, you will see that they effectively train much longer than we do.

[akshatdewan]

Thanks for your answer, Albert!

I am sorry for a possibly naive question, but in the config example you mention above, the newbob_learning_rate_decay is 0.7.

My understanding is: LR_epoch_t+1=decay*LR_epoch_t. So if I am starting from a baseline model trained for 12.5 epochs using newbob_learning_rate_decay= 0.9, and I want to train another model for say 25 epochs, I should increase newbob_learning_rate_decay to say 0.95 instead of reducing it to 0.7, right?

[albertz]

Yes sure.