Implement a unidirectional variant of local attention

[rwth-i6]

Implement a unidirectional variant of local attention

[albertz]

The first epochs in LibriSpeech are very short. Train a bit longer, until epoch 20 or 30. It is ok if the initial loss (score) is high. It only matter whether it goes down after a while of training.

[smdshakeelhassan]

Thanks for the quick reply. The training is currently going on. Could you also please have a look at the config file I am using and suggest if I am going in the right direction as the training takes a long time and if the config file is wrong it will result in a lot of time wastage for me?

https://gist.github.com/smdshakeelhassan/da72c8d091983075f05d1f4575a8cfde

Will adding a lstm_pool layer help in this case?

[albertz]

You should leave the pool layers in, and similar as in the original config, i.e. fix the pool sizes.

[smdshakeelhassan]

Hi, I have added the pool layer and made some changes to the custom_construction_algo function. Could you please have a look at the config file and verify?

https://gist.github.com/smdshakeelhassan/900afc7a4c1eea63f36e3570d0fa6357

Thanks.

[albertz]

I don't see the difference to the original config custom_construction_algo. Maybe you can show a diff, or just describe what you changed?
But anyway, you should just try. I guess you need some tuning here. See also my comments in #41 which is very related to your question.

[smdshakeelhassan]

Thanks for your help.
Changes I made are:
In the network, instead of:
"lstm0_pool": {"class": "pool", "mode": "max", "padding": "same", "pool_size": (2,), "from": ["lstm0_fw","lstm0_bw"], "trainable": False},
Using
lstm0_pool": {"class": "pool", "mode": "max", "padding": "same", "pool_size": (2,), "from": ["lstm0_fw"], "trainable": False},

and just removed line related to lstmi_bw from custom_construction_algo

I will try tuning the parameters as you suggested in #41

[albertz]

Yes, looks good. Just try it. Or then tune it.

[smdshakeelhassan]

Thank you.

[albertz]

Can you maybe report your results? Does it work? What changes did you need exactly? What is the WER you get in the end?

[smdshakeelhassan]

I am still trying to tune the hyperparameters. Decreasing learning rate and increasing repetition steps are showing promising results. Will update if the model converges.