Audio coverage issues

[rwth-i6]

Audio coverage issues

[albertz]

Do you have statistics about the sequence length, esp when this happens? Is it mostly for long sequences? Am Di., 13. Nov. 2018, 17:04 hat Akshat Dewan <[email protected]> geschrieben:

…

Hi, I trained models on around 1500h of audio data (Librispeech + WIPO proprietary) and obtain good performance (around 18% on proprietary test set). But sometimes we encounter an issue when the coverage of the audio is very low and lot of text (5-6 seconds) is missing. I just wanted to ask if anyone of you has encountered similar challenges. Best Akshat — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#11>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADm_HUM59vv2GxSVDgqiYkKeT8I5A3qks5uuplFgaJpZM4YbTbk> .

[akshatdewan]

The test sequences, like training sequences, are mostly between 1-15 sec long. (see histogram)

I do not have any statistic on the problematic segments though. However, I do have a feeling that it occurs for long segments. Do you think a solution could be to just reduce the length of the test sequences?

[albertz]

So, to make it more clean again: You have some sequences which are longer (e.g. 14 secs), and they contain some text (about 6 secs) which you don't have in your reference transcription?
Noisy/incorrect training transcriptions can definitely be problematic for stable training.
Also, I observed that the attention model kind of learns like a linear alignment initially, and to detect silence. So any sequence which are far away from this pattern will be hard.
Some sort of curriculum learning, to exclude such bad sequences in the beginning of the training (first 20 epochs or so), will probably help to some extend.

[akshatdewan]

Sorry, I guess I was not clear - for some longer test segments (e.g. 14.9 sec), the predicted sequence is missing a lot of text (5-6 seconds worth) as compared to the reference.

"Also, I observed that the attention model kind of learns like a linear alignment initially, and to detect silence. So any sequence which are far away from this pattern will be hard."
I thought that input and output sequences in S2T are always linearly aligned. Am I missing something here?

"Some sort of curriculum learning, to exclude such bad sequences in the beginning of the training (first 20 epochs or so), will probably help to some extend."
I currently do not have a mechanism to classify training segments as clean or noisy but I plan on working on it.

[albertz]

I think you mean monotonic. Of course that is true for speech recognition (but e.g. not for translation).
But linear is not true. Linear means that e.g. the attention for the first output label should be at t=0, and for the last output label the attention should be at t=T, and for all other output labels, the attention should focus exactly linearly interpolated in between.

[akshatdewan]

Thanks! I think I understand it better now. I will run some more experiments to see what can I do. Perhaps if I try to limit non-speech in my test segments, the "linearity" condition might get satisfied and coverage might improve.