Bug in whisper word-level timestamps (`tokenizer._decode_asr`)

[xenova]

Bug in whisper word-level timestamps (`tokenizer._decode_asr`)

[xenova]

To help with debugging, here are the decoded outputs of each chunk:

for output in model_outputs:
  print(tokenizer.batch_decode(output['tokens']))

["<|startoftranscript|><|notimestamps|> DO IT! Just DO IT! Don't let your dreams be dreams. Yesterday, you said tomorrow, so just DO IT! MAKE YOUR DRIMS! CONTRO! JUST DO IT! Some people dream success while you're gonna wake up and work hard at it. Nothing is impossible.<|endoftext|>"]
["<|startoftranscript|><|notimestamps|> Some people dream success while you're gonna wake up and work hard at it. Nothing is impossible. You should get to the point where anyone else would quit and you're not gonna stop there. No, what are you waiting for? Do it! Just do it! Yes, you can! Just do it!<|endoftext|>"]
['<|startoftranscript|><|notimestamps|> Just do it! Yes you can! Just do it! If your tire is starting over, stop giving up.<|endoftext|>']

Indeed, the duplicated phrasing is at the word boundaries, so we can see where the algorithm messes up.

[kamilakesbi]

Thanks for pointing out this issue @xenova !

It wonder if the problem comes from the chunking boundaries in model_outputs and not from how word level timestamps are converted and concatenated in tokenizer._decode_asr, but I need to dive a bit here.

Could you share the snippet you used to generate model_outputs so I can dive into what's going wrong here?

Thanks!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.