Facing issues with sequential long form decoding for long audio

[agandhigoto]

Facing issues with sequential long form decoding for long audio

[zucchini-nlp]

@agandhigoto I am not a whisper expert but after exploring the codebase a bit, this is what I found.

1. The error raises due to forced_decoder_ids in the distil-whisper model config. It was not failing in openai-whisper because it does not have it by default. I opened a PR to fix it, until it gets merged you can use this code as a workaround

result = model.generate(**inputs, condition_on_prev_tokens=False, temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0), logprob_threshold=-1.0, compression_ratio_threshold=1.35, forced_decoder_ids=None, return_timestamps=True)

2. Whisper longform in pipeline cannot do batched generation right now. You can still pass multiple samples and have batch size=1 by default, in which case the inputs will be processed one by one sequentially. To use longform whisper generation with batches more than 1, you can instantiate a WhisperForConditionalGeneration

from transformers import WhisperForConditionalGeneration, AutoProcessor

processor = AutoProcessor.from_pretrained("distil-whisper/distil-large-v2")
model = WhisperForConditionalGeneration.from_pretrained("distil-whisper/distil-large-v2").to("cuda:0")

inputs = processor(batch_of_long_audios, return_tensors="pt", truncation=False, padding=True, return_attention_mask=True, sampling_rate=16_000)
inputs = inputs.to("cuda:0")

result = model.generate(**inputs, return_timestamps=True)
decoded = processor.batch_decode(result, skip_special_tokens=True)
print(decoded)

3. This statement you cited is about comparing speed-ups when using batch_size 1 vs more than 1. And as stated in (2) batch size>1 is not possible for pipelines. Please use the code above for batched generation. I tried to measure time with a toy sample of 50 audio each 40-50 seconds with openai/whisper-large-v2 and validated there is speed up for higher batch size.

Hope this helps to understand how to use batched longform generation in Whisper 🤗

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.