Whisper - list index out of range with word level timestamps

[maxkvbn]

Whisper - list index out of range with word level timestamps

[amyeroberts]

cc @sanchit-gandhi @kamilakesbi

[maxkvbn]

Still an issue. Waiting on PR approval Sent from Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: github-actions[bot] ***@***.***> Sent: Monday, July 29, 2024 10:04:12 AM To: huggingface/transformers ***@***.***> Cc: Max Kristoffer Vendelbo Brown Nielsen ***@***.***>; Author ***@***.***> Subject: Re: [huggingface/transformers] Whisper - list index out of range with word level timestamps (Issue #31683) This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines<https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md> are likely to be ignored. — Reply to this email directly, view it on GitHub<#31683 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A2NOSB36G33GUCCOVNDA35LZOXZPZAVCNFSM6AAAAABKBNL5LWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJVGI3DSNZWGU>. You are receiving this because you authored the thread.Message ID: ***@***.***> MED VENLIG HILSEN / BEST REGARDS / MIT FREUNDLICHEN GRÜ?EN [cid:e4d080de-476d-4134-8d24-7e94debd5f87.jpg]<http://www.valeur.dk/> Max Kristoffer Vendelbo Brown Nielsen Phone. [phone] Email. ***@***.*** Michael Drewsens Vej 20a 8270 Højbjerg, Danmark <http://www.valeur.dk>www.valeur.dk<http://www.valeur.dk> [cid:015_fb_09e5f968-1567-45a3-9b3a-b8e2037096e5.png]<https://www.facebook.com/ValeurAs/> [cid:015_lin_a9d5adba-7aa6-42da-861d-f614b0f359e7.png] <https://www.linkedin.com/company/valeur-a-s/> <https://twitter.com/user_name_here> <https://www.youtube.com/user/user_name_here> <https://www.instagram.com/user_name_here> [cid:vss_4fd9b85a-6634-455e-92ad-b3dea1df1c61.png] <https://www.valeur.dk/international-telemarketing-2/> Valeur A/S places the highest priority on the security and privacy of our Clients. Therefore, we have put our efforts into ensuring that this message is free of errors and viruses. Despite our efforts, you should always scan all emails for any threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email.

[sanchit-gandhi]

Hey @maxkvbn, super sorry for the late reply here. I've been trying to reproduce the issue locally, but with every case I've tried the current code on main works as expected. For example, here's a reproducer that uses long-form generation with a single audio sample and word-level timestamps:

from transformers import pipeline, AutoProcessor, WhisperForConditionalGeneration
from transformers.utils import is_accelerate_available
from datasets import load_dataset

processor = AutoProcessor.from_pretrained("openai/whisper-tiny.en")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en", low_cpu_mem_usage=is_accelerate_available())

pipe = pipeline("automatic-speech-recognition", model="openai/whisper-tiny.en")

dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
sample = dataset[0]["audio"]

text = pipe(sample, batch_size=1, return_timestamps="word")

Could you please share a reproducer for the issue that I can run end-to-end? E.g. by sharing the audio file you're using, or updating the code example I've shared to trigger the error.

I've had a quick glance over your PR and the changes seem well-addressed, so if I can confirm it's a fix we need I'm confident we can merge it quickly.

[maxkvbn]

Hey @sanchit-gandhi.

Did you try with the uploaded audio from #31683 (comment)?