Issue related to dtype with F.conv1d in Whisper evaluation

[moncefbenaicha]

Issue related to dtype with F.conv1d in Whisper evaluation

[ylacombe]

Hey @moncefbenaicha, you're temporary solution is actually the right one since the processor only outputs torch.float32 arrays!

However, I do believe it should work with Flash Attention, have you got an error using bloat16 and FA?

[LysandreJik]

cc @sanchit-gandhi @ylacombe

[moncefbenaicha]

Update

A temporary solution is to force the input to bfloat16 and disable flash_attention.

batch = self.processor(
            audio=audio_arrays,
            sampling_rate=16000,
            padding="max_length",
            return_tensors="pt",
        )
batch["input_features"] = batch["input_features"].to(dtype=torch.bfloat16)

[sanchit-gandhi]

Hey @moncefbenaicha - it would be great to see:

1. How you're instantiating the model using .from_pretrained. Specifically, what argument you're passing to attn_implementation, torch_dtype, and whether you're moving the model manually to a torch device
2. The training args you're using. Specifically, what you set for fp16, bf16, fp16_full_eval and bf16_full_eval

Passing bf16_full_eval=True might be of interest to you if you're casting the model weights to bf16 manually yourself.