use_cache=False makes a huge difference in Paligemma

[edmondja]

use_cache=False makes a huge difference in Paligemma

[molbap]

Hi @edmondja, thanks for the issue! I'll take a look soon, but one note is that PaliGemma is creating a fully noncausal attention mask when you're passing attention_mask (full attention on image+ text prefix), but decoding is supposed to be causal if you use generate().

[molbap]

Hi @edmondja check out this discussion: https://huggingface.co/google/paligemma-3b-mix-448/discussions/6 I think I found a fix, will be able to push tomorrow, thought to let you know! thanks again for the investigation!

[edmondja]

Great, thanks for paying careful attention to our issue.

[edmondja]

Hi @edmondja, thanks for the issue! I'll take a look soon, but one note is that PaliGemma is creating a fully noncausal attention mask when you're passing attention_mask (full attention on image+ text prefix), but decoding is supposed to be causal if you use generate().

Alright thanks