Question about VisionEncoderDecoderModel

[nguyenquangtan]

Question about VisionEncoderDecoderModel

[ydshieh]

It turns out that it is not me added this modeling file, it's @NielsRogge 😄

https://github.com/huggingface/transformers/pull/13874/files#diff-d6b5547bc8ece59fd6c96645e48cce4f32d0995a6621b57f896ad800168017bd

In such composite modeling cases, the encoder and decoder are usually from 2 different models (well, we could use the same model if it is EncoderDecoderModel), and the input/output tie embedding doesn't really make sense as @amyeroberts mentioned.

You can definitely tweak the code to suit your custom modeling logic and needs.

[amyeroberts]

Hi, thanks for raising an issue!

This is a question best placed in our forums. We try to reserve the github issues for feature requests and bug reports.

cc @ydshieh who I believe added this model originally

since the input and output embeddings are from the decoder and the decoder is an XForCausalLM class's object,

I'm not sure this is completely correct - the input to the model are pixel values, and the vision encoder will have its own embeddings, which hare not related to the decoder's embeddings.