Question about the visualization for VIT?

[hila-chefer]

Question about the visualization for VIT?

[hila-chefer]

Hi @zhaoxin94 thanks for your interest in our work!
What you are referring to is a normalization we use for the multi modal update rule.
For pure self attention architectures, this issue of no normalization will indeed cause the values in the diagonal to be bigger than those outside it, but for self attention, we take the row corresponding to the CLS token, and disregard the CLS score itself.
So basically, the values in the diagonal do not influence the visualization in the self attention case, and regularization has little to no impact.
This question was raised by the reviewers as well, so we ran experiments of normalizing Eq. 6 and found results to be very similar with a marginal difference.

I hope this answers your question, please let me know if you require any further clarifications.

Best,
Hila.

[hila-chefer]

@zhaoxin94 closing this issue due to inactivity. Please reopen if necessary.