The meaning of pre_assign_attn

[dingjiansw101]

The meaning of pre_assign_attn

[xvjiarui]

Hi @dingjiansw101
(1) pre_assign_attn is used to aggregate information from image tokens to group tokens. It is omitted in the paper for simplicity.
(2) Yes.

[dingjiansw101]

Thanks for the reply. But I still have a little question about (1).
From my understanding, the Transformer Layers before the Grouping Block already have the information propagations. Why do we need the extra pre_assign_attn for aggregation? What are the differences between them?

[xvjiarui]

Hi @dingjiansw101
Yes. You are correct. The only difference is the cross attention and self attention. pre_assign_attn is there for some legacy reasons. We didn't ablate whether it is import for our model but I guess no.

[l784669877]

半夜找到相同的疑问的人 有点激动