Hi, some questions about the model!

[autonomousvision]

Hi, some questions about the model!

[ap229997]

The fusion can also be done at 64x64 resolution but that would be too computationally expensive since a transformer is used (quadratic complexity due to attention), so I reduced the size to 8x8 at each resolution of the intermediate feature maps.

[roger-cv]

Thanks for your quick reply. I guess that the input feature map of the transformer of each layer will be downsampled to 8*8 according to what you mean?

[ap229997]

that's correct, now there are several variants of transformer which address the quadratic complexity issue of the transformer (eg. Linformer) so maybe it's possible to use the transformer without downsampling.

[roger-cv]

that's correct, now there are several variants of transformer which address the quadratic complexity issue of the transformer (eg. Linformer) so maybe it's possible to use the transformer without downsampling.

Ok, Another interesting question is that can this fusion fashion based on the transformer be replaced with other transformers, such as swim or PVT. Because I notice that this transformer is developed based on the GPT suited for the NLP area.

[ap229997]

I agree, architecture design can be improved quite a bit.

[roger-cv]

Ok, Nice work, Thanks for your reply.

[Kin-Zhang]

But it may require more resources to train...

I agree, architecture design can be improved quite a bit.