code mismatch with the theory

[microsoft]

code mismatch with the theory

[basavaraj-hampiholi]

Hi @diaodeyi ,
In the present code, get_cls_model function is called by the registry.py. You can use build_model in build.py to call the model. Otherwise, you can remove the registry and directly call get_cls_model function. Both way should work

Good luck..

[askerlee]

This is after conv_proj_q, conv_proj_k and conv_proj_v. But I'm not sure why the authors still use the pointwise projections after the conv projections.

[basavaraj-hampiholi]

@askerlee, I think it is part of depthwise separable convolutions. Depthwise convolutions followed by pointwise projections.

[diaodeyi]

I want to know the code how to call the get_cls_model function in the cls_cvt.py

[diaodeyi]

By the way , theself.proj = nn.Linear(dim_out, dim_out)Means FFN only projection with same dimension?

[basavaraj-hampiholi]

@diaodeyi It's the single linear layer (with the same in/out dimension) right after the attention calculation. The FFN in this code is class MLP (line 53).

[diaodeyi]

Thanks, there are so many linear projections that aren't be mentioned by paper.

[basavaraj-hampiholi]

@diaodeyi Yes. I think they have left them out with the presumption that the reader has a prior good understanding of basic transformer architecture.

[diaodeyi]

@askerlee, I think it is part of depthwise separable convolutions. Depthwise convolutions followed by pointwise projections.

No, I think the proj_q\k\v are exactly the things the paper does not mention.

[Markin-Wang]

@askerlee, I think it is part of depthwise separable convolutions. Depthwise convolutions followed by pointwise projections.

No, I think the proj_q\k\v are exactly the things the paper does not mention.

Hi, the seperable depth conv contains two parts: depth-wise conv and point-wise conv. The author implemented the point-wise conv via the linear layer, maybe because it's convenience for the ablation study. The only difference between them is the bias term.