lironui/linear-attention-mechanism

Attention mechanism

Linear-Attention-Mechanism

⭐ Welcome to my HomePage ⭐

This repository implementatesLinear-Attention-Mechanism based on PyTorch.

The detailed formula can be seen in the Linear Attention Mechanism: An Efficient Attention for Semantic Segmentation or Multi-stage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images.

If our code is helpful to you, please cite

1. Li R, Jianlin Su, Duan C and Zheng S. Linear Attention Mechanism: An Efficient Attention for Semantic Segmentation[J]. arXiv preprint arXiv:2007.14902, 2020.
2. 苏剑林. (2020, Jul 04). 《线性Attention的探索：Attention必须有个Softmax吗？ 》[Blog post]. Retrieved from https://spaces.ac.cn/archives/7546
3. R. Li, S. Zheng, C. Duan, J. Su and C. Zhang. "Multistage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images." in IEEE Geoscience and Remote Sensing Letters, doi: 10.1109/LGRS.2021.3063381.

linear-attention-mechanism's Issues

Question about transformation from (10) to (11)

Hi, thanks for your brilliant work. I have a question about transformation from (10) to (11) in the paper.

Is it true that "dot product first, then summation" equals to "summation first, then dot product"?

Could you explain it in more details? Thank you!