vitae-transformer/vitae-vsa

The official repo for [ECCV'22] "VSA: Learning Varied-Size Window Attention in Vision Transformers"

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

Updates | Introduction | Statement |

Current applications

Image Classification: Please see ViTAE-Transformer for image classification;

Object Detection: Please see ViTAE-Transformer for object detection;

Sementic Segmentation: Please see ViTAE-Transformer for semantic segmentation;

Animal Pose Estimation: Please see ViTAE-Transformer for animal pose estimation;

Matting: Please see ViTAE-Transformer for matting;

Remote Sensing: Please see ViTAE-Transformer for Remote Sensing;

Updates

09/04/2021

• The pretrained models for ViTAE on matting and remote sensing are released! Please try and have fun!

24/03/2021

• The pretrained models for both ViTAE and ViTAEv2 are released. The code for downstream tasks are also provided for reference.

07/12/2021

• The code is released!

19/10/2021

• The paper is accepted by Neurips'2021! The code will be released soon!

06/08/2021

• The paper is post on arxiv! The code will be made public available once cleaned up.

Introduction

This repository contains the code, models, test results for the paper ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias. It contains several reduction cells and normal cells to introduce scale-invariance and locality into vision transformers. In ViTAEv2, we explore the usage of window attentions without shift operations to obtain a better balance between memory footprint, speed, and performance. We also stack the proposed RC and NC in a multi-stage manner to faciliate the learning on other vision tasks including detection, segmentation, and pose.

Fig.1 - The details of RC and NC design in ViTAE.

Fig.2 - The multi-stage design of ViTAEv2.

Statement

This project is for research purpose only. For any other questions please contact yufei.xu at outlook.com qmzhangzz at hotmail.com .

Citing ViTAE and ViTAEv2

@article{xu2021vitae,
  title={Vitae: Vision transformer advanced by exploring intrinsic inductive bias},
  author={Xu, Yufei and Zhang, Qiming and Zhang, Jing and Tao, Dacheng},
  journal={Advances in Neural Information Processing Systems},
  volume={34},
  year={2021}
}
@article{zhang2022vitaev2,
  title={ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond},
  author={Zhang, Qiming and Xu, Yufei and Zhang, Jing and Tao, Dacheng},
  journal={arXiv preprint arXiv:2202.10108},
  year={2022}
}

Other Links

Image Classification: See ViTAE for Image Classification

Object Detection: See ViTAE for Object Detection.

Semantic Segmentation: See ViTAE for Semantic Segmentation.

Animal Pose Estimation: See ViTAE for Animal Pose Estimation.

Matting: See ViTAE for Matting.

Remote Sensing: See ViTAE for Remote Sensing.

vitae-vsa's Issues

When will the code be released?

Is it possible the positional encoding rather than VSA works？

In your code file ViTAE-VSA\Image-Classification\vitaev2_vsa\NormalCell.py L130:
self.pos = nn.Conv2d(dim, dim, window_size//2*2+1, 1, window_size//2, groups=dim, bias=True)
your window_sizeis 7，so the self.pos convolution kernel is 7 too, in most Positional Encoding extractor it is so large.

So is it possible that the positional encoding rather than VSA is working ？

Swin+VSA

语义分割和 Swin+VSA 什么时候发布呀

img_size in VSA attention

Hello, i would like to ask about the setting of img_size=(1,1) in line 36 vsa.py. Which size does it mean? thanks a lot.

How to use the weights weight file provided？

I would like to ask how to use the provided weights pre-training file, .tar type zipped file extracted without a .pth file, but a folder

After 5 months，When will the code be released?

pre-trained model of Swin + VSA

hello, i am now doing a work about semantic segmentation and i would like to use swin with vsa model as the feature extractor backbone. I would like to ask, about when will you maybe upload the pre-trained model? Without pretrained model the results are really with low accuracy now... Thank you so much.

Can you release the pretrained model for ViTAEv2*-VSA-B?

@Qiming-Zhang1 @RogerZhangzz