If you use the code in this repo for your work, please cite the following bib entries:
@article{yu2023mix,
title={Mix-ViT: Mixing attentive vision transformer for ultra-fine-grained visual categorization},
author={Yu, Xiaohan and Wang, Jun and Zhao, Yang and Gao, Yongsheng},
journal={Pattern Recognition},
volume={135},
pages={109131},
year={2023},
publisher={Elsevier}
}
Abstract
Ultra-fine-grained visual categorization (ultra-FGVC) moves down the taxonomy level to classify sub-granularity categories of fine-grained objects. This inevitably poses a challenge, i.e., classifying highly similar objects with limited samples, which impedes the performance of recent advanced vision transformer methods. To that end, this paper introduces Mix-ViT, a novel mixing attentive vision transformer to address the above challenge towards improved ultra-FGVC. The core design is a self-supervised module that mixes the high-level sample tokens and learns to predict whether a token has been substituted after attentively substituting tokens. This drives the model to understand the contextual discriminative details among inter-class samples. Via incorporating such a self-supervised module, the network gains more knowledge from the intrinsic structure of input data and thus improves generalization capability with limited training sample. The proposed Mix-ViT achieves competitive performance on seven publicly available datasets, demonstrating the potential of vision transformer compared to CNN for the first time in addressing the challenging ultra-FGVC tasks.
Prerequisites
The following packages are required to run the scripts:
在使用MixViT-main/scripts/train_cub.sh复现mixvit时会报错:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/ml_collections/config_dict/config_dict.py", line 883, in getitem
field = self._fields[key]
KeyError: 'resnet'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/ml_collections/config_dict/config_dict.py", line 807, in getattr
return self[attribute]
File "/usr/local/lib/python3.6/dist-packages/ml_collections/config_dict/config_dict.py", line 889, in getitem
raise KeyError(self._generate_did_you_mean_message(key, str(e))) KeyError: "'resnet'"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/data/private/MixViT-main/train.py", line 458, in
main()
File "/opt/data/private/MixViT-main/train.py", line 451, in main
args, model = setup(args)
File "/opt/data/private/MixViT-main/train.py", line 102, in setup
model = make_model(config, args, zero_head=True, num_classes=num_classes, vis=True)
File "/opt/data/private/MixViT-main/models/make_model.py", line 94, in make_model
model = build_transformer(config, args, zero_head=True, num_classes=num_classes, vis=True)
File "/opt/data/private/MixViT-main/models/make_model.py", line 43, in init
self.inter_arch = ResNetV2(block_units=config.resnet.num_layers,
File "/usr/local/lib/python3.6/dist-packages/ml_collections/config_dict/config_dict.py", line 809, in getattr
raise AttributeError(e) AttributeError: "'resnet'"
请问是什么情况呢?
File "F:\Code\FGVC\MixViT-main\models\make_model.py", line 71, in load_from
self.vit.load_from(weights)
File "F:\Code\FGVC\MixViT-main\models\modeling.py", line 520, in load_from
unit.load_from(weights, n_block=uname)
File "F:\Code\FGVC\MixViT-main\models\modeling.py", line 229, in load_from
query_weight = np2th(weights[pjoin(ROOT, ATTENTION_Q, "kernel")]).view(self.hidden_size, self.hidden_size).t()
File "D:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\npyio.py", line 260, in getitem
raise KeyError("%s is not a file in the archive" % key)
KeyError: 'Transformer/encoderblock_0\MultiHeadDotProductAttention_1/query\kernel is not a file in the archive' Resized position embedding from size:torch.Size([1, 197, 768]) to size: torch.Size([1, 6, 768]) with height:5 width: 1
数据集用的网盘里面的数据集,soyloc,按照sh文件导入预训练权重,然后报错