charlespikachu/mcibi

ICCV2021 - Mining Contextual Information Beyond Image for Semantic Segmentation

News

• 2022-09-17: Our extended version of MCIBI has been accepted by TPAMI. Codes are available at sssegmentation, paper is available at MCIBI++.

Introduction

The official repository for "Mining Contextual Information Beyond Image for Semantic Segmentation". Our full code has been merged into sssegmentation. So, you can leverage sssegmentation to re-implement our results.

Abstract

This paper studies the context aggregation problem in semantic image segmentation. The existing researches focus on improving the pixel representations by aggregating the contextual information within individual images. Though impressive, these methods neglect the significance of the representations of the pixels of the corresponding class beyond the input image. To address this, this paper proposes to mine the contextual information beyond individual images to further augment the pixel representations. We first set up a feature memory module, which is updated dynamically during training, to store the dataset-level representations of various categories. Then, we learn class probability distribution of each pixel representation under the supervision of the ground-truth segmentation. At last, the representation of each pixel is augmented by aggregating the dataset-level representations based on the corresponding class probability distribution. Furthermore, by utilizing the stored dataset-level representations, we also propose a representation consistent learning strategy to make the classification head better address intra-class compactness and inter-class dispersion. The proposed method could be effortlessly incorporated into existing segmentation frameworks (e.g., FCN, PSPNet, OCRNet and DeepLabV3) and brings consistent performance improvements. Mining contextual information beyond image allows us to report state-of-the-art performance on various benchmarks: ADE20K, LIP, Cityscapes and COCO-Stuff.

Framework

Performance

COCOStuff-10k

Model	Backbone	Crop Size	Schedule	Train/Eval Set	mIoU/mIoU (ms+flip)	Download
DeepLabV3	R-50-D8	512x512	LR/POLICY/BS/EPOCH: 0.001/poly/16/110	train/test	38.84%/39.68%	cfg | model | log
DeepLabV3	R-101-D8	512x512	LR/POLICY/BS/EPOCH: 0.001/poly/16/110	train/test	39.84%/41.49%	cfg | model | log
DeepLabV3	S-101-D8	512x512	LR/POLICY/BS/EPOCH: 0.001/poly/32/150	train/test	41.18%/42.15%	cfg | model | log
DeepLabV3	HRNetV2p-W48	512x512	LR/POLICY/BS/EPOCH: 0.001/poly/16/110	train/test	39.77%/41.35%	cfg | model | log
DeepLabV3	ViT-Large	512x512	LR/POLICY/BS/EPOCH: 0.001/poly/16/110	train/test	44.01%/45.23%	cfg | model | log

ADE20k

Model	Backbone	Crop Size	Schedule	Train/Eval Set	mIoU/mIoU (ms+flip)	Download
DeepLabV3	R-50-D8	512x512	LR/POLICY/BS/EPOCH: 0.01/poly/16/130	train/val	44.39%/45.95%	cfg | model | log
DeepLabV3	R-101-D8	512x512	LR/POLICY/BS/EPOCH: 0.01/poly/16/130	train/val	45.66%/47.22%	cfg | model | log
DeepLabV3	S-101-D8	512x512	LR/POLICY/BS/EPOCH: 0.004/poly/16/180	train/val	46.63%/47.36%	cfg | model | log
DeepLabV3	HRNetV2p-W48	512x512	LR/POLICY/BS/EPOCH: 0.004/poly/16/180	train/val	45.79%/47.34%	cfg | model | log
DeepLabV3	ViT-Large	512x512	LR/POLICY/BS/EPOCH: 0.01/poly/16/130	train/val	49.73%/50.99%	cfg | model | log

CityScapes

Model	Backbone	Crop Size	Schedule	Train/Eval Set	mIoU (ms+flip)	Download
DeepLabV3	R-50-D8	512x1024	LR/POLICY/BS/EPOCH: 0.01/poly/16/440	trainval/test	79.90%	cfg | model | log
DeepLabV3	R-101-D8	512x1024	LR/POLICY/BS/EPOCH: 0.01/poly/16/440	trainval/test	82.03%	cfg | model | log
DeepLabV3	S-101-D8	512x1024	LR/POLICY/BS/EPOCH: 0.01/poly/16/500	trainval/test	81.59%	cfg | model | log
DeepLabV3	HRNetV2p-W48	512x1024	LR/POLICY/BS/EPOCH: 0.01/poly/16/500	trainval/test	82.55%	cfg | model | log

LIP

Model	Backbone	Crop Size	Schedule	Train/Eval Set	mIoU/mIoU (flip)	Download
DeepLabV3	R-50-D8	473x473	LR/POLICY/BS/EPOCH: 0.01/poly/32/150	train/val	53.73%/54.08%	cfg | model | log
DeepLabV3	R-101-D8	473x473	LR/POLICY/BS/EPOCH: 0.01/poly/32/150	train/val	55.02%/55.42%	cfg | model | log
DeepLabV3	S-101-D8	473x473	LR/POLICY/BS/EPOCH: 0.007/poly/40/150	train/val	56.21%/56.34%	cfg | model | log
DeepLabV3	HRNetV2p-W48	473x473	LR/POLICY/BS/EPOCH: 0.007/poly/40/150	train/val	56.40%/56.99%	cfg | model | log

Citation

If this code is useful for your research, please consider citing:

@inproceedings{jin2021mining,
    title={Mining Contextual Information Beyond Image for Semantic Segmentation},
    author={Jin, Zhenchao and Gong, Tao and Yu, Dongdong and Chu, Qi and Wang, Jian and Wang, Changhu and Shao, Jie},
    booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
    pages={7231--7241},
    year={2021}
}

@article{jin2022mcibi++,
    title={MCIBI++: Soft Mining Contextual Information Beyond Image for Semantic Segmentation},
    author={Jin, Zhenchao and Yu, Dongdong and Yuan, Zehuan and Yu, Lequan},
    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
    year={2022},
    publisher={IEEE}
}

mcibi's Issues

模型复杂度

您好，感谢您卓越的工作，我想问一下，这篇论文中的表3将ASPP与DCA放在一起的话，这个参数量是如何计算的呀，希望能收到您的回复，谢谢

Very good job! When will the code of the paper be open source?

Very good job! When will the code of the paper be open source?

Memory model 一些问题

您好，您在memory.py里面写了update这个函数，但是这个函数我没有看到调用，我想问一下您这么update具体是怎么调用的，谢谢

关于mt更新机制的问题

你好，我发现论文公式(7)中mt的更新和代码中mt的更新机制好像不一样。

mcibi/src/models/memorynet/memory.py

Line 158 in 62dfe50

momentum = momentum_cfg['base_momentum'] / momentum_cfg['base_lr'] * learning_rate

请问一下，论文的实验结果是采用代码中的机制呢？

Why OCRNet is worse than DeepLabv3 in table 5.

Or why you choose DeepLabv3 as your structure not other latest structure in your experiments? Maybe it can hit a higher score.
Just confused.

Thank you~

在memory.py文件里面，导入base以及backbones时，缺少文件

在memory.py文件里面，导入base以及backbones时，缺少文件

mcibi/src/models/memorynet/memory.py

Line 11 in 62dfe50

from ..base import SelfAttentionBlock

How to start training based on this code?

I didn't find a train.py or main .py script based on the code in this repository. If I want to train a model, what should I do?