lsa1997/pcaa

Code for "Partial Class Activation Attention for Semantic Segmentation" [CVPR2022]

PCAA for Semantic Segmentation

This repository is for the CVPR2022 paper Partial Class Activation Attention for Semantic Segmentation.

Introduction

For the first time, this paper explores modeling pixel relation via Class Activation Map (CAM). Beyond the previous CAM generated from imagelevel classification, we present Partial CAM, which subdivides the task into region-level prediction and achieves better localization performance. We further propose Partial Class Activation Attention (PCAA) that simultaneously utilizes local and global class-level representations for attention calculation. Notably, our method achieves state-of-the-art performance on several challenging benchmarks.

Usage

Requirements

python>=3.6, torch>=1.3, Pillow, opencv-python

Backbones

We use the pre-trained backbones provided by open-mmlab, including resnet50_v1c and resnet101_v1c.

Dataset

Prepare related datasets: Cityscapes and ADE20K. Data paths should be as follows:

.{YOUR_CS_PATH}
├── gtFine
│   ├── train
│   ├── val
├── leftImg8bit
│   ├── train
│   ├── val

.{YOUR_ADE_PATH}
├── annotations
│   ├── training
│   ├── validation
├── images
│   ├── training
│   ├── validation

Train

Multi-GPU training is required. You should have at least 4 GPUs (>= 11G) to train a model on Cityscapes. You will need 8 GPUs to train a model based on ResNet-101 on ADE20K.

E.g. To train a model on Cityscapes, modify DATA_PATH, BACKBONE, RESTORE_PATH, and SAVE_DIR in scripts/train_cs.sh then run:

sh scripts/train_cs.sh

Test

Single-GPU evaluation is supported.

E.g. To evaluate a model on Cityscapes, modify the settings in scripts/evaluate_cs.sh and run:

sh scripts/evaluate_cs.sh

References

This repo is mainly built based on pytorch-segmentation-toolbox, DNL and mmsegmentation. Thanks for their great work!

Citation

If you find our codes useful, please consider to cite with:

@inproceedings{liu2022partial,
  title={Partial Class Activation Attention for Semantic Segmentation},
  author={Liu, Sun-Ao and Xie, Hongtao and Xu, Hai and Zhang, Yongdong and Tian, Qi},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2022}
}

pcaa's Issues

About experimental results of Table.3 in paper

感谢作者的工作，有个小问题，Table3 在Citycapes的实验结果，使用哪个代码训练的FCN（baseline）、FCN+NL、FCN+OCR、FCN+PCAA等

How to visualize "partial cam" and "attention map"?

How to visualize "attention map for the corresponding classes " in Figure 5 in your paper?

询问local feats

我想询问一下GCN Class的作用，根据代码，它好像使16个patch的local feats进行交互，此时的local feats已经含有其他patch的信息了，是否还能表示local feats