dcvl-wsss/clustercam

Clustering-guided Class Activation for Weakly Supervised Semantic Segmentation

This repository is an official Pytorch implementation of the paper "Clustering-guided Class Activation for Weakly Supervised Semantic Segmentation"
Yeong Woo Kim and Wonjun Kim
IEEE Access, Jan. 2024.

The overall architecture of the proposed method.

Installation

• Requirements
  • Pytorch >= 1.10

# We suggest to create a new conda environment with python version 3.9
conda create -n ClusterCAM python=3.9 -y
conda activate ClusterCAM

# Install Pytorch that is compatible with your CUDA version
conda install pytorch==1.10.1 torchvision==0.11.2 cudatoolkit=11.3 -c pytorch -c conda-forge

# Install requirements
pip install -r requirements.txt
conda install -c  conda-forge pydensecrf

Dataset Preparation

• Download PASCAL VOC2012 dataset: http://host.robots.ox.ac.uk/pascal/VOC/voc2012 (augmented annotations from SBD dataset, DropBox)

• Download MS COCO dataset:

  wget http://images.cocodataset.org/zips/train2014.zip
  wget http://images.cocodataset.org/zips/val2014.zip

• The resulting data structure should follow the hierarchy as below.

  ${REPO_DIR}  
  |-- voc12  
  |   |-- VOC2012
  |       |-- JPEGImages
  |       |-- Annotations
  |       |-- ImageSets
  |       |-- SegmentationClass
  |       |-- SegmentationClassAug
  |       |-- ...
  |-- coco   
  |   |-- COCO14
  |       |-- anno
  |       |-- annotations
  |       |-- SegmentationClass
  |       |-- train2014
  |       |-- val2014
  |       |-- ...
  |-- run_e2e.sh 
  |-- infer_e2e.sh 
  |-- ...
  

How to use it

Train

sh run_e2e.sh       # for the PASCAL VOC 2012 experiment
sh run_e2e_coco.sh  # for the MS COCO 2014 experiment

Inference

sh infer_e2e.sh      # for the PASCAL VOC 2012 experiment
sh infer_e2e_coco.sh # for the MS COCO 2014 experiment

Results

Quantitative results

Model	Dataset	Valid	Test	Checkpoint
ClsuterCAM	PASCAL VOC 2012	70.3	70.7	Download
ClsuterCAM	MS COCO 2014	41.8	-	Download

Qualitative results

Results of semantic segmentation on the PASCAL VOC 2012 dataset [24]. From top to bottom: input images, ground truths, results by AFA, ToCo, and ClusterCAM (ours). Visualization examples of attention weights and ClusterCAMs. From top to bottom: input images, ground truths, patch-to-class attention weights, cluster-to-class attention weights, and ClusterCAMs.

Acknowledgments

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean Government [Ministry of Science and ICT (MSIT)] under Grant 2023R1A2C1003699.

Our implementation and experiments are built on top of open-source GitHub repositories. We thank all the authors who made their code public, which tremendously accelerates our project progress. If you find these works helpful, please consider citing them as well.

xulianuwa/MCTformer
rulixiang/afa

Citation

@ARTICLE{10381698,
  author={Kim, Yeong Woo and Kim, Wonjun},
  journal={IEEE Access}, 
  title={Clustering-Guided Class Activation for Weakly Supervised Semantic Segmentation}, 
  year={2024},
  volume={12},
  number={},
  pages={4871-4880},
  doi={10.1109/ACCESS.2024.3350176}}

clustercam's Issues

Hello, when will the code be made public?

Hello, when will the code be made public?