Code repository for our paper "CLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation" in CVPR 2022.
๐ Code for our paper "CCAM: Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation" in CVPR 2022 is also available at here.
Please to NOTE that this repository is an improved version of our camera-ready version (you can refer to the directory of previous_version/
). We recommend to use our improved version of CLIMS instead of camera-ready version.
You will need to download the images (JPEG format) in PASCAL VOC2012 dataset at here and train_aug ground-truth can be found at here. Make sure your data/VOC2012 folder
is structured as follows:
โโโ VOC2012/
| โโโ Annotations
| โโโ ImageSets
| โโโ SegmentationClass
| โโโ SegmentationClassAug
| โโโ SegmentationObject
You will need to download the images (JPEG format) in MSCOCO 2014 dataset at here and ground-truth mask can be found at here. Make sure your data/COCO folder
is structured as follows:
โโโ COCO/
| โโโ train2014
| โโโ val2014
| โโโ annotations
| | โโโ instances_train2014.json
| | โโโ instances_val2014.json
| โโโ mask
| | โโโ train2014
| | โโโ val2014
- Install CLIP.
$ pip install ftfy regex tqdm
$ pip install git+https://github.com/openai/CLIP.git
- Download pre-trained baseline CAM ('res50_cam.pth') at here and put it at the directory of
cam-baseline-voc12/
. - Train CLIMS on PASCAL V0C2012 dataset to generate initial CAMs.
CUDA_VISIBLE_DEVICES=0 python run_sample.py --voc12_root /data1/xjheng/dataset/VOC2012/ --hyper 10,24,1,0.2 --clims_num_epoches 15 --cam_eval_thres 0.15 --work_space clims_voc12 --cam_network net.resnet50_clims --train_clims_pass True --make_clims_pass True --eval_cam_pass True
- Train IRNet and generate pseudo semantic masks.
CUDA_VISIBLE_DEVICES=0 python run_sample.py --voc12_root /data1/xjheng/dataset/VOC2012/ --cam_eval_thres 0.15 --work_space clims_voc12 --cam_network net.resnet50_clims --cam_to_ir_label_pass True --train_irn_pass True --make_sem_seg_pass True --eval_sem_seg_pass True
- Train DeepLabv2 using pseudo semantic masks.
cd segmentation/
Method | backbone | CAMs | + RW | + IRNet |
---|---|---|---|---|
CLIMS(camera-ready) | R50 | 56.6 | 70.5 | - |
CLIMS(this repo) | R50 | 58.6 | ~73 | 74.1 |
Please cite the results of camera-ready version
Method | Supervision | Network | Pretrained | val | test |
---|---|---|---|---|---|
AdvCAM | I | DeepLabV2 | ImageNet | 68.1 | 68.0 |
EDAM | I+S | DeepLabV2 | COCO | 70.9 | 70.6 |
CLIMS(camera-ready) | I | DeepLabV2 | ImageNet | 69.3 | 68.7 |
CLIMS(camera-ready) | I | DeepLabV2 | COCO | 70.4 | 70.0 |
CLIMS(this repo) | I | DeepLabV2 | ImageNet | 70.3 | 70.6 |
CLIMS(this repo) | I | DeepLabV2 | COCO | 71.4 | 71.2 |
CLIMS(this repo) | I | DeepLabV1-R38 | ImageNet | 73.3 | 73.4 |
(Please cite the results of camera-ready version. Initial CAMs, pseudo semantic masks, and pre-trained models of camera-ready version can be found at Google Drive)
- Download pre-trained baseline CAM ('res50_cam.pth') at here and put it at the directory of
cam-baseline-coco/
. - Train CLIMS on MSCOCO 2014 dataset to generate initial CAMs.
CUDA_VISIBLE_DEVICES=6,7 python -m torch.distributed.launch --nproc_per_node=2 run_sample_coco.py --work_space clims_coco --clims_network net.resnet50_clims --train_clims_pass True --make_clims_pass True --eval_cam_pass True --clims_num_epoches 8 --cam_eval_thres 0.15 --hyper 2,14,1.25,0.2 --cam_batch_size 16 --clims_learning_rate 0.0005 --use_distributed_train True --cbs_loss_thresh 0.285
If you are using our code, please consider citing our paper.
@InProceedings{Xie_2022_CVPR,
author = {Xie, Jinheng and Hou, Xianxu and Ye, Kai and Shen, Linlin},
title = {CLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2022},
pages = {4483-4492}
}
@article{xie2022cross,
title={Cross Language Image Matching for Weakly Supervised Semantic Segmentation},
author={Xie, Jinheng and Hou, Xianxu and Ye, Kai and Shen, Linlin},
journal={arXiv preprint arXiv:2203.02668},
year={2022}
}
This repository was highly based on IRNet, thanks for Jiwoon Ahn's great code.