js-fan/icd

Learning Integral Objects with Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation

Learning Integral Objects with Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation

Learning Integral Objects with Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation, Junsong Fan, Zhaoxiang Zhang, Chunfeng Song, Tieniu Tan, CVPR2020 [paper].

Introduction

The previous class activation map (CAM) based approach learns inter-class boundaries, which focus on the difference between different foreground classes. It may not be optimal for the weakly supervised segmentation problem because the target object and the background share the same classes in the same image. To alleviate this problem, we propose an ICD approach to learning per-category intra-class boundaries between the foreground objects and the background, which is more appropriate for weakly supervised segmentation problems.

The proposed approach is end-to-end and can be trained together with the CAM branch in a single round. The pseudo-masks derived by the proposed ICD is more complete than the CAM based results.

Prerequisite

• Python 3.7, MXNet 1.3.1, Numpy, OpenCV, pydensecrf
• NVIDIA GPU

Usage

Prepare the data and pretrained parameters:

• Download the VOC 2012 dataset, which should contain VOC2012/JPEGImages and VOC2012/Annotations.
• Download the ImageNet pretrained parameters for the VGG16 backbone and put it in the folder ICD/data/pretrained
• Download the pre-computed superpixels here, untar it, and put it into the folder ICD/data/superpixels. You can also generate them by yourself, to do this, please follow the following commands:

cd external/superpixel
make -j8   ## relies on OpenCV, edit Makefile if necessary.
./makeSuperpixel ../../data/VOC2012/train_aug.txt Your/VOC2012/JPEGImages ./data/VOC 75
cd ../..
python generate_superpixel.py

Train ICD and generate pseudo-masks:

python run_icd.py --gpus 0,1 --image-root Your/VOC2012/JPEGImages --annotation-root Your/VOC2012/Annotations --superpixel-root ./data/superpixels/voc_superpixels

The above script contains codes for training and generating the seeds. By default, the seeds will be generated at ./snapshot/icd/results/seeds/crf. After obtaining the seeds, you can take them as pseudo labels and train any segmentation models on the training set, for example, the DeepLab-v2 used in the paper. Codes to train the Deeplab-v2 models will also be released soon.

Generate seeds with saliency maps:

To generate seeds with the saliency, you need pre-computed saliency maps firstly. We adopt the same saliency maps used by the OAA project, and the saliency maps can be downloaded here. To generate seeds with saliency maps, use the --use-sal flag, e.g.,

python run_icd.py --image-root Your/VOC2012/JPEGImages --annotation-root Your/VOC2012/Annotations --use-sal --only-gen-seeds

Seeds will be placed in ./snapshot/icd/results/seeds/sal by default.

Citation

@InProceedings{Fan_2020_CVPR,
author = {Fan, Junsong and Zhang, Zhaoxiang and Song, Chunfeng and Tan, Tieniu},
title = {Learning Integral Objects With Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}

icd's Issues

Configuration environment issues

Hi, Thanks for the great work!
when i use python3.7，I could not install pydensecrf successfully, I am using Ubuntu18.04. What platform do you use?And cuda version？

Can you share the code for computing the AP?

Hi, Thanks for the great work!
I'm interested in your Figure.5, which is the AP scores of different classes.
Can you share the code for computing the AP given the CAM/ICD score maps as input?
Or, can you give the detailed formulations? I read your description in the paper but remain a bit confused.

I appreciate your help.

The warm-up strategy for bottom-up estimation

Hey, I notice that you mention that 'initial bottom-up estimates are not reliable', so you utilize warm-up strategy. I meet the same problem when I try to reproduce your work in pytorch. After several epochs the loss turns into 'nan'. Would you please let me know the certain codes in your work for this strategy? I try to find it but failed. And I will really appreciate that if you would please let me know if there is any possible solutions to avoid instable training for the bottom-up estimation.(I guess it is also the reason why you only use single-class image for this step?) Thank you so much!

Reproduce the performance using saliency map

Hello, Junsong Fan! Thank you for sharing your code.

I am reproducing the performance on your paper. When I trained your code, the pseudo masks are generated well and the performance of them fits well.

Now I'd like to generate the pseudo mask using the saliency map you used. I get the saliency map from DSS.
How can I integrate the saliency map into the pseudo mask?
In the paper, you mentioned that "replace background score with the saliency scores", but I replace the score with saliency (or 1-saliency) it contain a lot of 'ignore_part'.

Can you share the details of integrating the saliency map?

Thank you.

论文里FickleNet的结果好像不是I.+S.的？

code?

Great work! when do you plan to release the code?

About the Localization Maps of ICD

Dear Dr Fan,

I am Lei Zhou, a researcher from Shanghai, China, now working on a project related to weakly supervised segmentation. Recently, I have read your paper entitled "Learning Integral Objects with Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation" published in CVPR 20. It is really an excellent work, superior performance and professional writing. Now I am trying to compare my method with your ICD, but I face up with some challenges when running your open-source code, because I am not familiar with MXNET and fail to reproduce your results.

Is it possible to share the localization maps (PASCAL VOC 2012) run by VGG16 with me via email? Or if there exist any web links from which I can download these localization maps?

Looking forward to your replay!! Thanks!!

Can you publish the best numbuer of epoch？

Hi, Thanks for the great work!

I am trying to reproduce your work.

When I used the default parameters for training:
python run_icd.py --gpus 0,1 --image-root Your/VOC2012/JPEGImages --annotation-root Your/VOC2012/Annotations --superpixel-root ./data/superpixels/voc_superpixels
I got only 47% mIoU (with CRF) and 53% mIoU (without CRF). In your paper, mIoU can reach 62.2%. I would like to know some settings of specific parameters. By the way, when I tried to increase the epoch, Code could not generate scores and seeds. I don't know why.

The performance without superpixel

Hey, thank you for your work.
Would you please let me know the performance with 'Bottom-up' 'Refine' and 'Top-down' but without CRF and superpixel.
I see the performance published in the Table 3 of the paper, the second last one is 59.9%, and you mention there may be a 6% in another closed issue.
Does it mean the performance is around 54% if only superpixel missed?
Thank you so much for your time~

Superpixels

Hello, thanks for your great work. I want to know the performance when you don't hierarchically merge the superpixles witch are generated by "Efficient graph-based image segmentation". Could you tell me that?