jbeomlee93/advcam

Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation (CVPR 2021)

Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation

Input Image	Initial CAM	Successive Maps with adversarial climbing

The implementation of Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation, Jungbeom Lee, Eunji Kim, and Sungroh Yoon, CVPR 2021. [paper]

Installation

• We kindly refer to the offical implementation of IRN.
• This repository is tested on Ubuntu 18.04, with Python 3.6, PyTorch 1.4, pydensecrf, scipy, chaniercv, imageio, and opencv-python.

Usage

Step 1. Prepare Dataset

• Download PASCAL VOC 2012 benchmark: Download.

Step 2. Prepare pre-trained classifier

• Pre-trained model used in this paper: Download.
• You can also train your own classifiers following IRN.

Step 3. Obtain the pseudo ground-truth masks for PASCAL VOC train_aug images and evaluate them

bash get_mask_quality.sh

Step 4. Train a semantic segmentation network

• To train DeepLab-v2, we refer to deeplab-pytorch. However, this repo contains only COCO pre-trained model. We provide ImageNet pre-trained model for a fair comparison with the other methods.

Acknowledgment

This code is heavily borrowed from IRN, thanks jiwoon-ahn!

advcam's Issues

obtain_CAM_masking.py takes a lot of time?

Hi,
your work is superb.
can i know whether obtain_CAM_masking.py takes a lot of time to gets completed ?
it seems there are two loops in GradCAM process. i have 103 classes.

Some questions about released code

@jbeomlee93 ! Sorry for disturbing you again. I still have two questions about the released code.
(1) In obtain_CAM_masking_super_pixel.py. Since you have used grad-cam to generate the class activation map(i.e., CAM), why don't you use resnet50.py with grad-cam to generate outputs. Actually, you used resnet50_cam.py with grad-cam to generate the outputs.

(2) Can you share the code of "SEAM+AdvCAM" with me. I try to reproduce it by myself but the performance is not good as yours. My email address is [email protected]

how to draw feature manifold

Thank you for your excellent work. I want to know how to draw feature manifold with T-SNE, can you give me more code details? thanks

Code for coco and weakly supervised object localization

Hi,
Thanks for your code and paper! Congratulations on your paper being accepted by TPAMI! I notice that in the version of TPAMI, additional experimental results on MS COCO dataset and ImageNet1k are presented. May I trouble you to release these code and fine-turned models?
Thanks!

combine with seam

Hello, thanks for your great work. I notice that you have combine the AdvCAM with SEAM in paper. Could you please teel me how ti combine them?

Question about Table 4.

Thanks for sharing this great work!

Just wondering if you could also share the code that reproduces table 4, especially AdvCAM with SEAM.

I found the SEAM GitHub and I assume that your AdvCAM should be replaced with cam in SEAM code as shown in the below link
[https://github.com/YudeWang/SEAM/blob/c55601649c5fa676836d3ec70ec044541b7d1d83/network/resnet38_SEAM.py#L32]

Could you please guide me on how to merge your AdvCAM into the SEAM module?

Thanks

Code

An excellent work！！Thanks for your work！！

'Namespace' object has no attribute 'bam'

When i run python run_sample.py --make_cam_pass True, an error is reported on line 66 of the /home/dancer/AdvCAM/step/make_cam.py

seed cam‘s miou on val set

Can you provide the weight? I would like to know the miou of the seed cam and the miou of the seed cam + CRF for your model on the val set. Thank you very much

Performance Gap on Init Seed

Hi, I ran your codes and apply all the hyper-parameters by default. But the init seed mIoU is only 48.75, which is reported as 55.6 in your paper. I wonder if i miss something? Here is my train command line:
CUDA_VISIBLE_DEVICES=4,5,6,7 python run_sample.py --voc12_root Dataset/VOC2012_SEG_AUG/ --cam_out_dir result/cam_res50 --sem_seg_out_dir result/seg_res50 --ir_label_out_dir result/ir_label_res50 --irn_weights_name sess/res50_irn.pth --make_ins_seg_pass False --eval_ins_seg_pass False

And here is my init seed result:

how to visualize the loss landscape?

in the paper you mentioned and showed an example of visualizing the loss landscape.
Can you please explain how to do it ?
I would really appreciate it if you can share the code for that since I am new to this topic and your implementation will be super useful!

regarding run_sample.py

can you please tell me what is the goal of run_sample.py ?

FileNotFoundError: [Errno 2] No such file or directory: 'result/cam_adv_mask/2007_000032.npy'

Thanks for sharing your code.
after downloading all the materials now I am running the bash get_mask_quality.sh but it is super slow and I am wondering how long that process should take? and what does that _work function in obtain_CAM_masking.py really do? do I need to do it?

Questions on performance gap

Hi,

Thank you for your code and paper. I've already tried your code, but I cannot achieve the same performance as the paper. Would you please help me figure out where the problem is?

In my experiments, all hyper-parameters follow the default setting in rum_sample.py and Deeplabv2. The performance comparison is as follows,

Model	Initial Seed	Pseudo Mask	DeeplabV2 on Val
my exp.	55.6	69.2	62.9 (64.9 after crf)
paper	55.6	69.9	68.1

I guess the main difference may be caused by the hyper-parameters of Deeplabv2, could you please point out the differences between my experiments and yours that may result in the gap or provide your training script?

Thank you!

About npy transformes png

Hello, in the process of my reproduction, can the npy generated in the cam_adv_mask file be converted into png for CAM visualization, can you provide the relevant code!

Could you release code for your semi-supervised part?

Questions about equation (1) and (3)

Hi @jbeomlee93 I have some questions for the equations of you paper. That is:
1. For equation(1), the paper[16] uses x' = x + gradient to adversarial attack the input image (i.e., X) and finally generate the image Xadv, which is opposited to your equation (1) (i.e., x' = x - gradient ).
2. For equation(3), the paper[28] uses iter(x' = x + gradient) to represents the adversarial attack process, which is same to equation (3)(i.e., x' = x + gradient ). I think adversarial climbing should be opposited to the adversarial attack.
Sincerely waiting for your reply !

Question about applying CRF for your init seed mask?

Could u offer more implementation details about this part?

the result of obtian_cam_masking.py and the pseudo mask

Hi，thanks for your sharing!But i still have some questions look forward to your answer!
is the result of the obtian_cam_masking.py a pseudo mask dirtectly used in training segmentation network?if not, can you tell how to generate it?
Thank you!

can you please provide the final trained models of your paper?

would it be possible to provide your trained model so we can use it for evaluation and regenerating the results?

how to visualize the loss landscape?

About IRN

Thanks for the great work.
I have a question about IRN. Have you reproduced the performance of IRN?
I run the code of IRN, but the results are not as good as the paper of IRN.
In your implementation, will it affect the performance of AdvCAM?

Thank you in advance for your help.

Pre-trained model used in this paper

Thank you very much for your excellent work. I found that your pre-trained model could not be downloaded, could you please provide a new link? Or send the pre-trained model to my email at [email protected]. Best wishes.

Semi-supervised learning setting

Hi thanks for sharing a great work!

In the previous issue, you said you used the CCT's code for semi-supervised learning with the replacement of pseudo labels. Does that mean you used the default configuration provided in CCT?