yiskw713/scorecam

the re-implementation of Score CAM with pytorch

Score CAM with pytorch

The re-implementation of Smooth Grad-CAM++ with pytorch. This repo also includes the code of CAM, Grad-CAM, Grad-CAM++, Smooth Grad-CAM++.

Requirements

• python 3.x
• pytorch >= 0.4
• pillow
• numpy
• opencv
• matplotlib

How to use

You can use the CAM, GradCAM, GradCAM++, Smooth Grad-CAM++ and Score CAM as a model wrapper described in cam.py. Please see demo.ipynb for the detail.

Results

image
CAM
Grad-CAM
Grad-CAM++
Smooth Grad-CAM++
Score CAM

References

• Score-CAM: Improved Visual Explanations Via Score-Weighted Class Activation Mapping, Haofan Wang, Mengnan Du, Fan Yang, Zijian Zhang [paper]
• Smooth Grad-CAM++: An Enhanced Inference Level Visualization Technique for Deep Convolutional Neural Network Models,
  Daniel Omeiza, Skyler Speakman, Celia Cintas, Komminist Weldermariam [paper]
• Learning Deep Features for Discriminative Localization, Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, Antonio Torralba [paper]
• Grad-CAM: Visual explanations from deep networks via gradient-based localization, Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra, [paper]
• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks, Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader and Vineeth N Balasubramanian[paper]

scorecam's Issues

'NoneType' object has no attribute 'parameters'

For MobileNet_V2, and EfficientNet_B1, I am getting this AttributeError. I used(line 63, cam.py) for weight_fc
self.model._modules.get('classsifier').parameters())[0].to('cpu').data

Info regarding normalization

In your scorecam implementation, why are you normalizing all the activations together? Why not normalize each activation map independently in a loop? Are these two the same?

ScoreCAM fix batch proposal

I found out that the ScoreCAM code is not set for batched inputs. This is my attempt to fix this thing:

    def forward(self, x):
        with torch.no_grad():
            B, _, H, W = x.size()
            device = x.device
            self.model.zero_grad()
            score = self.model(x)
            prob = torch.nn.functional.softmax(score, dim=1)
            _, idx = torch.max(prob, dim=1)
            prob = prob[:, 1].unsqueeze(-1).detach().float()

            # put activation maps through relu activation
            # because the values are not normalized with eq.(1) without relu.
            self.activations = torch.nn.functional.relu(self.values.activations).to('cpu').clone()
            self.activations = torch.nn.functional.interpolate(self.activations, (H, W), mode='bilinear')
            _, C, _, _ = self.activations.shape
            # normalization
            act_min, _ = self.activations.view(1, C, -1).min(dim=2)
            act_min = act_min.view(1, C, 1, 1)
            act_max, _ = self.activations.view(1, C, -1).max(dim=2)
            act_max = act_max.view(1, C, 1, 1)
            denominator = torch.where((act_max - act_min) != 0., act_max - act_min, torch.tensor(1.))
            self.activations = self.activations / denominator

            # generate masked images and calculate class probabilities
            probs = []
            #random_channels = np.random.randint(C, size=10)
            for i in range(0, C):
                mask = self.activations[:, i, :, :].unsqueeze(1)
                mask = mask.to(device)
                masked_x = x * mask
                score = self.model(masked_x)
                score = torch.nn.functional.softmax(score, dim=1)
                score = torch.amax(score, dim=1).to('cpu').data
                probs.append(score)

            probs = torch.stack(probs)
            weights = probs.view(B, C, 1, 1)

            cam = (weights * self.activations[:, :C, :, :]).sum(1, keepdim=True)
            cam = torch.nn.functional.relu(cam)
            cam -= torch.min(cam)
            cam /= torch.max(cam)

        return cam.data, idx.item()

Let me know if thee is something wrong my solution.

Object detection and Segmentation

@yiskw713 hi thanks for opensourcing the code , can we use ScoreCAM for models like MaskRCNN/Deeplab and FasterRCNN,Yolov2, RetinaNet

Size mismatch error

cam, idx = wrapped_model(tensor)

I get the following error at this step (no changes made to the notebook) : RuntimeError: size mismatch, m1: [1 x 30720], m2: [512 x 1000] at c:\programdata\miniconda3\conda-bld\pytorch_1533090265711\work\aten\src\th\generic/THTensorMath.cpp:2070