fhvilshoj/torchlrp

A PyTorch 1.6 implementation of Layer-Wise Relevance Propagation (LRP).

Implementation of LRP for pytorch

PyTorch implementation of some of the Layer-Wise Relevance Propagation (LRP) rules, [1, 2, 3], for linear layers and convolutional layers.

The modules decorates torch.nn.Sequential, torch.nn.Linear, and torch.nn.Conv2d to be able to use autograd backprop algorithm to compute explanations.

Installation

To install requirements, refer to the requirements.yml file.

If you use conda, then you can install an environment called torchlrp by executing the following command:

> conda env create -f requirements.yml

To be able to import lrp as below, make sure that the TorchLRP directory is included in your path.

Usage

The code can be used as follows:

import torch
import lrp

model = Sequential(
    lrp.Conv2d(1, 32, 3, 1, 1),
    torch.nn.ReLU(),
    torch.nn.MaxPool2d(2, 2),
    torch.nn.Flatten(),
    lrp.Linear(14*14*32, 10)
)

x = ... # business as usual
y_hat = model.forward(x, explain=True, rule="alpha2beta1")
y_hat = y_hat[torch.arange(batch_size), y_hat.max(1)[1]] # Choose maximizing output neuron
y_hat = y_hat.sum()

# Backward pass (do explanation)
y_hat.backward()
explanation = x.grad

Implemented rules:

Rule	Key	Note
epsilon-rule	"epsilon"	Implemented but epsilon fixed to 1e-1
gamma-rule	"gamma"	Implemented but gamma fixed to 1e-1
epsilon-rule	"epsilon"	gamma and epsilon fixed to 1e-1
alpha=1 beta=0	"alpha1beta0"
alpha=2 beta=1	"alpha2beta1"
PatternAttribution (all)	"patternattribution"	Use additional argument pattern=patterns_all
PatternAttribution (positive)	"patternattribution"	Use additional argument pattern=patterns_pos
PatternNet (all)	"patternnet"	Use additional argument pattern=patterns_all
PatternNet (positive)	"patternnet"	Use additional argument pattern=patterns_pos

To compute patterns for the two PatternAttribution methods, import lrp.patterns and call

import lrp.patterns.*
patterns_all = fit_patternnet(model, train_loader)
patterns_pos = fit_patternnet_positive(model, train_loader)

Note: Biases are currently ignored in the alphabeta-rule implementations.

Trace intermediate relevances

Thanks to francescomalandrino, you can now also trace the intermediate relevances by enabling traces:

... 
lrp.trace.enable_and_clean()
y_hat.backward()
all_relevances=lrp.trace.collect_and_disable()

for i,t in enumerate(all_relevances):
    print(i,t.shape)

MNIST

For a complete running example, please see examples/explain_mnist.py. The code generates this plot:

To run the example code, simply activate the conda environment and execute the code from the root of the project:

> conda activate torchlrp
> python examples/explain_mnist.py

VGG / ImageNet

It is also possible to use this code for pretrained vgg models from torchvision, by using the lrp.convert_vgg function to convert torch.nn.Conv2d and torch.nn.Linear layers to lrp.Conv2d and lrp.Linear, respectively.

It takes a bit to make the vgg example work. First, you need An imagenet dataloader. In the code, we use the dataloader from the torch_imagenet repo. You could also make your own.

The most interesting parts is converting the torch vgg models, such that they can be explained. To do so, do as follows:

vgg = torchvision.models.vgg16(pretrained=True).to(device)
vgg.eval()
lrp_vgg = lrp.convert_vgg(vgg).to(device)

The lrp_vgg model will then have the same parameters as the original network. Afterwards, explanations can be produced as the example above.

Note:

The code example reads a config.ini file from the root of this project. In that file you can specify the parent of the torch_imagenet repo such that the correct dataloader is loaded:

[DEFAULT]
ImageNetDir = /home/user/example/data

Possible bugs

Fixed - Description

• According to [3] Section 10.3.2, it is apparently a good idea to use gradient of average pooling for LRP backpropagation. I have started to implement this but not finished, as I didn't need it so far.

• _Fixed in commit 4277098f4f37a81ae9a21154c8cba49cae918770__. Judging from the plot, something is probably wrong with the positive PatternAttribution and PatternNet, as it doesn't compare visually to, e.g., this implementation.

References

[1] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R. and Samek, W., 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), p.e0130140.
[2] Kindermans, P.J., Schütt, K.T., Alber, M., Müller, K.R., Erhan, D., Kim, B. and Dähne, S., 2017. Learning how to explain neural networks: Patternnet and patternattribution. arXiv preprint arXiv:1705.05598.
[3] Montavon, G., Binder, A., Lapuschkin, S., Samek, W. and Müller, K.R., 2019. Layer-wise relevance propagation: an overview. In Explainable AI: interpreting, explaining and visualizing deep learning (pp. 193-209). Springer, Cham.

torchlrp's Issues

Relevance of model parameters

Is there a way to get the relevance of model parameters, in addition to the inputs (and intermediate results) thereof?

The PLOS paper ("On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation") mentions per-neuron relevance conservation, hence, I assume it would make sense to keep track of the incoming (or outgoing) relevance of each neuron (and, more generally, each DNN parameter), and make that accessible to the users of the library.

Does that make sense? Thanks!

Relevance of hidden layers

Is there a simple (or simple-ish) way to obtain the relevance scores of hidden layers, i.e., what is visualized in the cell starting with "for i,l in enumerate([31,21,11,1]):" here?
https://git.tu-berlin.de/gmontavon/lrp-tutorial

The library sure computes them, but I could not find a way to recover those. Thanks!

Update the conv_transpose2d usage?

I implemented the excellent scripts and found that conv_transpose2d does not work properly for my own work. So I updated it in functional.conv line27 as follows:

    # relevance_input  = F.conv_transpose2d(relevance_output, weight, None, padding=1)
    if ctx.stride[0] >= 2:
        output_padding = 1
    else:
        output_padding = 0
    relevance_input  = F.conv_transpose2d(relevance_output, weight, None, stride=ctx.stride, padding=ctx.padding, output_padding=output_padding)

and also here:

        def f(X1, X2, W1, W2, ctx): 

            # Z1  = F.conv2d(X1, W1, bias=None, stride=1, padding=1) 
            # Z2  = F.conv2d(X2, W2, bias=None, stride=1, padding=1)
            Z1 = F.conv2d(X1, W1, None, ctx.stride, ctx.padding, ctx.dilation, ctx.groups)
            Z2 = F.conv2d(X2, W2, None, ctx.stride, ctx.padding, ctx.dilation, ctx.groups)
            Z   = Z1 + Z2

            rel_out = relevance_output / (Z + (Z==0).float()* 1e-6)

            # t1 = F.conv_transpose2d(rel_out, W1, bias=None, padding=1) 
            # t2 = F.conv_transpose2d(rel_out, W2, bias=None, padding=1)
            if ctx.stride[0] >= 2:
                output_padding = 1
            else:
                output_padding = 0
            t1 = F.conv_transpose2d(rel_out, W1, None, stride=ctx.stride, padding=ctx.padding, output_padding=output_padding)
            t2 = F.conv_transpose2d(rel_out, W2, None, stride=ctx.stride, padding=ctx.padding, output_padding=output_padding)

            r1  = t1 * X1
            r2  = t2 * X2

            return r1 + r2

Not sure if this is my own issue, but the above change fixed my problem.

Apply rho and incr for weights in forward pharse

HI. Thank you for your great work !
I have a question that Do you apply rho and incr function for weights of layer while calculate in forward ?
Thank you so much !

LRP for unet with upsample or ConvTranspose2d layer model

I want to use LRP to explain the semantic segmentation task using Unet model (Pytorch). I tested the LRP in captum but not support nn.Upsample and nn.ConvTranspose2d. I would like to know if the semantic segmentation model like Unet can be supported, and if not, how should it be implemented? Any help would be appreciated!

issue with backward when use resnet

I just found an issue with the resnet conversion. Since the backward function has not been rewritten the size of the tensors are inconsistent.

LRP for resnet model

Thank for your works!
I see that you implement LRP for vgg model. But vgg is simple model with single Sequential and does not have residual connection. Could you help me to implement LRP for complex model, such as ResNet?
Thank you so much!