arrasl/lrp_for_lstm

Layer-wise Relevance Propagation (LRP) for LSTMs.

Description

This repository provides a reference implementation of Layer-wise Relevance Propagation (LRP) for LSTMs, as initially proposed in the paper Explaining Recurrent Neural Network Predictions in Sentiment Analysis, L. Arras, G. Montavon, K.-R. Müller and W. Samek WASSA@EMNLP'2017 [arXiv:1706.07206].

Additionally it includes an implementation of Sensitivity Analysis (SA) and Gradient x Input (GI), i.e. of gradient-based relevances.

Note that our implementation is generic and can be easily extended to unidirectional LSTMs, or to other application domains than Natural Language Processing.

A few hints on how to apply and extend the code to your needs can be found here.

Dependencies

Python>=3.5 + Numpy + Matplotlib, or alternatively, simply install Anaconda.

Using conda, you can e.g. create a Python 3.6 environment: conda create -n py36 python=3.6 anaconda

Then activate it with: source activate py36

Before being able to use the code, you might need to run in the terminal: export PYTHONPATH=$PYTHONPATH:$pwd

Usage

The folder model/ contains a word-based bidirectional LSTM model, that was trained for five-class sentiment prediction of phrases and sentences on the Stanford Sentiment Treebank (SST) dataset, as released by the authors in Visualizing and Understanding Neural Models in NLP, J. Li, X. Chen, E. Hovy and D. Jurafsky, 2016

The folder data/ contains the test set sentences of the Stanford Sentiment Treebank (SST), preprocessed by lowercasing, as was done in Visualizing and Understanding Neural Models in NLP, J. Li, X. Chen, E. Hovy and D. Jurafsky, 2016

The notebook run_example.ipynb provides a usage example of the code, its performs LRP, SA and GI on a test sentence.
(To correctly render the notebook on GitHub you can copy the notebook's URL to nbviewer)

Follow-up works

Publications

Here are some follow-up works using LRP on various recurrent neural network models and tasks (non-exhaustive list):

[1] Evaluating Neural Network Explanation Methods using Hybrid Documents and Morphosyntactic Agreement, N. Poerner, H. Schütze and B. Roth, ACL 2018 [arXiv:1801.06422] [code]

[2] Explaining Therapy Predictions with Layer-Wise Relevance Propagation in Neural Networks, Y. Yang, V. Tresp, M. Wunderle and P.A. Fasching, IEEE ICHI 2018 [preprint] [code]

[3] Analyzing Neuroimaging Data Through Recurrent Deep Learning Models, A.W. Thomas, H.R. Heekeren, K.-R. Müller and W. Samek, Frontiers in Neuroscience 2019 [blog]

[4] Evaluating Recurrent Neural Network Explanations, L. Arras, A. Osman, K.-R. Müller and W. Samek, BlackboxNLP@ACL 2019 [arXiv:1904.11829] [oral presentation slides]

[5] Explaining and Interpreting LSTMs, L. Arras, J. Arjona-Medina, M. Widrich, G. Montavon, M. Gillhofer, K.-R. Müller, S. Hochreiter and W. Samek, Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer LNCS vol 11700 2019 [arXiv:1909.12114]

[6] Evaluating Explanation Methods for Deep Learning in Security, A. Warnecke, D. Arp, C. Wressnegger and K. Rieck, IEEE European Symposium on Security and Privacy 2020 [arXiv:1906.02108] [code_main] [code_lrp]

Software

LRP for LSTMs self-contained implementation in Tensorflow 2.1 (allowing efficient batch and GPU processing) by Alexander Warnecke: github repository

Acknowledgments

Visualizing and Understanding Neural Models in NLP, J. Li, X. Chen, E. Hovy and D. Jurafsky, code

Visualizing and Understanding Neural Models in NLP, J. Li, X. Chen, E. Hovy and D. Jurafsky, NAACL 2016

Long Short Term Memory Units, repo by W. Zaremba

Stanford Sentiment Treebank (SST), dataset by R. Socher et al., 2013

Citation

If you find this project useful, please cite our original paper (or one of our follow-up publications see above [4,5]):

@INPROCEEDINGS{arras2017,
    title     = {{Explaining Recurrent Neural Network Predictions in Sentiment Analysis}},
    author    = {Leila Arras and Gr{\'e}goire Montavon and Klaus-Robert M{\"u}ller and Wojciech Samek},
    booktitle = {Proceedings of the EMNLP 2017 Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis},
    year      = {2017},
    pages     = {159-168},
    publisher = {Association for Computational Linguistics},
    doi       = {10.18653/v1/W17-5221},
    url       = {https://www.aclweb.org/anthology/W17-5221}
}

More information

For other research and projects involving LRP, you can visit the website heatmapping.org

lrp_for_lstm's Issues

Relevance sensitive sentence representations

Hello @ArrasL,
Thank you for open sourcing your implementation of LRP.
Your work here and the comparative analysis in the paper is really interesting.
I tried to add LIME, LIMSSE to the list of methods and conduct similar experiments as suggested in the latter paper.
I rechecked the scores of my implementation of gradient based methods with yours in run_example.ipynb.

I used relevance as weights for creating sentence representations, I used them in 2 settings,
a. as raw scores
b. normalized values (softmax)
I have attached the plot for your reference. I wanted to check with you how you had used them because my plot for Gradient Input is not close to the one that we see in the paper.

Any guidelines or directions would be extremely helpful as i wish to reproduce the results.

Enhancement: work for 2 classes

can this be expanded for 2 classes instead of 5 sentiment classes? Would it require to change LSTM_bidi file?

regression

Is it possible to adapt this code in order to deal with a regression problem? Let's say that each word of a sentence is a timestep of a timeseries and the number of words actually represents the history feed to the LSTM.

Using LRP for clickstream data

I've trained a bidirectional LSTM model on clickstream data sequences, and I want to explain the predictions using LRP. Would it be possible to adjust this implementation, so it can be used on sequential data?
A sequence, for example, looks like: [5,4,3,4,4,3,3,4,5]

will it work on windows computer?

will it work on windows computer?
is Torch or TF needed?

error for run for first line

---

ModuleNotFoundError Traceback (most recent call last)
in ()
----> 1 from code.LSTM.LSTM_bidi import *
2 from code.util.heatmap import html_heatmap
3
4 import codecs
5 import numpy as np

ModuleNotFoundError: No module named 'code.LSTM'; 'code' is not a package

Can you please define the train() function for LSTM_bidi class?

The current class only supports loading of pretrained models and I am having a little trouble changing the init and implementing the train() function to train this model from scratch.

Thanks,
Sharan

no colores for heatmap in web based demo as shown in paper

A question about single LSTM

Hello, I try to write a LRP for single LSTM, I found that the relevance focus on the first sequence. Just like this,

I don't think that the first sequence is more important. Could you please expain it?
Thank you~

Stacked/multilayer LSTM with Attention

My model consists of a multi-layer LSTM, an attention layer and a linear layer for binary classification. I need the stacked LSTMs and attention layer to achieve decent performance for a relatively complex task. I'm chugging through your detailed explanation for issue #8 (thank you!) for attention layer.

How would I propagate back the scores through the stacked LSTMs? Any guidance and advice would be much appreciated.

can we train bidirectional model?

is it possible to train the bilsmt mode, does it require changing the model file with new weights?

Architecture differences: Keras LSTM architecture to LSTM_bidi.py

I've been reviewing the bidirectional LSTM used in this example because I'd like to apply the LRP technique to my own model. However, I noticed that the weights and biases in the Keras LSTM are different from here and I would like to verify my observations are correct.

1. In this example, there are weights on the LSTM outputs from the Left and Right LSTM. Is that the implementation of the Dense layer?

2. In this example, there are separate biases for the h_Left/h_Right arrays and the x_Left/x_Right arrays? Keras provides only a single bias array for the two and I was wondering whether there are different LSTM architectures being followed. I also noticed that alewarne (at https://github.com/alewarne/Layerwise-Relevance-Propagation-for-LSTMs/blob/master/lstm_network.py) implemented this code with a single bias array.

Thanks for providing this reference code - I appreciate it.

Trained LSTM model adaptation

Hello! May I assume the trained LSTM in the example consists of 1 LSTM layer? Can it be adapted into stacked LSTM scenario? Thx.

How to propropagate individual hidden layer relevance scores of attention through LSTM?

My model consists of an encoder LSTM, an attention layer and a Linear decoder layer for the task of binary classification. So far I have propagated LRP all the way till the hidden layer inputs to the attention layer and am not sure how to propagate each hidden layer relevance to the input layer through the encoder LSTM.

This repo only assumes that the model is a simple encoder LSTM, and a linear decoder which takes the final hidden state as input to produce the output class, if I am right.

How can I propagate these individual hidden layer scores throught the LSTM using this approach? If I only try and propagate the last hidden layer scores through the LSTM using this code it 1. doesn't take the other hidden state scores into account 2. assumes that the attention layer only takes the last hidden state as the input.

I understand that this may be an open question, any help/advice on how to proceed will be greatly appreciated.