kgourgou/setfit-integrated-gradients

Hacking SetFit so that it works with integrated gradients.

setfit-integrated-gradients

Hacking SetFit so that it works with integrated gradients. See demo.ipynb for an example.

Integrated gradients is a way to explain the decisions of the model by scoring what parts of the input influenced a particular decision.

Note: This mini-library only supports binary classification with a scikit-learn logistic-regression head.

Installation

pip install -e . 

The "-e" switch installs the package in develop mode.

Notes

I wrote this mini-library before SetFit 0.6.0. At the time, there was no SetFitHead class yet, so I just took the sklearn LogisticRegression and passed its parameters to an equivalent Torch class. I did my best to break the forward pass of SetFit into pieces so that I can push gradients through the head and up to the token embeddings.

Attributions from integrated gradients are computed per token and then averaged to get word-level attributions.

I'm leaving this here for posterity and in case it is useful to others for further hacking.

setfit-integrated-gradients's Issues

Need to fix this doctext.

setfit-integrated-gradients/setfit_ig/setfit_extensions.py

Line 91 in 008005e

- positive_class_probability, grads, sentence_token_embedding

TODO: One pass instead of for loop.

setfit-integrated-gradients/setfit_ig/integrated_gradients.py

Line 105 in ccf8739

for t, w in tqdm(list(zip(integration_steps, weights))):

There's a weird bug when running with large sequences of tokens

Will reproduce here when I get some time, but basically the SetFit model's predict_proba and the model_pass function from this module sometimes return very different probabilities for the same input.

multiclass classification

Hi,

I know this only supports binary classification, but if there's a quick solution, would you be able to show how I can change it to support multiclass? I'm just looking to get attributions for the top class.