christophm/interpretable-ml-book

Book about interpretable machine learning

Interpretable machine learning

Explaining the decisions and behaviour of machine learning models.

Summary

You can find the current version of the book here: https://christophm.github.io/interpretable-ml-book/

This book is about interpretable machine learning. Machine learning is being built into many products and processes of our daily lives, yet decisions made by machines don't automatically come with an explanation. An explanation increases the trust in the decision and in the machine learning model. As the programmer of an algorithm you want to know whether you can trust the learned model. Did it learn generalizable features? Or are there some odd artifacts in the training data which the algorithm picked up? This book will give an overview over techniques that can be used to make black boxes as transparent as possible and explain decisions. In the first chapter algorithms that produce simple, interpretable models are introduced together with instructions how to interpret the output. The later chapters focus on analyzing complex models and their decisions. In an ideal future, machines will be able to explain their decisions and make a transition into an algorithmic age more human. This books is recommended for machine learning practitioners, data scientists, statisticians and also for stakeholders deciding on the use of machine learning and intelligent algorithms.

The book is automatically built from the master branch and pushed to gh-pages by GitHub Actions.

Contributing

See how to contribute

Rendering the book

Clone the repository.

git clone [email protected]:christophM/interpretable-ml-book.git

Make sure all dependencies for the book are installed. This book has the structure of an R package, so dependencies can be installed easily, only R and the devtools library is required. Start an R session in the folder of the book repository and type:

devtools::install_deps()

For rendering the book, start an R session and type:

setwd("manuscript")
# first, generate the references
source("../scripts/references.R")
bookdown::render_book('.', 'bookdown::gitbook')

After rendering, the HTML files of the book will be in the "_book" folder. You can either double-click index.html directly or, of course, do it in R:

browseURL('_book/index.html')

Notes on Printing with lulu.com

• Export from Leanpub in 7.44" x 9.68" 18.9cm x 24.6cm
• For cover: 7.565 x 9.925", 19.226 x 25.224cm, see recommended sizes
• Font for front cover: Francois One

Writing

Stuff that both works for leanpub and for bookdown:

• Titles start with #, subtitles with ## and so on.
• Titles can be tagged using {#tag-of-the-title}
• Chapters can be referenced by using [text of the link](#tag-of-the-title)
• Figures can be referenced by using [text of the link](#fig:tag-of-r-chunk-that-produced-figure)
• Start and end mathematical expressions with $ (inline) or with $$ (extra line). Will be automatically changed for leanpub with a regexpr. Conversion script only works if no empty spaces are in the formula.
• Leave empty lines between formulas and text (if formula not inline). Formulas (with $$ ... $$) should be in one line and not over multiple lines (due to parser).
• References have to be written like this: [^ref-tag] and must be at the end of the respective file with [^ref]: Details of the reference .... Make sure the space is included. References are collected in 10-reference.Rmd with the script references.R. Make sure not to use [^ref-tag]: anywhere in the text, only at the bottom for the actual reference.

Printing for proofreading with extra line spacing: Build HTML book, go to manuscript/_book/libs/gitbook*/css/style.css, change line-height:1.7 to line-height:2.5, open local html with chrome, print to pdf with custom margin.

Changelog

All notable changes to the book will be documented here.

v2.0 (IN PROGRESS) [html version]

• Added "Preface by the Author" chapter
• Started section on neural network interpretation
• Added chapter on feature visualization
• Added SHAP chapter
• Added Anchors chapter
• Fixed error in logistic regression chapter: Logistic regression was predicting class "Healthy", but interpretation in the text was for class "Cancer". Now regression weights have the correct sign.
• Renamed Feature Importance chapter to "Permutation Feature Importance"
• Added chapter about functional decomposition
• Rearranged interpretation methods by local, global and deep learning (before: model-agnostic, example-based, deep learning)
• Errata:
  • Chapter 4.3 GLM, GAM and more: Logistic regression uses logit, not logistic function as link function.
  • Chapter Linear models: Formula for adjusted R-squared was corrected (twice)
    • Chapter Decision Rules: Newly introduced mix up between Healthy and Cancer in OneR chapter was fixed.
    • Chapter RuleFit: The importance of the linear term in the total importance formulate was indexed with an $l$ instead of $j$.
• Updated images

v1.1 (2019-03-23) [Print version, ebook version]

• Fixes wrong index in Cooks Distance summation (i -> j)
• fixed boxplot formula (1.5 instead of 1.58)
• Change to colorblind-friendly color palettes (viridis)
• Make sure plots work in black and white as well
• Extends counterfactual chapter with MOC (by Susanne Dandl)

v1.0 (2019-02-21)

• Extensive proofreading and polishing

v0.7 (2018-11-21)

• Renamed Definitions chapter to Terminology
• Added mathematical notation to Terminology (former Definitions) chapter
• Added LASSO example
• Restructured lm chapter and added pros/cons
• Renamed "Criteria of Interpretability Methods" to "Taxonomy of Interpretability Methods"
• Added advantages and disadvantages of logistic regression
• Added list of references at the end of book
• Added images to the short stories
• Added drawback of shapley value: feature have to be independent
• Added tree decomposition and feature importance to tree chapter
• Improved explanation of individual prediction in lm
• Added "What's Wrong With my Dog" example to Adversarial Examples
• Added links to data files and pre-processing R scripts

v0.6 (2018-11-02)

• Added chapter on accumulated local effects plots
• Added some advantages and disadvantages to pdps
• Added chapter on extending linear models
• Fixed missing square in the Friedman H-statistic
• Added discussion about training vs. test data in feature importance chapter
• Improved the definitions, also added some graphics
• Added an example with a categorical feature to PDP

v0.5 (2018-08-14)

• Added chapter on influential instances
• Added chapter on Decision Rules
• Added chapter on adversarial machine examples
• Added chapter on prototypes and criticisms
• Added chapter on counterfactual explanations
• Added section on LIME images (by Verena Haunschmid)
• Added section on when we don't need interpretability
• Renamed chapter: Human-style Explanations -> Human-friendly Explanations

v0.4 (2018-05-23)

• Added chapter on global surrogate models
• Added improved Shapley pictograms
• Added acknowledgements chapter
• Added feature interaction chapter
• Improved example in partial dependence plot chapter
• The weights in LIME text chapter where shown with the wrong words. This has been fixed.
• Improved introduction text
• Added chapter about the future of interpretability
• Added Criteria for Interpretability Methods

v0.3 (2018-04-24)

• Reworked the Feature Importance Chapter
• Added third short story
• Removed xkcd comic
• Merged introduction and about the book chapters
• Added pros & cons to pdp and ice chapters
• Started using the iml package for plots in ice and pdp
• Restructured the book files for Leanpub
• Added a cover
• Added some CSS for nicer formatting

v0.2 (2018-02-13)

• Added chapter about Shapley value explanations
• Added short story chapters
• Added donation links in Preface
• Reworked RuleFit with examples and theory.
• Interpretability chapter extended
• Add chapter on human-style explanations
• Making it easier to collaborate: Travis checks if book can be rendered for pull requests

v0.1 (2017-12-03)

• First release of the Interpretable Machine Learning book

interpretable-ml-book's Issues

List alternatives and software for feature importance measures

I am extending all chapters with a section for software implementations and to alternative algorithms (also with software implementation). The software can be any free and open source software: R, Python, Weka, ...

You can help out by posting links to alternatives and software implementations of feature importances measure algorithms as comments to this issue.

Empty _book directory after build

After installing all packages and running all commands recommended in README file I have no error messages in console, but no resulting HTML files in _book directory.

What can you recommend to look at (for R beginner)?

Figure text for figure 5.10 and 5.11 is the same.

I guess this text is for figure 5.10 and not figure 5.11?

FIGURE 5.11: The interaction strength for each feature with all other features for a random forest predicting the probability of cervical cancer. The number of diagnosed sexually transmitted diseases has the highest interaction effect with all other features, followed by the number of pregnancies.

List shapley value alternatives and software

I am extending all chapters with a section for software implementations and to alternative algorithms (also with software implementation). The software can be any free and open source software: R, Python, Weka, ...

You can help out by posting links to papers and software implementations of the shapley value for machine learning as comments to this issue.

nonconsistent value in caption and chart

Hi,
Caption to this chart:

interpretable-ml-book/manuscript/05.3-agnostic-ice.Rmd

Line 91 in 062d1e2

```{r ice-cervical-centered, fig.cap=sprintf("Centered ICE plot for predicted cancer probability by age. Lines are fixed to 0 at age %i. Compared to age %i, the predictions for most women remain unchanged until the age of 45 where the predicted probability increases.", min(cervical_subset$Age), min(cervical_subset$Age))}

seems to be ok ('Lines are fixed to 0 at age 13. Compared to age 13'), but it is inconsistent with y-axis label: 'Cancer probability difference to age 18'.

Decision Trees Chapter

On 4.4.2. on the example you talk about purity (of Gini index).
But you don't reference it anywhere else, which might add to the confusion of the reader.

I have some background so I know what you are talking about, but others will not!

So, either reference a link to Gini Index and what is this purity,
or just give the intuition that purity is relative to the subsets, and can be dirty if it has instances that do not "belong" to that group.

As always, keep up the good work,
Pedro

List alternatives and software for partial dependence plots

I am extending all chapters with a section for software implementations and to alternative algorithms (also with software implementation). The software can be any free and open source software: R, Python, Weka, ...

You can help out by posting links to alternatives and software implementations of partial dependence plots algorithms as comments to this issue.

Make the "next page >" symbol more obvious on the Preface page

On my cell phone the next page symbol ">" is a light grey and very small. Even though I had my cell read "Desktop site" no menu or side bar was visible. I almost ignored your book since I could only see the preface.

Suggest making a regular link on the preface page to the next page.

Reference List

As a reader I would like to have all references in one place to look them up.

I stumbeled upon this when I read Chapter 2.1.
In the text it references Miller 2017, but this is not in the footnotes. Ultimately, I found it in the introduction of Chapter 2.
To prevent a search ike mine, it would be nice if all references could be found in a common place.

Additions

Brilliant work.

---

Adding LIME advantages and disadvantages.

Explaining what is meant by sparse explanations - Shapley Value section.

Looking forward to your future work.

PDF version

Hi Christoph,

lovely, thanks a lot for the book.
I might sound semi-old asking for the possibility to build a PDF version of your book, but in order to read and make notes on e.g. an iPad or other tablet this would make a lot of sense to me.

Is there any way to do so?

Model-agnostic Permutation Feature Importance

Hi Christoph, and thank you for the great book!

In the Permutation Feature Importance section you say:

I haven’t found any paper that generalises permutation feature importance, so that it can be applied model-agnostic. Please drop me a mail if you know about model-agnostic feature importance.

But you propose a model-agnostic version of the algorithm in the same section. Did you forget to remove that couple of sentences? Or am I missing something?

Add confidence interval formula to linear model chapter

Linear Regression :
A formula for the confidence interval of estimated weights could be useful, especially to illustrate the dependency of its lenght to the number of instances.

"Improve" is written twice in the last part of 6.2

Can't see whole table for 4.3.2.

Here is an image

This leads me to believe that the table is incomplete.

I'm using Chrome on a Mac. Also tested on Brave browser (firefox based I believe).
Maybe table is just too long!

Cheers

python version of scrpit

Hi, Chris

Do you have python version of your scripts for this book?

Thank you

List CART (decision tree) alternatives and software

I am extending all chapters with a section for software implementations and to alternative algorithms (also with software implementation). The software can be any free and open source software: R, Python, Weka, ...

You can help out by posting links to papers and software implementations of decision tree algorithms as comments to this issue.

Error in the RuleFit -> Guidelines -> Disadvantages

In the Disadvantages section of the RuleFit Guidelines you write:

"For example one decision rule (feature) for the bike prediction could be: “temp > 15” and another rule could be “temp > 10 & weather=‘GOOD’”. When the weather is good and the temperature is above 10 degrees, the temperature is automatically also always bigger then 15, which means in the cases where the second rule applies, the first one also always applies."

I think that you have swapped the numbers around and the rules should be:

• temp > 10, and
• temp > 15 & weather=‘GOOD’.

temp > 10 & weather=‘GOOD’ does not imply temp > 15, e.g. weather=‘GOOD’ & temp = 13.

Possible mistakes in lime-text-explanations example.

Hai, thanks for the efforts on the interpretation of ML methods. In Ch5, the last example on lime-text-explanations, it seems that the generated image has both instance labeld as spam. At least in the HTML version. Some minor mistakes?

Feature importance disadvantages

I think there could be added some disadvantages in the feature importance chapter.

There is this paper: https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-8-25

Moreover there is the problem of correlated variables. If two variables are highly correlated, their feature importance could be massively decreased, because one variable can substitute the other variable (for example when we grow a lot of trees). Here is a good blogpost about it, there are for sure some papers about this topic, I guess: https://freakonometrics.hypotheses.org/20545

Btw. you should cite Breiman, 2001, not 2011, as he already died in 2005. ;)

LIME: better explain how different features can be used

Better explain the following statement: "the explanations created with local surrogate models can use other features than the original model.". Can I use a feature the model hasn't been trained on ? How would that work for tabular data ?

A good example: model is trained on PCA components of features, but explanation is based on features directly.

Write a chapter about your favorite tree ensemble interpretability method

Tree ensembles like boosted trees (e.g. xgboost) and random forests perform extremely well on tabular data.

A few interpretability methods are created to interpret specifically tree ensembles.

Some pointers:

• https://github.com/andosa/treeinterpreter
• https://github.com/lantanacamara/lightgbmExplainer
• https://github.com/AppliedDataSciencePartners/xgboostExplainer
• https://github.com/araastat/reprtree

Leave a comment if you are interested in writing a chapter about a tree ensemble-specific method!

explanation on how weights calculated in 5.7.2.1 Example

Very good book, thanks for writing it! :)

Just one issue: would you please give us some explanation on how the weights were calculated in the 5.7.2.1 Example. (second table, last column, after the probs). I understand it is a distance between the generated sample text and the original sentence. Could you please give us some hints on how the distance between sentence (or set of words) were calculated?

Sorry, this issue is addressed somewhere.

List linear regression variants and software

I am extending all chapters with a section for software implementations and to alternative algorithms (also with software implementation). The software can be any free and open source software: R, Python, Weka, ...

You can help out by posting links to papers and software implementations of the linear regression model as comments to this issue.

Confusing difference for Figure 5.22

I guess it's because of rounding but at first I was a bit confused that the "difference" between the actual prediction (0.43) and the average prediction (0.03) is 0.41. I think it might confuse readers and could be explained in the text. If you agree I can add a sentence to explain it.

Add subset (m) to notation in Shapley value chapter

$x^m_{∗+j}$ instead of $x^{∗+j}$
and $x^m_{∗-j}$ instead of $x^{∗-j}$

Blogpost about interesting package

https://www.r-bloggers.com/dalex-understand-a-black-box-model-conditional-responses-for-a-single-variable/

List local and global surrogate models (like LIME or tree surrogates) alternatives and software

I am extending all chapters with a section for software implementations and to alternative algorithms (also with software implementation). The software can be any free and open source software: R, Python, Weka, ...

You can help out by posting links to papers and software implementations of local and global surrogate models (like LIME or tree surrogates) as comments to this issue.

5.3.2 Bike rentals (Classification) should be Bike rentals (Regression)

as title

Significance of gbm interactions

In section 5.4.5 you say:
"It is unclear whether an interaction is significantly greater than 0. We would need to conduct a statistical test, but this test is not (yet) available in a model-agnostic version."
I know that Friedman and Propescu 2008 recommend a method to create a null distribution in order to determine the significance of interactions for a gbm model. Do you know if this method is implemented in R anywhere, even if just for gbm models?

Write a chapter about your favorite deep learning interpretability method

There are many interpretability methods for deep neural networks.
The field is very young and most methods are quite new and under development. It's hard for a single person to keep track. This issue serves as a placeholder for interpretability methods specific for deep learning.

Leave a comment if you are interested in adding a chapter. Any method that helps interpreting neural networks is interesting. A good starting point is to take a paper or some code library and explain it in simpler words, possibly with an example.

Some possible starting points:

Software:

• https://medium.com/merantix/picasso-a-free-open-source-visualizer-for-cnns-d8ed3a35cfc5
• https://github.com/yosinski/deep-visualization-toolbox
• https://www.oreilly.com/ideas/interpretability-via-attentional-and-memory-based-interfaces-using-tensorflow
• https://github.com/taolei87/rcnn

Scientific Paper:

• Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations
• Rationalizing neural predictions
• Generating Visual Explanations
• Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations
• Methods for interpreting and understanding deep neural networks

Leave a comment if you are interested in writing a chapter!

Add GAMs to chapter Interpretable Models

Hi Christoph,

First of all a big bravo and thank you for writing such an interesting book and making it open to everyone!

I would suggest adding a section on generalized additive models, although not very popular in the data science community, these methods are very powerful and simple for balancing accuracy vs. interpretability.

You probably saw the Caruana talks abou GAMS for HealthCare, in any case here goes the link: https://www.microsoft.com/en-us/research/video/intelligible-machine-learning-models-for-healthcare/

Cheers,

List alternatives and software for interaction effect measures

I am extending all chapters with a section for software implementations and to alternative algorithms (also with software implementation). The software can be any free and open source software: R, Python, Weka, ...

You can help out by posting links to alternatives and software implementations of interaction effect measure algorithms as comments to this issue.

Gitbook

Gitbook seems to now only have a legacy option. Did you use the editor to create this on your desktop ? Can you possibly recommend an easy way to get gitbook going. I love your work and have linked to it via one of my websites.

render_book error: Cannot open file '10-references.Rmd'

I'm progressing through your README file and upon executing line 34:

bookdown::render_book('', 'bookdown::gitbook')

I receive the following error:

Error` in file(con, "r") : cannot open the connection In addition: Warning message: In file(con, "r") : cannot open file '10-references.Rmd': No such file or directory

I checked the 'manuscript' folder for the following file: '10-references.Rmd' but did not see it in the folder.

Thank you for your time!

Cross-references

Hi!

Thanks for your effort with this manuscript.

I've noticed that in many places you use "in this section" or "as can be seen here" as cross-references. These work fine if the manuscript is read as an interactive document, i.e., freely on your web-site. However, if read in a printed PDF, these cross-references are not particularly useful.

I'll work on pull-request to fix these issues, but I thought you should be aware of them.

Cheers,
Isak

Add acknowledgements

• Chapter contributors (Abi, Verena)
• smaller fixes (go through PRs)
• Cover (Yvonne)
• Images (Shapley: Abi, flaticon; future: this japanese website)
• Funding from ZD.B
• Early readers

Unclear formulation: sensitive M

The explanation of the Monte-Carlo approximation of the shapely value says "It is unclear how to choose a sensitive M".

I'm not entirely sure what you mean by sensitive here.

Semi-related question: couldn't the correlation problem be solved by running a PCA first? (orthogonalize to the feature of interest, then run PCA on the remaining columns).

Create a pretty version of tree-to-rules graphic

The graphic that shows how a tree can be turned into decision rules is currently drawn by hand.
It would be great to have a pretty version of it, created on a computer.
The format can be SVG (preferred), png or jpg.
The filename is: rulefit.jpg

epub version

An ePub version could be easily supported by adding the option to the _output.yml, e.g.:

bookdown::epub_book:
  dev: svglite

An example can be seen here: https://github.com/rstudio/bookdown/blob/master/inst/examples/_output.yml
Rendering seems to work fine with standard settings.

Friedman’s H-statistic formula

In https://christophm.github.io/interpretable-ml-book/interaction.html#theory-friedmans-h-statistic

The two formula's under

In mathematical terms, the H-statistic for the interaction between feature [...] proposed by Friedman and Popescu is:

Should have a ^2 (i.e. be squared) added around the square brackets in de numerator as well, when comparing to the paper, I believe.

Create better pictograms for explaining the Shapley value

They are hand drawn and I am not super happy with them.
Help to improve those is appreciated.

The location in the book: https://github.com/christophM/interpretable-ml-book/blob/master/chapters/05.6-agnostic-shapley.Rmd

Fix centering of ice plots

should all be centered at x = 0.

Maybe the problem is with the iml package.

Labels on Figure 6.9

The labels are inconsistent with the description in the preceding paragraph.

Price prediction is 310.000 instead of 301.000 in Figure 5.20

The price is correct in figure text and also in the figure in the pdf version, but it is incorrect in online version.

Write a paragraph on how Shapely fairs in image datasets.

Write a paragraph on how Shapely fairs in images. See Page 196 of Image Analysis and Recognition: 12th International Conference, ICIAR 2015 for an example

List logistic regression alternatives and software

I am extending all chapters with a section for software implementations and to alternative algorithms (also with software implementation). The software can be any free and open source software: R, Python, Weka, ...

You can help out by posting links to papers and software implementations of logistic regresision algorithms as comments to this issue.

List decision rule algorithm alternatives and software

I am extending all chapters with a section for software implementations and to alternative algorithms (also with software implementation). The software can be any free and open source software: R, Python, Weka, ...

You can help out by posting links to papers and software implementations of decision rule algorithms as comments to this issue.

Need more clarity on 5.1.5.

I think this paragraph needs more clarity when explaining how the effects on each feature (for this instance) affect the prediction.

Most of the confusion come from expressions like "unusually little or much".

First and foremost, providing the used instance allows the reader to have a little bit more context.

More concepts that are not clear:

1. Is "Temperature (2 degrees)" the data from this instance, or meaning that two unit increases?
2. "contributes less towards the predicted value compared to the average" - what is the average? Average predictions? Mean value of distribution?
3. "“days_since_2011” unusually much, because this instance is from late 2011 (5 days)." - so the features contributes a lot because it is from late 2011 (only 5 days????). Shouldn't it be 300 days or so?

Thanks a lot for the book, it is very well written, but I do feel that this example needs a little bit more of step-by-step explanation.
e.g.: Providing the instance, showing how it maps to the plot (calculating effects), and cross-referencing to the weight table, in order to have a full grasp of what is going on.

P.S.: It would also be really nice to have the data with the extracted features in order to reproduce the work.

Prototypes/criticisms : add disadvantage

I fail to see the difference between the two concepts. As per your example we can see that in the top right figure with 0.316 MMD, the middle point is considered a prototype. But you selected another configuration and the central blob ends as a criticism. Shouldn't be the conclusion to add another prototype ? It appears to me that criticisms are very dependant on the prototypes selected. Isn't that dangerous for interpretability to use two opposite concepts when blobs of data can be one or the other depending on an arbitrary cut off value ?