kohpangwei/influence-release

Understanding Black-box Predictions via Influence Functions

This code replicates the experiments from the following paper:

Pang Wei Koh and Percy Liang

Understanding Black-box Predictions via Influence Functions

International Conference on Machine Learning (ICML), 2017.

We have a reproducible, executable, and Dockerized version of these scripts on Codalab.

The datasets for the experiments can also be found at the Codalab link.

Dependencies:

• Numpy/Scipy/Scikit-learn/Pandas
• Tensorflow (tested on v1.1.0)
• Keras (tested on v2.0.4)
• Spacy (tested on v1.8.2)
• h5py (tested on v2.7.0)
• Matplotlib/Seaborn (for visualizations)

A Dockerfile with these dependencies can be found here: https://hub.docker.com/r/pangwei/tf1.1/

---

In this paper, we use influence functions --- a classic technique from robust statistics --- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks.

If you have questions, please contact Pang Wei Koh ([email protected]).

influence-release's Issues

Maybe a bug: retrain without re-initialization?

Hi, when I read the code, I noticed the parameters are only initialized at the creation of a model. In other words, the retrain() actually continue training without re-initialization. Is that alright?

Computing influence on a CNN

Hello,

I want to find the most influential training images for a CNN I created. Is there a way to accomplish this without reproducing my layer structure in your code and then retraining?

I was thinking about just copying the weights in the calculation process of the influence, but could not work out how to do this.

I would be very grateful if someone could get me a hint. Thanks in advance!

Adapting this approach to regressions tasks not just classification

Will this out of the box work with regressions tasks instead of classification.
or will the Influence values not be as descriptive as in the case of classification

How much system memory is required in this source ?

Hi, @kohpangwei

How much system memory is required in this source ?

I have 64GB Memory Machine, But this code used over 63GB nearly at data loading.

Is this normal?

Thanksm in advance ~

How is the damping term determined?

Hi,

@kohpangwei, How did you determine the damping term, 1e-2, in the 'train_minst.py'?

Is this general or specific to the CNN in the code?

I cannot find an explanation for that in the paper,

Thanks!

Local issue running Hospital Readmission notebook

I have been working on replicating the hospital readmission notebook locally and I am getting output that differs substantially from the expected output. Below, I show side-by-side my output vs the expected output. I think the major issue is that the number of parameters that my model outputs is 122 when it's supposed to be 127. I can't localize where this change is occurring because the dimensions of my input data are as expected: 127 columns. Any guidance you can provide on why this might be occurring would be greatly appreciated!

Typo in paper

Hi, the formula just below the equation (3) has -frac{1}{n} as factor, but I think it should be positive. In chapter 2.1, it is negative because of removing.

Why the hessian vector product is calculated by mini-batch?

Hi, I found the hessian vector product in the file genericNeuralNet.py is implemented with mini-batch (

influence-release/influence/genericNeuralNet.py

Lines 529 to 537 in 578bc45

	for i in xrange(num_iter):
	feed_dict = self.fill_feed_dict_with_batch(self.data_sets.train, batch_size=batch_size)
	# Can optimize this
	feed_dict = self.update_feed_dict_with_v_placeholder(feed_dict, v)
	hessian_vector_val_temp = self.sess.run(self.hessian_vector, feed_dict=feed_dict)
	if hessian_vector_val is None:
	hessian_vector_val = [b / float(num_iter) for b in hessian_vector_val_temp]
	else:
	hessian_vector_val = [a + (b / float(num_iter)) for (a,b) in zip(hessian_vector_val, hessian_vector_val_temp)]

). Why don't we directly compute the accurate hessian vector?

Error in influence calculation for spam experiment

In my opinion there is an error in the get_loo_influences() function that is used for the spam experiment:

influence-release/influence/binaryLogisticRegressionWithLBFGS.py

Lines 134 to 161 in 578bc45

	def get_loo_influences(self):
	X_train = self.data_sets.train.x
	Y_train = self.data_sets.train.labels * 2 - 1
	theta = self.sess.run(self.params)[0]
	# Pre-calculate inverse covariance matrix
	n = X_train.shape[0]
	dim = X_train.shape[1]
	cov = np.zeros([dim, dim])
	probs = expit(np.dot(X_train, theta.T))
	weighted_X_train = np.reshape(probs * (1 - probs), (-1, 1)) * X_train
	cov = np.dot(X_train.T, weighted_X_train) / n
	cov += self.weight_decay * np.eye(dim)
	cov_lu_factor = slin.lu_factor(cov)
	assert(len(Y_train.shape) == 1)
	x_train_theta = np.reshape(X_train.dot(theta.T), [-1])
	sigma = expit(-Y_train * x_train_theta)
	d_theta = slin.lu_solve(cov_lu_factor, X_train.T).T
	quad_x = np.sum(X_train * d_theta, axis=1)
	return sigma * quad_x

This piece of code calculates the influence "by hand" according to the formula from section 2.3 since gradients and hessian can be computed as closed form expressions in the case of logistic regression. The code uses the LU-decomposition of H to compute the quadratic-form-like term from the formula (to avoid calculation of H^-1) and everything is fine until the last line. Looking in the formula in section 2.3 we find that there are two sigma terms (one for x_test and one for x). In the case of this experiment, x_test == x and so the last line should be
return sigma**2 * quad_x
to account for this. For the saturated regions of the sigmoid function the difference is not so big but clearly, in between it can be quite big.

Pytorch implementation

Hi,
this might not be a question for the repo owner but maybe someone else sees this - I hope it is ok I put this question here.
Is anyone aware of a Pytorch implementation of influence functions? I think I got the implementation of the hessian vector product right but there is also a lot of data handling involved (to replace the Tensorflow feed_dict stuff by more Pytorchy data types). If no one has done it - I am currently working on it and can also share it (but this might take some time).

Best regards
Verena

how to download the datasets?

OSError: [Errno 2] No such file or directory: 'data/koda'

Mismatch between IHVP computation in code and paper?

Hi, thanks for the great work! I've learned a lot from working to understand the paper.

I'm still having some trouble understanding how the current implementation mirrors the recursive computation of the IHVP as written in the paper. Specifically the following line (500 in influence/genericNeuralNet.py):

cur_estimate = [a + (1-damping) * b - c/scale for (a,b,c) in zip(v, cur_estimate, hessian_vector_val)]

Ignoring the damping and scaling term for now, this seems to implement something like:

H_inv v <- v + H_inv v - H_est v

Where v is the gradient, Hinv is the inverse Hessian estimate, and H_est is the estimate of the Hessian for a single data point (computed as an HVP with v). However, in the paper the algorithm says to compute:

H_inv v <- v + (I - H_est) H_inv v = v + H_inv v - H_est H_inv v

Which differs by a factor H_inv in the last term. Am I misunderstanding something?

why does inverse_hvp / scale after the iteration in get_inverse_hvp_lissa

From #4 , I can understand why to use scale in HVP since the real loss function is scaled. But I don't know why to use scale at the end on line 507-510.

Upweighting a training point vs Perturbing a training input

Hi, thanks again for this amazing work. I'm not sure this is the right place to ask this question, but I'm curious about the differences between these two approaches. Specifically, when should we use one approach vs. the other? Thanks in advance for the help!

Need help: what does `set_params_op` do?

When I read the code at line 190 of file logisticRegressionWithLBFGS.py as follows:

self.sess.run(self.set_params_op, feed_dict=params_feed_dict).

I am curious what exactly does it do? It does not return a value. If I commented it, the log message printed by self.print_model_eval() changed. Would someone please explain?

Can't understand what function "get_inverse_hvp_lissa" do

It seems a little different with the illustration in paper.
I don't understand why do we need parameter damping and scale.
Escecially, how to understand this line of code
cur_estimate = [a + (1-damping) * b - c/scale for (a,b,c) in zip(v, cur_estimate, hessian_vector_val)]

Perturbation to find influential features in training data

Hi,

I am trying to find the influential features of a training instance which affect the test prediction on the MNIST dataset.

This is described in Section 2.2 "Perturbing a training input" as Ipert,loss(Z, Ztest) Link.The authors further describe that this equation can be used to find the features of Z that are most responsible for the prediction on Ztest.

In my current, approach I use the sign of Ipert,loss(Z,Ztest) to determine the perturbation value. However, I am unsure of how the perturbation of the image should actually be done.
I cannot find an explanation for that in the paper.

Thanks in advance.

How can we implement this method on LSTM

How to Computing Influence Function

Hi,
I want to get the
of my model.
But I can't find the useful .py for me...

Do you have any questions for me?
I have a trained model, train set and test set.
And I just want to get a pic likes this: (even without Euclidean distance)

Thanks

Influence value for RNN models

I am implementing many NLP models using RNN cell. Do you have any example using RNN?

Crashes at keras.layers.Flatten for keras 2.0.6

tensorflow version: 1.3.0
run_rbf_comparison.py crashes at keras.layers.Flatten with keras 2.0.6.
Fixed by switching to keras 2.0.0

Suggestion: Please state the required version of keras (and other packages). Perhaps this information is available somewhere but I haven't found.

run_rbf_comparison.py in ()
172 train_dir='output',
173 log_dir='log',
--> 174 model_name=full_model_name)
175 train_inception_features_val = generate_inception_features(
176 full_model,

influence/inceptionModel.py in init(self, img_side, num_channels, weight_decay, **kwargs)
39 self.num_features = 2048 # Hardcoded for inception. For some reason Flatten() doesn't register num_features.
40
---> 41 super(BinaryInceptionModel, self).init(**kwargs)
42
43 self.load_inception_weights()

influence/genericNeuralNet.pyc in init(self, **kwargs)
121 self.logits = self.inference(self.input_placeholder, self.labels_placeholder)
122 else:
--> 123 self.logits = self.inference(self.input_placeholder)
124
125 self.total_loss, self.loss_no_reg, self.indiv_loss_no_reg = self.loss(

influence/inceptionModel.py in inference(self, input)
168 pooled_inception_features = AveragePooling2D((8, 8), strides=(8, 8), name='avg_pool')(raw_inception_features)
169 print (pooled_inception_features)
--> 170 self.inception_features = Flatten(name='flatten')(pooled_inception_features)
171
172

......./virtualenv_folder/tensorflow/local/lib/python2.7/site-packages/keras/engine/topology.pyc in call(self, inputs, **kwargs)
613 # Infering the output shape is only relevant for Theano.
614 if all([s is not None for s in _to_list(input_shape)]):
--> 615 output_shape = self.compute_output_shape(input_shape)
616 else:
617 if isinstance(input_shape, list):

......./virtualenv_folder/tensorflow/local/lib/python2.7/site-packages/keras/layers/core.pyc in compute_output_shape(self, input_shape)
475 raise ValueError('The shape of the input to "Flatten" '
476 'is not fully defined '
--> 477 '(got ' + str(input_shape[1:]) + '. '
478 'Make sure to pass a complete "input_shape" '
479 'or "batch_input_shape" argument to the first '

ValueError: The shape of the input to "Flatten" is not fully defined (got (None, None, 2048). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model.

Minor error in documentation

Thanks for the detailed documentation of your code!

I notice here that the correct hessian-vector product should be 1/2 (A + A.T) v