pjoachims/nnuncert

Repository of my master's thesis: "Uncertainty Quantification with Bayesian Neural Networks"

nnuncert: Uncertainty Quantification with BNNs

Disclaimer: Code adapted or copied from other authors is marked as such at the top of the corresponding .py file or function / class.

Abstract

Neural networks (NNs) are used to a great extent in various fields and are becoming increasingly relevant to decisions in our everyday life. However, NNs are typically not aware of the uncertainty that accompanies their predictions, inhibiting their use in safety-critical applications. Bayesian NNs (BNNs) allow quantifying uncertainty by modeling weights as distributions over possible values. It is usually infeasible to compute BNNs analytically. Various adaptations that lower the computational cost of BNNs and non-Bayesian approaches have been proposed to model uncertainty in neural networks. As of now, there is no clear winner among those methods. In this thesis, we benchmark the popular models Monte Carlo dropout (MC dropout), deep ensemble (PNN-E), and probabilistic backpropagation (PBP) as well as the last-layer inference methods neural linear model (NLM) and marginally calibrated deep distributional regression (DNNC), an adaption to NLM that ensures marginal calibration. We assess their -- marginal and probabilistic -- calibration and obtain robust predictive performance results by evaluating the models' ability to predict uncertainty for in-between and out-of-range inputs. Further, we provide empirical evidence that DNNC may be used as a post-calibration method on a NN fit with standardized inputs and outputs. This enables a wider usability of the general approach.

Clone repo and install requirements

You can clone the repo to your local or cloud machine inside the target folder using:

git clone https://github.com/pjoachims/nnuncert.git

Next, change to the main repo folder and install all necessary packages (tested on Google Colab):

cd nnuncert
pip install -r requirements.txt

You can find a version of the packages used on that machine here.

Now we can import the package in Python with:

import nnuncert

Implemented methods

Implemented are non-Bayesian (PNN, PNN-E) and Bayesian neural network methods that allow for quantifying uncertainty.

The methods can be initialized and run models in a convenient way (see notebooks for details):

from nnuncert.models import make_model

# specialize model architecture / description of hidden layers as list of tuples of 3
# [(#hidden_units, activation_function, dropout_rate), ... ]
# e.g., [[50, "relu", 0]] -> 1 hidden layer with 50 neurons without dropout
# e.g., [[50, "tang", 0.5], [50, "relu", 0.5]] -> 2 hidden layers a 50 neurons without 50% dropout with tanh / ReLU act. func
architecture = [[50, "relu", 0]]

# init model with architecture and input shape (# features)
# other model types: "DNNC-R", "PBP", "NLM", NLM-E", "PNN", "PNN-E"
input_shape = (13, ) # argument will be passed to tf.keras.Input
model = make_model("MC dropout", input_shape, architecture)

# compile and fit
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.01),
              metrics=["mae", "mse"])

# fit to x_train, y_train
model.fit(x_train, y_train, epochs=40, verbose=0)

# make prediction object, which allows to get mean prediction and predictive variance
# mean: pred_test.pred_mean, variance: pred_test.var_total
# get predictive density for single input evaluated at y0 (array) : pred_test.pdfi(i=1, y0)
pred_test = model.make_prediction(x_test, npreds=1000)

In the table below, you find links to the implementations, the research papers, and example notebooks where the methods are applied on toy data and Boston Housing (other UCI datasets available). Note that it may take some reloads for the .ipynb due to the Github backend. All notebooks are best run in Google Colab by clicking the link on top.

You can run all models in the same notebook for Toy 1D data, Toy 2D data, as well as random, gap, and tail splits of the UCI data. The calibration is assessed here.

Note that the DNNC code cannot be run as I had to remove the code adapted from MATLAB due to licensing.

Abbreviation	Method	Link to Implementation	Paper	Toy	Uci
DNNC	Marginally Calibrated Deep Distributional Regression	nnuncert/dnnc	arxiv-link	toy.ipynb	uci.ipynb
MC dropout	Monte Carlo Dropout	nnuncert/mc_dropout	arvix-link	toy.ipynb	uci.ipynb
NLM	Neural Linear Model	nnuncert/nlm	arvix-link	toy.ipynb	uci.ipynb
NLM-E	Ensemble of NLMs	nnuncert/ensemble	see NLM	toy.ipynb	uci.ipynb
PBP	Probabilistic Backpropagation	nnuncert/pbp	arvix-link	toy.ipynb	uci.ipynb
PNN	Probabilistic NN (learns mean and aleatoric variance)	nnuncert/pnn	ieeexplore-link	toy.ipynb	uci.ipynb
PNN-E	Deep Ensemble (ensemble of PNNs)	nnuncert/ensemble	arvix-link	toy.ipynb	uci.ipynb

Some results

DNNC-R works well without transforming the targets by the empirical density and instead standardizing the targets to have zero mean and unit variance suffices. See this notebook for empirical evidence.

Note that 'MC dropout' in thesis corresponds to 'MC dropout 400', i.e., MC dropout -- after grid search for best dropout rate -- trained with epochs = 400. Other methods are trained for 40 epochs.

Toy datasets (click on the plot to see high resolution)

---

Toy 1D	Toy 2D

UCI datasets: predictive performance

You can find predictive boxplots for all split types: random, gap, and tail.

All results are saved as .json in their respective folder, e.g., PBP results on the first random split can be found here.

To view individual model performance at iteration level, consult one of the following tables:

Random Splits	Gap Splits	Tail Splits
train.csv test.csv boston boston concrete concrete energy energy kin8nm kin8nm powerplant powerplant wine wine yacht yacht	train.csv test.csv boston boston concrete concrete energy energy kin8nm kin8nm powerplant powerplant wine wine yacht yacht	train.csv test.csv boston boston concrete concrete energy energy kin8nm kin8nm powerplant powerplant wine wine yacht yacht

UCI datasets: calibration (click on plot to see high resolution)

Marginal Calibration	Probabilistic Calibration