Disclaimer: Code adapted or copied from other authors is marked as such at the top of the corresponding .py file or function / class.
Neural networks (NNs) are used to a great extent in various fields and are becoming increasingly relevant to decisions in our everyday life.
However, NNs are typically not aware of the uncertainty that accompanies their predictions, inhibiting their use in safety-critical applications.
Bayesian NNs (BNNs) allow quantifying uncertainty by modeling weights as distributions over possible values.
It is usually infeasible to compute BNNs analytically.
Various adaptations that lower the computational cost of BNNs and non-Bayesian approaches have been proposed to model uncertainty in neural networks.
As of now, there is no clear winner among those methods.
In this thesis, we benchmark the popular models Monte Carlo dropout (MC dropout), deep ensemble (PNN-E), and probabilistic backpropagation (PBP) as well as the last-layer inference methods neural linear model (NLM) and marginally calibrated deep distributional regression (DNNC), an adaption to NLM that ensures marginal calibration.
We assess their -- marginal and probabilistic -- calibration and obtain robust predictive performance results by evaluating the models' ability to predict uncertainty for in-between and out-of-range inputs.
Further, we provide empirical evidence that DNNC may be used as a post-calibration method on a NN fit with standardized inputs and outputs. This enables a wider usability of the general approach.
You can clone the repo to your local or cloud machine inside the target folder using:
git clone https://github.com/pjoachims/nnuncert.git
Next, change to the main repo folder and install all necessary packages (tested on Google Colab):
cd nnuncert
pip install -r requirements.txt
You can find a version of the packages used on that machine here.
Now we can import the package in Python with:
import nnuncert
Implemented are non-Bayesian (PNN, PNN-E) and Bayesian neural network methods that allow for quantifying uncertainty.
The methods can be initialized and run models in a convenient way (see notebooks for details):
from nnuncert.models import make_model
# specialize model architecture / description of hidden layers as list of tuples of 3
# [(#hidden_units, activation_function, dropout_rate), ... ]
# e.g., [[50, "relu", 0]] -> 1 hidden layer with 50 neurons without dropout
# e.g., [[50, "tang", 0.5], [50, "relu", 0.5]] -> 2 hidden layers a 50 neurons without 50% dropout with tanh / ReLU act. func
architecture = [[50, "relu", 0]]
# init model with architecture and input shape (# features)
# other model types: "DNNC-R", "PBP", "NLM", NLM-E", "PNN", "PNN-E"
input_shape = (13, ) # argument will be passed to tf.keras.Input
model = make_model("MC dropout", input_shape, architecture)
# compile and fit
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.01),
metrics=["mae", "mse"])
# fit to x_train, y_train
model.fit(x_train, y_train, epochs=40, verbose=0)
# make prediction object, which allows to get mean prediction and predictive variance
# mean: pred_test.pred_mean, variance: pred_test.var_total
# get predictive density for single input evaluated at y0 (array) : pred_test.pdfi(i=1, y0)
pred_test = model.make_prediction(x_test, npreds=1000)
In the table below, you find links to the implementations, the research papers, and example notebooks where the methods are applied on toy data and Boston Housing (other UCI datasets available). Note that it may take some reloads for the .ipynb due to the Github backend. All notebooks are best run in Google Colab by clicking the link on top.
You can run all models in the same notebook for Toy 1D data, Toy 2D data, as well as random, gap, and tail splits of the UCI data. The calibration is assessed here.
Note that the DNNC code cannot be run as I had to remove the code adapted from MATLAB due to licensing.
Abbreviation | Method | Link to Implementation | Paper | Toy | Uci |
---|---|---|---|---|---|
DNNC | Marginally Calibrated Deep Distributional Regression | nnuncert/dnnc | arxiv-link | toy.ipynb | uci.ipynb |
MC dropout | Monte Carlo Dropout | nnuncert/mc_dropout | arvix-link | toy.ipynb | uci.ipynb |
NLM | Neural Linear Model | nnuncert/nlm | arvix-link | toy.ipynb | uci.ipynb |
NLM-E | Ensemble of NLMs | nnuncert/ensemble | see NLM | toy.ipynb | uci.ipynb |
PBP | Probabilistic Backpropagation | nnuncert/pbp | arvix-link | toy.ipynb | uci.ipynb |
PNN | Probabilistic NN (learns mean and aleatoric variance) | nnuncert/pnn | ieeexplore-link | toy.ipynb | uci.ipynb |
PNN-E | Deep Ensemble (ensemble of PNNs) | nnuncert/ensemble | arvix-link | toy.ipynb | uci.ipynb |
DNNC-R works well without transforming the targets by the empirical density and instead standardizing the targets to have zero mean and unit variance suffices. See this notebook for empirical evidence.
Note that 'MC dropout' in thesis corresponds to 'MC dropout 400', i.e., MC dropout -- after grid search for best dropout rate -- trained with epochs = 400. Other methods are trained for 40 epochs.
Toy 1D | Toy 2D |
---|---|
![]() |
![]() |
You can find predictive boxplots for all split types: random, gap, and tail.
All results are saved as .json in their respective folder, e.g., PBP results on the first random split can be found here.
To view individual model performance at iteration level, consult one of the following tables:
Random Splits | Gap Splits | Tail Splits | ||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
Marginal Calibration | Probabilistic Calibration |
---|---|
![]() |
![]() |