kyle-dorman/bayesian-neural-network-blogpost

Building a Bayesian deep learning classifier

bayesian-neural-network-blogpost's Issues

about BNN in NLP

Hi, I have read your explanation about Bayesian neural network, aleatoric uncertainty and epistemic uncertainty. It is very excellent and straightforward. But I also notice an interesting phenomenon that when we talk about Bayesian deep learning, we seldom talk about those interesting tasks in NLP. Why is that? I think that making weights distribution and aleatoric / epistemic uncertainty are still useful in NLP tasks, but there are less applications then in CV area. And also in my experiments, Bayesian neural networks fail to capture those useful features and make wrong judgement about aleatoric / epistemic uncertainty. I really wonder why. If we can have a discussion about this, I will be be be very grateful. Thanks.

Index Error

Code is not working for results_dict['min loss'].append("{:.2E}".format(Decimal(Y[Z[min_var_idx,idx],idx])))

Error in bayesian_categorical_crossentropy_internal in bnn/loss_equations.py

I ran bin/create_batch_data.py successfully. But when I run bin/train.py an error TypeError: Failed to convert object of type <class 'theano.tensor.var.TensorVariable'> to Tensor. Contents: Elemwise{second,no_inplace}.0. Consider casting elements to a supported type. occurs in bnn/loss_equations.py in bayesian_categorical_crossentropy_internal function in
dist = distributions.Normal(loc=K.zeros_like(std), scale=std). I run project under Python2.7, TensorFlow version 1.3.0.

Here is an error:
Using Theano backend.
('encoder:', 'resnet50')
('batch_size:', 32)
('epochs:', 1)
('dataset:', 'cifar10')
('monte_carlo_simulations:', 100)
Unpickling file batch_data/resnet50_cifar10/train.p
Unpickling file batch_data/resnet50_cifar10/test.p
Compiling model.
Traceback (most recent call last):
  File "bin/train.py", line 88, in <module>
    tf.app.run()
  File "/home/oshevech/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "bin/train.py", line 66, in main
    loss_weights={'logits_variance': .2, 'softmax_output': 1.})
  File "/home/oshevech/anaconda2/lib/python2.7/site-packages/keras/engine/training.py", line 899, in compile
    sample_weight, mask)
  File "/home/oshevech/anaconda2/lib/python2.7/site-packages/keras/engine/training.py", line 430, in weighted
    score_array = fn(y_true, y_pred)
  File "/home/oshevech/Documents/Work/bayesian-neural-network-blogpost/bnn/loss_equations.py", line 72, in bayesian_categorical_crossentropy_internal
    dist = distributions.Normal(loc=K.zeros_like(std), scale=std)
  File "/home/oshevech/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/distributions/normal.py", line 136, in init
    self._loc = array_ops.identity(loc, name="loc")
  File "/home/oshevech/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1400, in identity
    result = _op_def_lib.apply_op("Identity", input=input, name=name)
  File "/home/oshevech/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 493, in apply_op
    raise err
TypeError: Failed to convert object of type <class 'theano.tensor.var.TensorVariable'> to Tensor. Contents: Elemwise{second,no_inplace}.0. Consider casting elements to a supported type.

about the variance of classification task

Hi, when modeling aleatoric uncertainty, why is pred_var, predicted logit values and variance's shape: (N, C + 1)? Shouldn't it be (N, C * 2)?

The categorical_cross_entropy in the Bayesian loss function is wrong.

The function definition is:
tf.keras.backend.categorical_crossentropy(
target,
output,
from_logits=False,
axis=-1
)

But the code on line 68 is:
undistorted_loss = K.categorical_crossentropy(pred, true, from_logits=True)

How to get things run?

Hi, I have been reading the document carefully, but do not quite get how to at least get things work?

Negative loss & logits_variance_loss

Hi, I have created a Bayesian CNN classifier as described in this repo, but my model's loss is always negative as well as the logits_variance_loss (see screenshot below). Any idea why is that happening?

Code is breaking at the following line results_dict['min loss'].append("{:.2E}".format(Y[Z[min_var_idx,idx],idx]))

analyze_aleatoric_uncertainty notebook
Z[min_var_idx,idx] is giving negative value.

should we sample when modeling aleatoric uncertainty

Hi, in What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision, it seems that when modeling aleatoric uncertainty, dropout is disabled, the model is just a normal neural network which predicts mean and variance of the input. Please see Section 2.2:

Note that here, unlike the above, variational inference is not performed over the weights, but instead we perform MAP inference – finding a single value for the model parameters θ. This approach does not capture epistemic model uncertainty, as epistemic uncertainty is a property of the model and not of the data.

Otherwise, modeling aleatoric uncertainty will be the same as modeling both two kinds of uncertainties.

Unknown entry in loss dictionary: "logits_variance".

model.compile(optimizer=optimizers.Adam(lr=1e-3, decay=0.001), loss={'logits_variance': bayesian_categorical_crossentropy(100, 10), 'softmax_output': 'categorical_crossentropy'}, metrics={'softmax_output': metrics.categorical_accuracy}, loss_weights={'logits_variance': .2, 'softmax_output': 1.})

returns the "logits_variance" issue in the title, or the following:

TypeError: Expected float32, got <tile.Value ZerosLike FLOAT32(<tile.Value Ceiling UINT64()>, -7)> of type 'Value' instead.
...after many file path dumps in the error.

I've tried using the code here, but I keep running into this. Scanning the code, I'm unclear of where this is defined. Any advice would be helpful. I've tried defining my own model, I've also tried generating a model as follows:

model=create_bayesian_model('resnet50', (400, 400, 3), 2)

Unable to download the models and dataset files

I tried running the ./bayesian-neural-network-blogpost/bin/download_model_info.py script to download the model along with the cifar10 data. But i was unable to access to access the s3 bucket . Could someone help me to get the model?

Regards,
Jayanth BR

Evaluating variance in output

The model trains with both logits_variance and softmax as the output.

softmax is trained with the classification value

What is logits variance trained with?