https://github.com/ramon-oliveira/deepstats
Repository for "Known Unknowns: Uncertainty Quality in Bayesian Neural Networks" paper.
https://github.com/ramon-oliveira/deepstats
bayesian-neural-networks deep-learning
Last synced: 5 months ago
JSON representation
Repository for "Known Unknowns: Uncertainty Quality in Bayesian Neural Networks" paper.
- Host: GitHub
- URL: https://github.com/ramon-oliveira/deepstats
- Owner: ramon-oliveira
- Created: 2016-05-17T19:18:04.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2017-03-03T16:32:22.000Z (about 8 years ago)
- Last Synced: 2023-10-20T22:45:19.802Z (over 1 year ago)
- Topics: bayesian-neural-networks, deep-learning
- Language: Jupyter Notebook
- Homepage:
- Size: 55.6 MB
- Stars: 12
- Watchers: 3
- Forks: 5
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Known Unknowns: Uncertainty Quality in Bayesian Neural Networks
This repository holds the code for the paper "Known Unknowns: Uncertainty Quality in Bayesian Neural Networks" accepted to the Bayesian Deep Learning Workshop at NIPS 2016.
Arxiv link: https://arxiv.org/abs/1612.01251
We evaluate the uncertainty quality in neural networks using anomaly detection. We extract uncertainty measures (e.g. entropy) from the predictions of candidate models, use those measures as features for an anomaly detector, and gauge how well the detector differentiates known from unknown classes. We assign higher uncertainty quality to candidate models that lead to better detectors. We also propose a novel method for sampling a variational approximation of a Bayesian neural network, called One-Sample Bayesian Approximation (OSBA). We experiment on two datasets, MNIST and CIFAR10. We compare the following candidate neural network models: Maximum Likelihood, Bayesian Dropout, OSBA, and --- for MNIST --- the standard variational approximation. We show that Bayesian Dropout and OSBA provide better uncertainty information than Maximum Likelihood, and are essentially equivalent to the standard variational approximation, but much faster.
## Reproducing results (Python 3.5)
### Installing dependences
```bash
pip install -r requirements.txt
```### Running an experiment
```bash
python run_experiment.py --dataset=mnist --model=mlp-dropout
```Available dataset options:
* mnist
* cifar10
* svhn (experimental)Available model options:
* mlp
* mlp-dropout
* mlp-poor-bayesian
* mlp-bayesian
* convolutional
* convolutional-dropout
* convolutional-poor-bayesian### Plotting results
```bash
python plots_anova.py --dataset=mnist
```## ANOVA Results
### MNIST
![]()
![]()
![]()
### CIFAR10 (Updated Results)
![]()
![]()
![]()
![]()