{"id":15027668,"url":"https://github.com/javierantoran/bayesian-neural-networks","last_synced_at":"2025-05-15T15:08:36.752Z","repository":{"id":38008331,"uuid":"174975949","full_name":"JavierAntoran/Bayesian-Neural-Networks","owner":"JavierAntoran","description":"Pytorch implementations of Bayes By Backprop, MC Dropout, SGLD, the Local Reparametrization Trick, KF-Laplace, SG-HMC and more","archived":false,"fork":false,"pushed_at":"2023-10-20T01:10:33.000Z","size":16355,"stargazers_count":1889,"open_issues_count":5,"forks_count":305,"subscribers_count":29,"default_branch":"master","last_synced_at":"2025-04-07T20:11:36.014Z","etag":null,"topics":["approximate-inference","bayes-by-backprop","bayesian-inference","bayesian-neural-networks","classification","deep-learning","hmc","kronecker-factored-approximation","langevin-dynamics","local-reparametrization-trick","mc-dropout","mcmc","out-of-distribution-detection","pytorch","regression","reproducible-research","sgld","uncertainty","uncertainty-neural-networks","variational-inference"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JavierAntoran.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-03-11T10:21:08.000Z","updated_at":"2025-04-01T11:15:55.000Z","dependencies_parsed_at":"2022-08-08T22:45:55.620Z","dependency_job_id":"e462ed20-6164-4fb3-aeea-52a873ce7218","html_url":"https://github.com/JavierAntoran/Bayesian-Neural-Networks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JavierAntoran%2FBayesian-Neural-Networks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JavierAntoran%2FBayesian-Neural-Networks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JavierAntoran%2FBayesian-Neural-Networks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JavierAntoran%2FBayesian-Neural-Networks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JavierAntoran","download_url":"https://codeload.github.com/JavierAntoran/Bayesian-Neural-Networks/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247721898,"owners_count":20985084,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["approximate-inference","bayes-by-backprop","bayesian-inference","bayesian-neural-networks","classification","deep-learning","hmc","kronecker-factored-approximation","langevin-dynamics","local-reparametrization-trick","mc-dropout","mcmc","out-of-distribution-detection","pytorch","regression","reproducible-research","sgld","uncertainty","uncertainty-neural-networks","variational-inference"],"created_at":"2024-09-24T20:06:52.143Z","updated_at":"2025-04-07T20:11:46.713Z","avatar_url":"https://github.com/JavierAntoran.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Bayesian Neural Networks\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/JavierAntoran/Bayesian-Neural-Networks/blob/master/LICENSE) [![Python 2.7+](https://img.shields.io/badge/python-2.7+-blue.svg)](https://www.python.org/downloads/release/python-2716/)\n[![Pytorch 1.0](https://img.shields.io/badge/pytorch-1.0.1-blue.svg)](https://pytorch.org/)\n\n\n\nPytorch implementations for the following approximate inference methods:\n\n* [Bayes by Backprop](#bayes-by-backprop-bbp)\n* [Bayes by Backprop + Local Reparametrisation Trick](#local-reparametrisation-trick)\n* [MC dropout](#mc-dropout)\n* [Stochastic Gradient Langevin Dynamics](#stochastic-gradient-langevin-dynamics-sgld)\n* [Preconditioned SGLD](#psgld)\n* [Kronecker-Factorised Laplace Approximation](#kronecker-factorised-laplace)\n* [Stochastic Gradient Hamiltonian Monte Carlo with Scale Adaption](#stochastic-gradient-hamiltonian-monte-carlo)\n\nWe also provide code for:\n* [Bootstrap MAP Ensemble](#bootstrap-map-ensemble)\n\n### Prerequisites\n* PyTorch\n* Numpy\n* Matplotlib\n\nThe project is written in python 2.7 and Pytorch 1.0.1. If CUDA is available, it will be\nused automatically. The models can also run on CPU as they are not excessively big.\n\n## Usage\n\n### Structure\n\n#### Regression experiments\n\n\nWe carried out homoscedastic and heteroscedastic regression\n  experiements on toy datasets, generated with [(Gaussian Process ground truth)](https://colab.research.google.com/drive/1t-OmK57w31ukbuftqk-1zAFzIgZhSMwG),\n    as well as on real data (six [UCI datasets](https://archive.ics.uci.edu/ml/datasets.php)). \n    \n    \n*Notebooks/classification/(ModelName)_(ExperimentType).ipynb*: Contains experiments using (ModelName)\n on (ExperimentType), i.e. homoscedastic/heteroscedastic. The heteroscedastic\n  notebooks contain both toy and UCI dataset experiments for a given (ModelName).\n\nWe also provide [Google Colab](https://colab.research.google.com/) notebooks. This means that\n  you can run on a GPU (for free!). No modifications required - all dependencies \n  and datasets are added from within the notebooks - except for selecting \n  Runtime -\u003e Change runtime type -\u003e Hardware accelerator -\u003e GPU.\n\n\n\u003c!--* [Regression Results](#homoscedastic-regression)--\u003e\n\n#### MNIST classification experiments\n\n*train_(ModelName)_(Dataset).py*: Trains (ModelName) on (Dataset). Training\nmetrics and model weights will be saved to the specified directories.\n\n*src/*: General utilities and model definitions.\n\n*Notebooks/classification*: An asortment of notebooks which allow for model training, evaluation and\nrunning of digit rotation uncertainty experiments. They also allow for weight\ndistribution plotting and weight pruning.  They allow for loading of pre-trained models\nfor experimentation.\n\n### Bayes by Backprop (BBP)\n(https://arxiv.org/abs/1505.05424)\n\nColab notebooks with regression models: [BBP homoscedastic](https://colab.research.google.com/drive/1K1I_UNRFwPt9l6RRkp8IYg1504PR9q4L) / [heteroscedastic](https://colab.research.google.com/drive/13oTnT6oKnB6NNBPVAczx8X-QEot2hfp9)\n\nTrain a model on MNIST:\n```bash\npython train_BayesByBackprop_MNIST.py [--model [MODEL]] [--prior_sig [PRIOR_SIG]] [--epochs [EPOCHS]] [--lr [LR]] [--n_samples [N_SAMPLES]] [--models_dir [MODELS_DIR]] [--results_dir [RESULTS_DIR]]\n```\nFor an explanation of the script's arguments:\n```bash\npython train_BayesByBackprop_MNIST.py -h\n```\n\nBest results are obtained with a Laplace prior.\n\n#### Local Reparametrisation Trick\n(https://arxiv.org/abs/1506.02557)\n\nBayes By Backprop inference where the mean and variance of activations\n are calculated in closed form. Activations are sampled instead of\n weights. This makes the variance of the Monte Carlo ELBO estimator scale\n as 1/M, where M is the minibatch size. Sampling weights scales (M-1)/M.\n The KL divergence between gaussians can also be computed in closed form,\n further reducing variance. Computation of each epoch is faster and so is convergence.\n\nTrain a model on MNIST:\n```bash\npython train_BayesByBackprop_MNIST.py --model Local_Reparam [--prior_sig [PRIOR_SIG]] [--epochs [EPOCHS]] [--lr [LR]] [--n_samples [N_SAMPLES]] [--models_dir [MODELS_DIR]] [--results_dir [RESULTS_DIR]]\n```\n\n### MC Dropout\n(https://arxiv.org/abs/1506.02142)\n\nA fixed dropout rate of 0.5 is set.\n\nColab notebooks with regression models: [MC Dropout homoscedastic](https://colab.research.google.com/drive/11knF0-7F7hK3Yhsj8VkG9fHbdB-LvtpQ) [heteroscedastic](https://colab.research.google.com/drive/11wYjIF9_mPTpJJ-M-TqLJ1X6sfeXtrOd)\n\nTrain a model on MNIST:\n```bash\npython train_MCDropout_MNIST.py [--weight_decay [WEIGHT_DECAY]] [--epochs [EPOCHS]] [--lr [LR]] [--models_dir [MODELS_DIR]] [--results_dir [RESULTS_DIR]]\n```\nFor an explanation of the script's arguments:\n```bash\npython train_MCDropout_MNIST.py -h\n```\n\n\n### Stochastic Gradient Langevin Dynamics (SGLD)\n(https://www.ics.uci.edu/~welling/publications/papers/stoclangevin_v6.pdf)\n\nIn order to converge to the true posterior over w, the learning rate\nshould be annealed according to the [Robbins-Monro](https://en.wikipedia.org/wiki/Stochastic_approximation)\n conditions. In practise, we use a fixed learning rate.\n \nColab notebooks with regression models: [SGLD homoscedastic](https://colab.research.google.com/drive/1vV5bsp7o6SyhAXErHwUC1FYxb-9Dc9SK) / [heteroscedastic](https://colab.research.google.com/drive/1Qk-gGWLwVrYc6hn7-EKIbkIYeZoWBx4f)\n\nTrain a model on MNIST:\n```bash\npython train_SGLD_MNIST.py [--use_preconditioning [USE_PRECONDITIONING]] [--prior_sig [PRIOR_SIG]] [--epochs [EPOCHS]] [--lr [LR]] [--models_dir [MODELS_DIR]] [--results_dir [RESULTS_DIR]]\n```\nFor an explanation of the script's arguments:\n```bash\npython train_SGLD_MNIST.py -h\n```\n\n#### pSGLD\n(https://arxiv.org/abs/1512.07666)\n\nSGLD with RMSprop preconditioning. A higher learning rate should be used\nthan for vanilla SGLD.\n\nTrain a model on MNIST:\n```bash\npython train_SGLD_MNIST.py --use_preconditioning True [--prior_sig [PRIOR_SIG]] [--epochs [EPOCHS]] [--lr [LR]] [--models_dir [MODELS_DIR]] [--results_dir [RESULTS_DIR]]\n```\n\n### Bootstrap MAP Ensemble\n\nMultiple networks are trained on subsamples of the dataset.\n\nColab notebooks with regression models: [MAP Ensemble homoscedastic](https://colab.research.google.com/drive/1iA3IUjaEHlb0XpLUF_WafbMS70UQnSaA) / [heteroscedastic](https://colab.research.google.com/drive/1B14--H3mduShIDb7M-CPbDOG8lOu2jvK)\n\nTrain an ensemble on MNIST:\n```bash\npython train_Bootrap_Ensemble_MNIST.py [--weight_decay [WEIGHT_DECAY]] [--subsample [SUBSAMPLE]] [--n_nets [N_NETS]] [--epochs [EPOCHS]] [--lr [LR]] [--models_dir [MODELS_DIR]] [--results_dir [RESULTS_DIR]]\n```\nFor an explanation of the script's arguments:\n```bash\npython train_Bootrap_Ensemble_MNIST.py -h\n```\n\n## Kronecker-Factorised Laplace\n(https://openreview.net/pdf?id=Skdvd2xAZ)\n\nTrain a MAP network and then calculate a second order taylor series aproxiamtion\n to the curvature around a mode of the posterior. A block diagonal Hessian\n approximation is used, where only intra-layer dependencies are accounted\n for. The Hessian is further approximated as the kronecker product of the \n expectation of a single datapoint's Hessian factors. Approximating the Hessian\n can take a while. Fortunately it only needs to be done once. \n\nTrain a MAP network on MNIST and approximate Hessian:\n```bash\npython train_KFLaplace_MNIST.py [--weight_decay [WEIGHT_DECAY]] [--hessian_diag_sig [HESSIAN_DIAG_SIG]] [--epochs [EPOCHS]] [--lr [LR]] [--models_dir [MODELS_DIR]] [--results_dir [RESULTS_DIR]]\n```\n\nFor an explanation of the script's arguments:\n```bash\npython train_KFLaplace_MNIST.py -h\n```\n\nNote that we save the unscaled and uninverted Hessian factors. This will\nallow for computationally cheap changes to the prior at inference time as the\nHessian will not need to be re-computed. Inference will require inverting\n the approximated Hessian factors and sampling from a matrix normal distribution.\n This is shown in [notebooks/KFAC_Laplace_MNIST.ipynb](https://github.com/JavierAntoran/Bayesian-Neural-Networks/blob/master/notebooks/classification/KFAC_Laplace_MNIST.ipynb)\n\n## Stochastic Gradient Hamiltonian Monte Carlo\n(https://arxiv.org/abs/1402.4102)\n\nWe implement the scale-adapted version of this algorithm, proposed [here](https://papers.nips.cc/paper/6117-bayesian-optimization-with-robust-bayesian-neural-networks.pdf)\nto find hyperparameters automatically during burn-in. We place a Gaussian prior\nover network weights and a Gamma hyperprior over the Gaussian's precision.\n\nRun SG-HMC-SA burn in and sampler, saving weights in specified file.\n```bash\npython train_SGHMC_MNIST.py [--epochs [EPOCHS]] [--sample_freq [SAMPLE_FREQ]] [--burn_in [BURN_IN]] [--lr [LR]] [--models_dir [MODELS_DIR]] [--results_dir [RESULTS_DIR]]\n```\n\nFor an explanation of the script's arguments:\n```bash\npython train_SGHMC_MNIST.py -h\n```\n \n\n## Approximate Inference in Neural Networks\n\nMap inference provides a point estimate of parameter values. When provided with\nout of distribution inputs, such as rotated digits, these models then to\nmake wrong predictions with high confidence.\n\n\u003cimg src=\"images/MAP.png\" width=\"430\" height=\"270\"/\u003e\n\n### Uncertainty Decomposition\nWe can measure uncertainty in our models' predictions through predictive entropy.\nWe can decompose this term in order to distinguish between 2 types of uncertainty.\nUncertainty caused by noise in the data, or **Aleatoric uncertainty**, can be\n quantified as the expected entropy of model predictions. Model uncertainty\n or **Epistemic uncertainty** can be measured as the difference between total entropy\n and aleatoric entropy.\n\n## Results\n\n### Homoscedastic Regression\n\nToy homoscedastic regression task. Data is generated by a GP with a RBF\n kernel (l = 1, σn = 0.3). We use a single-output FC network with one hidden layer of\n  200 ReLU units to predict the regression mean μ(x). A fixed log σ is learnt separately.\n\u003cp float=\"center\"\u003e\n  \u003cimg src=\"images/map_homo.png\" width=\"170\" /\u003e\n  \u003cimg src=\"images/bbp_homo.png\" width=\"150\" /\u003e \n  \u003cimg src=\"images/mc_dropout_homo.png\" width=\"150\" /\u003e\n  \u003cimg src=\"images/sgld_homo.png\" width=\"150\" /\u003e \n  \u003cimg src=\"images/gp_homo.png\" width=\"150\" /\u003e\n\u003c/p\u003e\n\n### Heteroscedastic Regression\nSame scenario as previous section but log σ(x) is predicted from the input.\n\u003cp float=\"center\"\u003e\n  \u003cimg src=\"images/map_hetero.png\" width=\"170\" /\u003e\n  \u003cimg src=\"images/bbp_hetero.png\" width=\"150\" /\u003e \n  \u003cimg src=\"images/mc_dropout_hetero.png\" width=\"150\" /\u003e\n  \u003cimg src=\"images/sgld_hetero.png\" width=\"150\" /\u003e \n  \u003cimg src=\"images/gp_hetero.png\" width=\"150\" /\u003e\n\u003c/p\u003e\n\nToy heteroscedastic regression task. Data is generated by a GP with a RBF\n kernel (l = 1 σn = 0.3 · |x + 2|). We use a two-head network with 200 ReLU units to predict the regression mean μ(x) and log-standard deviation log σ(x).\n \n### Regression on UCI datasets\n\nWe performed heteroscedastic regression on the six UCI datasets\n ([housing](https://archive.ics.uci.edu/ml/machine-learning-databases/housing/),\n  [concrete](http://archive.ics.uci.edu/ml/datasets/concrete+compressive+strength),\n   [energy efficiency](https://archive.ics.uci.edu/ml/datasets/energy+efficiency),\n    [power plant](https://archive.ics.uci.edu/ml/datasets/combined+cycle+power+plant),\n     [red wine](https://archive.ics.uci.edu/ml/datasets/wine+quality) and [yacht](http://archive.ics.uci.edu/ml/datasets/yacht+hydrodynamics) datasets),\n      using 10-foild cross validation. All these experiments are contained in the heteroscedastic notebooks. \n      Note that results depend heavily on hyperparameter selection. Plots below show log-likelihoods and RMSEs \non the train (semi-transparent colour) and test (solid colour). Circles and error bars correspond \nto the 10-fold cross validation mean and standard deviations respectively.\n\n\u003cp float=\"center\"\u003e\n  \u003cimg src=\"images/uci_regression_log_liks.png\" float=\"center\" width=\"400\" /\u003e\n  \u003cimg src=\"images/uci_regression_rmses.png\" float=\"center\" width=\"400\" /\u003e\n\u003c/p\u003e\n\n### MNIST Classification\n\nW is marginalised with 100 samples of the weights for all models except\nMAP, where only one set of weights is used.\n\n|      MNIST Test\t|   MAP   \t| MAP  Ensemble \t| BBP  Gaussian \t| BBP  GMM \t| BBP  Laplace \t| BBP Local Reparam \t| MC Dropout \t|   SGLD  \t|  pSGLD  \t|\n|:--------------:\t|:-------:\t|:-------------------:\t|:--------------:\t|:---------:\t|:-------------:\t|:-------------------:\t|:----------:\t|:-------:\t|:-------:\t|\n| Log Like\t| -572.9 \t|       -496.54       \t|    -1100.29    \t|  -1008.28 \t|    -892.85    \t|       -1086.43      \t|  -435.458  \t| -828.29 \t| -661.25 \t|\n|    Error \\%    \t|   1.58  \t|         1.53        \t|      2.60      \t|    2.38   \t|      2.28     \t|         2.61        \t|    1.37    \t|   1.76  \t|   1.76  \t|\n\nMNIST test results for methods under consideration. Estensive hyperparameter\ntunning has not been performed. We approximate\n the posterior predictive distribution with 100 MC samples. We use a FC\n  network with two 1200 unit ReLU layers. If unspecified, the prior is\n   Gaussian with std=0.1. P-SGLD uses RMSprop preconditioning.\n\n [The original paper](https://arxiv.org/abs/1505.05424) for Bayes By Backprop\n reports around 1% error on MNIST. We find that this result is attainable\n only if approximate posterior variances are initialised to be very small (BBP Gauss 2).\n In this scenario, the distributions over weights resemble deltas, giving \n good predictive performance but bad uncertainty estimates. \n However, when initialising the variances to match the prior (BBP Gauss 1), we obtain the above results.\n The training curves for both of these hyperparameter configuration schemes\n are shown below:\n\n\u003cimg src=\"images/BBP_train.png\" width=\"500\" height=\"420\"/\u003e\n\n\n### MNIST Uncertainty\n\nTotal, aleatoric and epistemic uncertainties obtained when\ncreating OOD samples by augmenting the MNIST test set with rotations:\n\n\u003cimg src=\"images/all_rotations.png\" width=\"900\" height=\"420\"/\u003e\n\nTotal and epistemic uncertainties obtained by testing our models, - which \nhave been trained on MNIST -, on the KMNIST dataset:\n\n\u003cimg src=\"images/KMNIST_entropies.png\" width=\"770\" height=\"240\"/\u003e\n\n### Adversarial robustness\n\n\u003cimg src=\"https://i.imgur.com/R61K4Ab.png\" width=\"600\" height=\"280\"/\u003e\n\nTotal, aleatoric and epistemic uncertainties obtained when\nfeeding our models with adversarial samples (fgsm).\n\n\u003cimg src=\"https://i.imgur.com/Mc484A3.png\" width=\"900\" height=\"420\"/\u003e\n\n### Weight Distributions\n\n\u003cimg src=\"images/Wdistribution.png\" width=\"450\" height=\"270\"/\u003e\n\nHistograms of weights sampled from each model trained on MNIST. We draw 10 samples of w for each model.\n\n### Weight Pruning\n\n\\#TODO\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjavierantoran%2Fbayesian-neural-networks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjavierantoran%2Fbayesian-neural-networks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjavierantoran%2Fbayesian-neural-networks/lists"}