{"id":15027783,"url":"https://github.com/kumar-shridhar/pytorch-bayesiancnn","last_synced_at":"2025-05-15T16:09:07.279Z","repository":{"id":38184452,"uuid":"142577747","full_name":"kumar-shridhar/PyTorch-BayesianCNN","owner":"kumar-shridhar","description":"Bayesian Convolutional Neural Network with Variational Inference based on Bayes by Backprop in PyTorch. ","archived":false,"fork":false,"pushed_at":"2024-04-19T14:39:29.000Z","size":69465,"stargazers_count":1493,"open_issues_count":19,"forks_count":328,"subscribers_count":29,"default_branch":"master","last_synced_at":"2025-04-07T21:14:49.636Z","etag":null,"topics":["aleatoric-uncertainties","bayes","bayes-by-backprop","bayesian-convnets","bayesian-deep-learning","bayesian-inference","bayesian-network","bayesian-networks","bayesian-neural-networks","bayesian-statistics","convolutional-neural-networks","image-recognition","python","pytorch","pytorch-cnn","variational-bayes","variational-inference"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kumar-shridhar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-27T13:00:45.000Z","updated_at":"2025-04-06T14:29:49.000Z","dependencies_parsed_at":"2022-08-08T23:16:57.394Z","dependency_job_id":"5aff55c8-0e22-4701-ae07-3d3ed3ae9734","html_url":"https://github.com/kumar-shridhar/PyTorch-BayesianCNN","commit_stats":{"total_commits":286,"total_committers":8,"mean_commits":35.75,"dds":0.2797202797202797,"last_synced_commit":"d93bad543c3226cd0fe05c0cb0ba033c41b3caa6"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kumar-shridhar%2FPyTorch-BayesianCNN","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kumar-shridhar%2FPyTorch-BayesianCNN/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kumar-shridhar%2FPyTorch-BayesianCNN/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kumar-shridhar%2FPyTorch-BayesianCNN/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kumar-shridhar","download_url":"https://codeload.github.com/kumar-shridhar/PyTorch-BayesianCNN/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247730069,"owners_count":20986404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aleatoric-uncertainties","bayes","bayes-by-backprop","bayesian-convnets","bayesian-deep-learning","bayesian-inference","bayesian-network","bayesian-networks","bayesian-neural-networks","bayesian-statistics","convolutional-neural-networks","image-recognition","python","pytorch","pytorch-cnn","variational-bayes","variational-inference"],"created_at":"2024-09-24T20:07:02.438Z","updated_at":"2025-04-07T21:14:59.863Z","avatar_url":"https://github.com/kumar-shridhar.png","language":"Python","readme":"\n[![Python 3.7+](https://img.shields.io/badge/python-3.7+-blue.svg)](https://www.python.org/downloads/release/python-376/)\n[![Pytorch 1.3](https://img.shields.io/badge/pytorch-1.3.1-blue.svg)](https://pytorch.org/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/kumar-shridhar/PyTorch-BayesianCNN/blob/master/LICENSE)\n[![arxiv](https://img.shields.io/badge/stat.ML-arXiv%3A2002.02797-B31B1B.svg)](https://arxiv.org/abs/1901.02731)\n\nWe introduce **Bayesian convolutional neural networks with variational inference**, a variant of convolutional neural networks (CNNs), in which the intractable posterior probability distributions over weights are inferred by **Bayes by Backprop**. We demonstrate how our proposed variational inference method achieves performances equivalent to frequentist inference in identical architectures on several datasets (MNIST, CIFAR10, CIFAR100) as described in the [paper](https://arxiv.org/abs/1901.02731).\n\n---------------------------------------------------------------------------------------------------------\n\n\n### Filter weight distributions in a Bayesian Vs Frequentist approach\n\n![Distribution over weights in a CNN's filter.](experiments/figures/BayesCNNwithdist.png)\n\n---------------------------------------------------------------------------------------------------------\n\n### Fully Bayesian perspective of an entire CNN\n\n![Distributions must be over weights in convolutional layers and weights in fully-connected layers.](experiments/figures/CNNwithdist_git.png)\n\n---------------------------------------------------------------------------------------------------------\n\n\n\n### Layer types\n\nThis repository contains two types of bayesian lauer implementation:  \n* **BBB (Bayes by Backprop):**  \n  Based on [this paper](https://arxiv.org/abs/1505.05424). This layer samples all the weights individually and then combines them with the inputs to compute a sample from the activations.\n\n* **BBB_LRT (Bayes by Backprop w/ Local Reparametrization Trick):**  \n  This layer combines Bayes by Backprop with local reparametrization trick from [this paper](https://arxiv.org/abs/1506.02557). This trick makes it possible to directly sample from the distribution over activations.\n---------------------------------------------------------------------------------------------------------\n\n\n\n### Make your custom Bayesian Network?\nTo make a custom Bayesian Network, inherit `layers.misc.ModuleWrapper` instead of `torch.nn.Module` and use `BBBLinear` and `BBBConv2d` from any of the given layers (`BBB` or `BBB_LRT`) instead of `torch.nn.Linear` and `torch.nn.Conv2d`. Moreover, no need to define `forward` method. It'll automatically be taken care of by `ModuleWrapper`. \n\nFor example:  \n```python\nclass Net(nn.Module):\n\n  def __init__(self):\n    super().__init__()\n    self.conv = nn.Conv2d(3, 16, 5, strides=2)\n    self.bn = nn.BatchNorm2d(16)\n    self.relu = nn.ReLU()\n    self.fc = nn.Linear(800, 10)\n\n  def forward(self, x):\n    x = self.conv(x)\n    x = self.bn(x)\n    x = self.relu(x)\n    x = x.view(-1, 800)\n    x = self.fc(x)\n    return x\n```\nAbove Network can be converted to Bayesian as follows:\n```python\nclass Net(ModuleWrapper):\n\n  def __init__(self):\n    super().__init__()\n    self.conv = BBBConv2d(3, 16, 5, strides=2)\n    self.bn = nn.BatchNorm2d(16)\n    self.relu = nn.ReLU()\n    self.flatten = FlattenLayer(800)\n    self.fc = BBBLinear(800, 10)\n```\n\n#### Notes:\n1. Add `FlattenLayer` before first `BBBLinear` block.  \n2. `forward` method of the model will return a tuple as `(logits, kl)`.\n3. `priors` can be passed as an argument to the layers. Default value is:  \n```python\npriors={\n    'prior_mu': 0,\n    'prior_sigma': 0.1,\n    'posterior_mu_initial': (0, 0.1),  # (mean, std) normal_\n    'posterior_rho_initial': (-3, 0.1),  # (mean, std) normal_\n}\n```\n\n---------------------------------------------------------------------------------------------------------\n\n### How to perform standard experiments?\nCurrently, following datasets and models are supported.  \n* Datasets: MNIST, CIFAR10, CIFAR100  \n* Models: AlexNet, LeNet, 3Conv3FC  \n\n#### Bayesian\n\n`python main_bayesian.py`\n* set hyperparameters in `config_bayesian.py`\n\n\n#### Frequentist\n\n`python main_frequentist.py`\n* set hyperparameters in `config_frequentist.py`\n\n---------------------------------------------------------------------------------------------------------\n\n\n\n### Directory Structure:\n`layers/`:  Contains `ModuleWrapper`, `FlattenLayer`, `BBBLinear` and `BBBConv2d`.  \n`models/BayesianModels/`: Contains standard Bayesian models (BBBLeNet, BBBAlexNet, BBB3Conv3FC).  \n`models/NonBayesianModels/`: Contains standard Non-Bayesian models (LeNet, AlexNet).  \n`checkpoints/`: Checkpoint directory: Models will be saved here.  \n`tests/`: Basic unittest cases for layers and models.  \n`main_bayesian.py`: Train and Evaluate Bayesian models.  \n`config_bayesian.py`: Hyperparameters for `main_bayesian` file.  \n`main_frequentist.py`: Train and Evaluate non-Bayesian (Frequentist) models.  \n`config_frequentist.py`: Hyperparameters for `main_frequentist` file.  \n\n---------------------------------------------------------------------------------------------------------\n\n\n\n### Uncertainty Estimation:  \nThere are two types of uncertainties: **Aleatoric** and **Epistemic**.  \nAleatoric uncertainty is a measure for the variation of data and Epistemic uncertainty is caused by the model.  \nHere, two methods are provided in `uncertainty_estimation.py`, those are `'softmax'` \u0026 `'normalized'` and are respectively based on equation 4 from [this paper](https://openreview.net/pdf?id=Sk_P2Q9sG) and equation 15 from [this paper](https://arxiv.org/pdf/1806.05978.pdf).  \nAlso, `uncertainty_estimation.py` can be used to compare uncertainties by a Bayesian Neural Network on `MNIST` and `notMNIST` dataset. You can provide arguments like:     \n1. `net_type`: `lenet`, `alexnet` or `3conv3fc`. Default is `lenet`.   \n2. `weights_path`: Weights for the given `net_type`. Default is `'checkpoints/MNIST/bayesian/model_lenet.pt'`.  \n3. `not_mnist_dir`: Directory of `notMNIST` dataset. Default is `'data\\'`. \n4. `num_batches`: Number of batches for which uncertainties need to be calculated.  \n\n**Notes**:  \n1. You need to download the [notMNIST](http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html) dataset from [here](http://yaroslavvb.com/upload/notMNIST/notMNIST_small.tar.gz).  \n2. Parameters `layer_type` and `activation_type` used in `uncertainty_etimation.py` needs to be set from `config_bayesian.py` in order to match with provided weights. \n\n---------------------------------------------------------------------------------------------------------\n\n\n\nIf you are using this work, please cite:\n\n```\n@article{shridhar2019comprehensive,\n  title={A comprehensive guide to bayesian convolutional neural network with variational inference},\n  author={Shridhar, Kumar and Laumann, Felix and Liwicki, Marcus},\n  journal={arXiv preprint arXiv:1901.02731},\n  year={2019}\n}\n```\n\n```\n@article{shridhar2018uncertainty,\n  title={Uncertainty estimations by softplus normalization in bayesian convolutional neural networks with variational inference},\n  author={Shridhar, Kumar and Laumann, Felix and Liwicki, Marcus},\n  journal={arXiv preprint arXiv:1806.05978},\n  year={2018}\n}\n}\n```\n\n--------------------------------------------------------------------------------------------------------\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkumar-shridhar%2Fpytorch-bayesiancnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkumar-shridhar%2Fpytorch-bayesiancnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkumar-shridhar%2Fpytorch-bayesiancnn/lists"}