{"id":19412792,"url":"https://github.com/techshot25/bayesian-neural-network","last_synced_at":"2025-02-25T02:44:08.015Z","repository":{"id":213791555,"uuid":"193388767","full_name":"techshot25/Bayesian-Neural-Network","owner":"techshot25","description":"You know I'm all about the Bayes!","archived":false,"fork":false,"pushed_at":"2019-06-25T01:33:08.000Z","size":119,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-07T15:41:29.804Z","etag":null,"topics":["bayesian-inference","bayesian-networks","probabilistic-neural-network","pyro-ppl"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/techshot25.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-06-23T19:43:43.000Z","updated_at":"2019-06-25T01:33:10.000Z","dependencies_parsed_at":"2023-12-23T05:35:15.154Z","dependency_job_id":null,"html_url":"https://github.com/techshot25/Bayesian-Neural-Network","commit_stats":null,"previous_names":["techshot25/bayesian-neural-network"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techshot25%2FBayesian-Neural-Network","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techshot25%2FBayesian-Neural-Network/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techshot25%2FBayesian-Neural-Network/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techshot25%2FBayesian-Neural-Network/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/techshot25","download_url":"https://codeload.github.com/techshot25/Bayesian-Neural-Network/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240593169,"owners_count":19825930,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bayesian-inference","bayesian-networks","probabilistic-neural-network","pyro-ppl"],"created_at":"2024-11-10T12:28:15.276Z","updated_at":"2025-02-25T02:44:07.991Z","avatar_url":"https://github.com/techshot25.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n```python\n#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nCreated on Thu Jun 20 15:47:23 2019\n\nCredit:\n    https://github.com/paraschopra/bayesian-neural-network-mnist\n\n@author: Ali Shannon\n\"\"\"\n\n#%% Libraries to use\n\nimport torch\nfrom torch import nn\nfrom torch.utils.data import DataLoader\nimport torch.nn.functional as F\nimport numpy as np\nfrom torchvision import datasets, transforms\nfrom matplotlib import pyplot as plt\nfrom matplotlib import colors\n\nimport pyro # pip install pyro-ppl\nfrom pyro import optim\nfrom pyro.infer import SVI, Trace_ELBO\nimport pyro.distributions as dist\n\ndevice = 'cuda' if torch.cuda.is_available() else 'cpu'\n```\n\nThis is our pytorch network that we will \"lift\" into a pyro module\n\n\n```python\nclass NN(nn.Module):\n    def __init__(self, in_shape, hidden_shape, out_shape):\n        super(NN, self).__init__()\n        \n        self.fc1 = nn.Linear(in_shape, hidden_shape)\n        self.out = nn.Linear(hidden_shape, out_shape)\n        \n    def forward(self, x):\n        x = x.flatten(1)\n        x = F.relu(self.fc1(x))\n        return self.out(x)\n        \n    \ntrain_loader = DataLoader(\n        datasets.MNIST('emnist', train=True, download=True,\n                       transform=transforms.ToTensor()),\n                batch_size=128, shuffle=True, num_workers=4)\n        \ntest_loader = DataLoader( \n        datasets.MNIST('emnist', train=False,\n                       transform=transforms.ToTensor()),\n                batch_size=128, shuffle=True, num_workers=4)\n        \n# this will be our neural network\nnet = NN(28*28, 1024, 10).to(device)\n```\n\nHere we will defined the converter to random variables `pyro.random_module` where the output of our network are random variables\n\n\n```python\ndef model(x, y):\n    priors = {}\n    for name, param in net.named_parameters():\n        priors[name] = dist.Normal(loc=torch.zeros_like(param.data), \n                                        scale=torch.ones_like(param.data))\n        \n    lifted_module = pyro.random_module(\"module\", net, priors)\n    lifted_clf_model = lifted_module()\n    lhat = F.log_softmax(lifted_clf_model(x), 1)\n    pyro.sample(\"obs\", dist.Categorical(logits=lhat), obs=y)\n```\n\nThis is how the Bayesian network works, it uses this form of Bayes Theorem\n\n$$P(A|B) = \\frac{P(B|A)\\; P(A)}{P(B)}$$\n\nWhere:\n\n$$P(B) = \\sum_{i} P(B|A_{i})\\; P(A_{i})$$\n\nBut this sum is very hard to compute because there will be too many parameters. So instead we will draw random samples and report the posterior mean as our point estimate and posterior uncertainty as our error. This is at the heart of Stochastic Variational Inference (SVI): https://www.youtube.com/watch?v=DYRK0-_K2UU\n\nRoughly, this method assumes that the prior is always some normal distribution and updates the point estimates through the network.\n\n\n\n\n\n```python\ndef guide(x, y):\n    priors = {}\n    for name, param in net.named_parameters():\n        mu = torch.randn_like(param)\n        sigma = torch.randn_like(param)\n        mu_param = pyro.param(name+'.mu', mu)\n        sigma_param = F.softplus(pyro.param(name+'.sigma', sigma))\n        prior = dist.Normal(loc=mu_param, scale=sigma_param)\n        priors[name] = prior\n        \n    lifted_module = pyro.random_module('module', net, priors)\n    return lifted_module()\n```\n\nSet up optimizer and loss function\n\n\n```python\nopt = optim.Adam({'lr': 0.01})\nsvi = SVI(model, guide, opt, loss=Trace_ELBO())\n```\n\nTrain Model\n\n\n```python\ndef train():\n    pyro.clear_param_store()\n    epochs = 10\n    loss = 0\n    for epoch in range(1, epochs + 1):\n        loss = 0\n        for x, y in train_loader:\n            loss += svi.step(x.flatten(1).to(device), y.to(device))\n        total_loss = loss / len(train_loader.dataset)\n        print(f'epoch: {epoch}\\tLoss: {total_loss:.4g}')\n        \ntrain()\n    \n```\n\n    epoch: 1\tLoss: 2073\n    epoch: 2\tLoss: 363.2\n    epoch: 3\tLoss: 155.8\n    epoch: 4\tLoss: 110.5\n    epoch: 5\tLoss: 95.18\n    epoch: 6\tLoss: 89.92\n    epoch: 7\tLoss: 87.03\n    epoch: 8\tLoss: 86.42\n    epoch: 9\tLoss: 85.71\n    epoch: 10\tLoss: 85.49\n\n\nThere are two ways to evaluate this network.\n\n- The first is to force it to predict even if it's unsure, which is what traditional neural networks do\n- The second is to make the network say 'I don't know.' Which is what we want. This works by drawing samples and checking if they agree on the result.\n\nLet's try the first one:\n\n\n\n```python\nnum_samples = 10\ndef predict(x):\n    sampled_models = [guide(None, None) for _ in range(num_samples)]\n    yhats = [model(x.to(device)).data for model in sampled_models]\n    mean = torch.mean(torch.stack(yhats), 0)\n    return np.argmax(mean.cpu().numpy(), axis=1)\n\nprint('Prediction when network is forced to predict')\ncorrect = 0\ntotal = 0\nfor x, y in test_loader:\n    predicted = predict(x.flatten(1))\n    total += y.size(0)\n    correct += (predicted == y.numpy()).sum().item()\nprint(f'Accuracy: {correct/total:.2%}')\n```\n\n    Prediction when network is forced to predict\n    Accuracy: 88.69%\n\n\nNow we let the network decide whether or not to predict \n\n\n```python\nclasses = [str(i) for i in range(10)]\n\nnum_samples = 100\ndef give_uncertainities(x):\n    sampled_models = [guide(None, None) for _ in range(num_samples)]\n    yhats = [F.log_softmax(model(x.flatten(1).to(device)).data, 1).cpu().detach().numpy() for model in sampled_models]\n    return np.asarray(yhats)\n    #mean = torch.mean(torch.stack(yhats), 0)\n    #return np.argmax(mean, axis=1)\n\n\ndef test_batch(images, labels, plot=True):\n    y = give_uncertainities(images)\n    predicted_for_images = 0\n    correct_predictions=0\n\n    for i in range(len(labels)):\n        if(plot):\n            print(\"Real: \",labels[i].item())\n            fig, axs = plt.subplots(1, 10, sharey=True,figsize=(20,2))\n    \n        all_digits_prob = []\n        highted_something = False\n        for j in range(len(classes)):\n        \n            highlight=False\n            histo = []\n            histo_exp = []\n            for z in range(y.shape[0]):\n                histo.append(y[z][i][j])\n                histo_exp.append(np.exp(y[z][i][j]))\n            \n            prob = np.percentile(histo_exp, 50) #sampling median probability\n        \n            if(prob\u003e0.2): #select if network thinks this sample is 20% chance of this being a label\n                highlight = True #possibly an answer\n        \n            all_digits_prob.append(prob)\n            \n            if(plot):\n            \n                N, bins, patches = axs[j].hist(histo, bins=8, color = \"lightgray\", lw=0,  weights=np.ones(len(histo)) / len(histo), density=False)\n                axs[j].set_title(str(j)+\" (\"+str(round(prob,2))+\")\") \n        \n            if(highlight):\n            \n                highted_something = True\n                \n                if(plot):\n\n                    # We'll color code by height, but you could use any scalar\n                    fracs = N / N.max()\n\n                    # we need to normalize the data to 0..1 for the full range of the colormap\n                    norm = colors.Normalize(fracs.min(), fracs.max())\n\n                    # Now, we'll loop through our objects and set the color of each accordingly\n                    for thisfrac, thispatch in zip(fracs, patches):\n                        color = plt.cm.viridis(norm(thisfrac))\n                        thispatch.set_facecolor(color)\n\n    \n        if(plot):\n            plt.show()\n    \n        predicted = np.argmax(all_digits_prob)\n    \n        if(highted_something):\n            predicted_for_images+=1\n            if(labels[i].item()==predicted):\n                if(plot):\n                    print(\"Correct\\n\")\n                correct_predictions +=1.0\n            else:\n                if(plot):\n                    print(\"Incorrect :()\\n\")\n        else:\n            if(plot):\n                print(\"Undecided.\\n\")\n        \n        if(plot):\n            plt.imshow(images[i].squeeze())\n        \n    \n    if(plot):\n        print(\"Summary\")\n        print(\"Total images: \",len(labels))\n        print(\"Predicted for: \",predicted_for_images)\n        print(\"Accuracy when predicted: \",correct_predictions/predicted_for_images)\n        \n    return len(labels), correct_predictions, predicted_for_images\n\n\n\n#%% Prediction when network can decide not to predict\n\nprint('Prediction when network can refuse')\ncorrect = 0\ntotal = 0\ntotal_predicted_for = 0\nfor j, data in enumerate(test_loader):\n    images, labels = data\n    \n    total_minibatch, correct_minibatch, predictions_minibatch = test_batch(images, labels, plot=False)\n    total += total_minibatch\n    correct += correct_minibatch\n    total_predicted_for += predictions_minibatch\n\nprint(\"Total images: \", total)\nprint(\"Skipped: \", total-total_predicted_for)\nprint(f\"Accuracy when made predictions: {correct/total_predicted_for:.2%}\")\n```\n\n    Prediction when network can refuse\n    Total images:  10000\n    Skipped:  1101\n    Accuracy when made predictions: 95.03%\n\n\nPrint out samples of distributions for each digit\n\n\n```python\ntest_batch(images[:10], labels[:10])\n```\n\n    Real:  6\n\n\n\n![png](output_16_1.png)\n\n\n    Correct\n    \n    Real:  6\n\n\n\n![png](output_16_3.png)\n\n\n\n![png](output_16_4.png)\n\n\n    Correct\n    \n    Real:  0\n\n\n\n![png](output_16_6.png)\n\n\n\n![png](output_16_7.png)\n\n\n    Correct\n    \n    Real:  9\n\n\n\n![png](output_16_9.png)\n\n\n\n![png](output_16_10.png)\n\n\n    Incorrect :()\n    \n    Real:  8\n\n\n\n![png](output_16_12.png)\n\n\n\n![png](output_16_13.png)\n\n\n    Undecided.\n    \n    Real:  4\n\n\n\n![png](output_16_15.png)\n\n\n\n![png](output_16_16.png)\n\n\n    Correct\n    \n    Real:  3\n\n\n\n![png](output_16_18.png)\n\n\n\n![png](output_16_19.png)\n\n\n    Incorrect :()\n    \n    Real:  3\n\n\n\n![png](output_16_21.png)\n\n\n\n![png](output_16_22.png)\n\n\n    Correct\n    \n    Real:  3\n\n\n\n![png](output_16_24.png)\n\n\n\n![png](output_16_25.png)\n\n\n    Incorrect :()\n    \n    Real:  8\n\n\n\n![png](output_16_27.png)\n\n\n\n![png](output_16_28.png)\n\n\n    Correct\n    \n    Summary\n    Total images:  10\n    Predicted for:  9\n    Accuracy when predicted:  0.6666666666666666\n\n\n\n\n\n    (10, 6.0, 9)\n\n\n\n\n![png](output_16_31.png)\n\n\n\n```python\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechshot25%2Fbayesian-neural-network","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftechshot25%2Fbayesian-neural-network","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechshot25%2Fbayesian-neural-network/lists"}