{"id":21152160,"url":"https://github.com/bayesgroup/variational-dropout-sparsifies-dnn","last_synced_at":"2026-03-16T10:33:49.131Z","repository":{"id":89468700,"uuid":"84544677","full_name":"bayesgroup/variational-dropout-sparsifies-dnn","owner":"bayesgroup","description":"Sparse Variational Dropout, ICML 2017","archived":false,"fork":false,"pushed_at":"2020-05-20T15:16:59.000Z","size":3929,"stargazers_count":312,"open_issues_count":3,"forks_count":45,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-02T11:06:06.741Z","etag":null,"topics":["bayesian-deep-learning","compression","icml"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bayesgroup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-10T09:46:36.000Z","updated_at":"2025-03-31T09:25:17.000Z","dependencies_parsed_at":"2023-03-08T21:30:47.854Z","dependency_job_id":null,"html_url":"https://github.com/bayesgroup/variational-dropout-sparsifies-dnn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bayesgroup%2Fvariational-dropout-sparsifies-dnn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bayesgroup%2Fvariational-dropout-sparsifies-dnn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bayesgroup%2Fvariational-dropout-sparsifies-dnn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bayesgroup%2Fvariational-dropout-sparsifies-dnn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bayesgroup","download_url":"https://codeload.github.com/bayesgroup/variational-dropout-sparsifies-dnn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248045231,"owners_count":21038553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bayesian-deep-learning","compression","icml"],"created_at":"2024-11-20T10:35:42.548Z","updated_at":"2026-03-16T10:33:44.073Z","avatar_url":"https://github.com/bayesgroup.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Variational Dropout Sparsifies Deep Neural Networks\n\n### Tensorflow implementation\nGoogle AI Research has released [State of Sparsity in Deep Neural Networks](https://arxiv.org/abs/1902.09574) - a nice large scale study of sparsification methods. The code contains an implementation of [Sparse variational dropout on Tensorflow](https://github.com/google-research/google-research/blob/master/state_of_sparsity/layers/variational_dropout/nn.py#L585).\n\n### Play around w/ SparseVD (PyTorch)\nYou can play with compression of a small neural network using the following [IPython notebook @ Colab](https://colab.research.google.com/github/bayesgroup/deepbayes-2019/blob/master/seminars/day6/SparseVD-solution.ipynb), which is also available as an [assigment @ Colab](https://colab.research.google.com/github/bayesgroup/deepbayes-2019/blob/master/seminars/day6/SparseVD-assignment-colab.ipynb) from [DeepBayes](https://deepbayes.ru/) Summer School.\nThe code is not highly tuned but it is simple.\n\nThis repo contains the code for our ICML17 paper, [Variational Dropout Sparsifies Deep Neural Networks](https://arxiv.org/abs/1701.05369) ([talk](https://vimeo.com/238221185), [slides](https://docs.google.com/presentation/d/1Lg86MnGbksn3AtehADxSG-FcrbT2DMmGmjHUqG-EQYw/edit?usp=sharing), [poster](https://senya-ashukha.github.io/projects/svdo_icml17/svdo-poster-icml17.pdf), [blog-post](https://research.yandex.com/news/yandex-at-icml-2017-variational-dropout-sparsifies-deep-neural-networks)). \nWe showed that Variational Dropout leads to extremely sparse solutions both in fully-connected and convolutional layers. \nSparse VD reduced the number of parameters up to 280 times on LeNet architectures and up to 68 times on VGG-like networks with a negligible decrease of accuracy. \nThis effect is similar to the Automatic Relevance Determination effect in empirical Bayes.\nHowever, in Sparse VD the prior distribution remaines fixed, so there is no additional risk of overfitting.\n\nWe visualize the weights of Sparse VD LeNet-5-Caffe network and demonstrate several filters of the first convolutional layer and a piece of the fully-connected layer :)\n\n\u003cp align=\"center\"\u003e\n\u003cimg height=\"318\" src=\"http://senya-ashukha.github.io/projects/svdo_icml17/conv.gif\"/\u003e\n\u003cimg height=\"320\" src=\"http://senya-ashukha.github.io/projects/svdo_icml17/fc.gif\"/\u003e\n\u003c/p\u003e\n\n## ICML 2017 Oral Presentation by Dmitry Molchanov\n\n[![ICML 2017 Oral Presentation by Dmitry Molchanov](http://senya-ashukha.github.io/projects/svdo_icml17/icml2017-oral.png)](https://vimeo.com/238221185)\n\n## MNIST Experiments \n\nThe table containes the comparison of different sparsity-inducing techniques (Pruning (Han et al., 2015b;a), DNS (Guo et al., 2016), SWS (Ullrich et al., 2017)) on LeNet architectures.\nOur method provides the highest level of sparsity with a similar accuracy\n\n| Network       | Method   | Error | Sparsity per Layer  |  Compression |\n| -------------: | -------- | ----- | ------------------- | :--------------: |\n|               | Original | 1.64  |                     | 1              |\n|               | Pruning  | 1.59  | 92.0 − 91.0 − 74.0  | 12             |\n| LeNet-300-100 | DNS      | 1.99  | 98.2 − 98.2 − 94.5  | 56             |\n|               | SWS      | 1.94  |                     | 23             |\n| (ours)        | SparseVD | 1.92  | 98.9 − 97.2 − 62.0  | **68**         |\n||||||\n|               | Original | 0.8   |                     | 1              |\n|               | Pruning  | 0.77  | 34 − 88 − 92.0 − 81 | 12             |\n| LeNet-5       | DNS      | 0.91  | 86 − 97 − 99.3 − 96 | 111            |\n|               | SWH      | 0.97  |                     | 200            |\n| (ours)        | SparseVD | 0.75  | 67 − 98 − 99.8 − 95 | **280**        |\n\n\n## CIFAR Experiments\n\nThe plot contains the accuracy and sparsity level for VGG-like architectures of different sizes.\nThe number of neurons and filters scales as _k_.\nDense networks were trained with Binary Dropout, and Sparse VD networks were trained with Sparse Variational Dropout on all layers.\nThe overall sparsity level, achieved by our method, is reported as a dashed line.\nThe accuracy drop is negligible in most cases, and the sparsity level is high, especially in larger networks.\n\n\u003cp align=\"center\"\u003e\n\u003cimg height=\"318\" src=\"http://senya-ashukha.github.io/projects/svdo_icml17/vgg-compression.png\"/\u003e\n\u003c/p\u003e\n\n# Environment setup\n\n```(bash)\nsudo apt install virtualenv python-pip python-dev\nvirtualenv venv --system-site-packages\nsource venv/bin/activate\n\npip install numpy tabulate 'ipython[all]' sklearn matplotlib seaborn  \npip install --upgrade https://github.com/Theano/Theano/archive/rel-0.9.0.zip\npip install --upgrade https://github.com/Lasagne/Lasagne/archive/master.zip\n```\n\n# Launch experiments \n\n```(bash)\nsource ~/venv/bin/activate\ncd variational-dropout-sparsifies-dnn\nTHEANO_FLAGS='floatX=float32,device=gpu0,lib.cnmem=1' ipython ./experiments/\u003cexperiment\u003e.py\n```\n\n- If you have CuDNN problem please look at this [issue](https://github.com/ars-ashuha/variational-dropout-sparsifies-dnn/issues/3).\n- This [repo](https://github.com/nnnet/Variational-dropout-sparsifies-dnn) seems to use more up-to-date libs (Python 3.5 and Theano 1.0.0).\n\n# Further extensions\n\nThese two papers heavily rely on the Sparse Variational Dropout technique and extend it to other applications:\n* [Structured Bayesian Pruning via Log-Normal Multiplicative Noise](https://arxiv.org/abs/1705.07283) provides a way to enforse _structured_ sparsity using a similar technique. This method allows to remove entire neurons and convolutional filters, which results in lighter architectures and a significant inference speed-up with standard deep learning frameworks.\n* [Bayesian Compression for Natural Language Processing](https://arxiv.org/abs/1810.10927) adapts the Sparse Variational Dropout techniques for sparsification of various recurrent architectures. \n\n# Citation\n\nIf you found this code useful please cite our paper\n```\n@InProceedings{\n  molchanov2017variational,\n  title={Variational Dropout Sparsifies Deep Neural Networks},\n  author={Dmitry Molchanov and Arsenii Ashukha and Dmitry Vetrov},\n  booktitle={Proceedings of the 34th International Conference on Machine Learning},\n  year={2017}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbayesgroup%2Fvariational-dropout-sparsifies-dnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbayesgroup%2Fvariational-dropout-sparsifies-dnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbayesgroup%2Fvariational-dropout-sparsifies-dnn/lists"}