{"id":19630927,"url":"https://github.com/aurelio-amerio/concretedropout","last_synced_at":"2026-03-17T20:37:29.443Z","repository":{"id":41881812,"uuid":"510382368","full_name":"aurelio-amerio/ConcreteDropout","owner":"aurelio-amerio","description":"Concrete Dropout implementation for Tensorflow 2.0 and PyTorch","archived":false,"fork":false,"pushed_at":"2024-12-18T14:20:42.000Z","size":151,"stargazers_count":13,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-05T07:11:08.590Z","etag":null,"topics":["concrete-dropout","deep-learning","dropout","machine-learning","neural-network","pytorch","pytorchlightning","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aurelio-amerio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-04T14:02:23.000Z","updated_at":"2025-01-31T01:49:21.000Z","dependencies_parsed_at":"2023-01-21T07:35:52.365Z","dependency_job_id":null,"html_url":"https://github.com/aurelio-amerio/ConcreteDropout","commit_stats":null,"previous_names":["aurelio-amerio/concretedropout-tf2"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aurelio-amerio%2FConcreteDropout","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aurelio-amerio%2FConcreteDropout/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aurelio-amerio%2FConcreteDropout/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aurelio-amerio%2FConcreteDropout/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aurelio-amerio","download_url":"https://codeload.github.com/aurelio-amerio/ConcreteDropout/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251265306,"owners_count":21561650,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concrete-dropout","deep-learning","dropout","machine-learning","neural-network","pytorch","pytorchlightning","tensorflow"],"created_at":"2024-11-11T12:07:09.758Z","updated_at":"2026-03-17T20:37:29.395Z","avatar_url":"https://github.com/aurelio-amerio.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ConcreteDropout\n[![Downloads](https://pepy.tech/badge/concretedropout)](https://pepy.tech/project/concretedropout)\n[![](https://img.shields.io/pypi/v/concretedropout.svg?maxAge=3600)](https://pypi.org/project/concretedropout/)\n\nConcrete Dropout updated implementation for Tensorflow 2.0 and PyTorch, following the [original code](https://github.com/yaringal/ConcreteDropout) from the paper.\n# Installation\nTo install this package, please use:\n```bash\npip install concretedropout\n```\n\n# Introduction\nConcrete dropout allows for the dropout probability of a layer to become a trainable parameter. For more information, see the original paper: [https://arxiv.org/abs/1705.07832](https://arxiv.org/abs/1705.07832)\n\nThis package implements Concrete Dropout for the following layers:\nTensorflow:\n- Dense - `tensorflow.ConcreteDenseDropout`\n- Conv1D - `tensorflow.ConcreteSpatialDropout1D`\n- Conv2D - `tensorflow.ConcreteSpatialDropout2D`\n- Conv3D - `tensorflow.ConcreteSpatialDropout3D`\n- DepthwiseConv1D - `tensorflow.ConcreteSpatialDropoutDepthwise1D`\n- DepthwiseConv2D - `tensorflow.ConcreteSpatialDropoutDepthwise2D`\n\nPyTorch:\n- Linear - `pytorch.ConcreteLinearDropout`\n- Conv1d - `pytorch.ConcreteDropout1d`\n- Conv2d - `pytorch.ConcreteDropout2d`\n- Conv3d - `pytorch.ConcreteDropout3d`\n\nPlease notice that the dropout layer will be applied **before** the chosen layer.\n\n# Arguments\nEach concrete dropout layer supports the following arguments:\n- `layer`: \n    an instance of the layer to which concrete dropout will be applied. Only required for Tensorflow.\n- `weight_regularizer=1e-6`:\n    A positive number which satisfies weight_regularizer = $l^2 / (\\tau * N)$ with prior lengthscale l, model precision τ (inverse observation noise), and N the number of instances in the dataset.\n    Note that kernel_regularizer is not needed.\n    The appropriate weight_regularizer value can be computed with the utility function `get_weight_regularizer(N, l, tau)`\n- `dropout_regularizer=1e-5`:\n    A positive number which satisfies dropout_regularizer = $2 / (\\tau * N)$ with model precision τ (inverse observation noise) and N the number of instances in the dataset.\n    Note the relation between dropout_regularizer and weight_regularizer: weight_regularizer / dropout_regularizer = $l^2 / 2$ with prior lengthscale l. Note also that the factor of two should be\n    ignored for cross-entropy loss, and used only for the eculedian loss.\n    The appropriate dropout_regularizer value can be computed with the utility function `get_dropout_regularizer(N, tau, cross_entropy_loss=False)`. By default, a regression problem will be assumed. \n- `init_min=0.1`: minimum value for the random initial dropout probability\n- `init_max=0.1`: maximum value for the random initial dropout probability\n- `is_mc_dropout=False`: enables Monte Carlo Dropout (i.e. dropout will remain active also at prediction time). Default: False. \n- `data_format=None`: channels_last or channels_first (only for Tensorflow). Defaults to channels_last for Tensorflow. PyTorch defaults to channel_first.\n- `temperature`: temperature of the concrete distribution. For more information see [arXiv:1611.00712](https://arxiv.org/abs/1611.00712). Defaults to `0.1` for dense layers, and `2/3` for convolution layers.\n\n# Example\nThe suggested way to employ concrete dropout layers is the following.\nTensorflow:\n```python\nimport tensorflow as tf\nfrom concretedropout.tensorflow import ConcreteDenseDropout, get_weight_regularizer, get_dropout_regularizer \n\n#... import the dataset\nNs = x_train.shape[0]\n# get the regularizers\nwr = get_weight_regularizer(Ns, l=1e-2, tau=1.0) # tau is the inverse \ndr = get_dropout_regularizer(Ns, tau=1.0, cross_entropy_loss=True)\n\n# ... a neural network with output x\ndense1 = tf.keras.layers.Dense(N_neurons)\nx = ConcreteDenseDropout(dense1, weight_regularizer=wr, dropout_regularizer=dr)(x)\n```\nPyTorch:\n```python\nimport torch \nfrom concretedropout.pytorch import ConcreteDropout, ConcreteLinearDropout, get_weight_regularizer, get_dropout_regularizer\n\n#... import the dataset\nNs = x_train.shape[0]\n# get the regularizers\nwr = get_weight_regularizer(Ns, l=1e-2, tau=1.0) # tau is the inverse \ndr = get_dropout_regularizer(Ns, tau=1.0, cross_entropy_loss=True)\n\n# ... a neural network with output x\nlinear = torch.nn.Linear(n_input, N_neurons)\nx = ConcreteLinearDropout(weight_regularizer=wr, dropout_regularizer=dr)(x, linear)\n\n# inside the train step of your model, you need to add a new regularization term, which is due to the concrete dropout:\ndef training_step(self, batch, batch_nb):\n    x, y = batch\n    output = self(x)\n    \n    reg = torch.zeros(1) # get the regularization term\n    for module in filter(lambda x: isinstance(x, ConcreteDropout), self.modules()):\n        reg += module.regularization\n\n    loss = self.loss(output, y) + reg # add the reg term\n    return loss\n```\nFor a practical example on how to use concrete dropout for the mnist dataset, see this [Tensorflow example](https://github.com/aurelio-amerio/ConcreteDropout-TF2/blob/main/examples/Tensorflowmnist_convnet_concrete_dropout.ipynb) and this [PyTorch example](https://github.com/aurelio-amerio/ConcreteDropout-TF2/blob/main/examples/PyTorch/MNIST_pytorch.ipynb).\n\n# Bayesian neural network with MCDropout\nYou can find [here](https://github.com/aurelio-amerio/ConcreteDropout-TF2/blob/main/examples/regression_MCDropout.ipynb) an example on how to use MCDropout and Concrete Dropout to implement a Bayesian Neural Network with MCDropout on Tensorflow. For more information, see [arXiv:1506.02142](https://arxiv.org/abs/1506.02142).\n\n# Known issues\nDue to the way the additional dropout loss term is added to the main loss term, during training and evaluation the **model loss** might become a **negative number**. This has no impact on the actual optimisation of the model. If you desire to track your loss function separately, as a work around it is advised to add it to the list of metrics. \n\n# Aknowledgements\nThis library stems from a research project supported with Cloud TPUs from Google's TPU Research Cloud (TRC).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faurelio-amerio%2Fconcretedropout","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faurelio-amerio%2Fconcretedropout","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faurelio-amerio%2Fconcretedropout/lists"}