{"id":49471633,"url":"https://github.com/dingo-actual/dropgrad","last_synced_at":"2026-04-30T16:36:16.211Z","repository":{"id":211792885,"uuid":"729960237","full_name":"dingo-actual/dropgrad","owner":"dingo-actual","description":"PyTorch Implementation of DropGrad","archived":false,"fork":false,"pushed_at":"2024-04-27T05:40:08.000Z","size":21,"stargazers_count":4,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-30T16:36:12.967Z","etag":null,"topics":["neural-network","pytorch","regularization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dingo-actual.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-12-10T21:43:28.000Z","updated_at":"2024-09-03T19:59:11.000Z","dependencies_parsed_at":"2023-12-13T14:53:31.027Z","dependency_job_id":null,"html_url":"https://github.com/dingo-actual/dropgrad","commit_stats":null,"previous_names":["dingo-actual/dropgrad"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dingo-actual/dropgrad","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dingo-actual%2Fdropgrad","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dingo-actual%2Fdropgrad/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dingo-actual%2Fdropgrad/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dingo-actual%2Fdropgrad/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dingo-actual","download_url":"https://codeload.github.com/dingo-actual/dropgrad/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dingo-actual%2Fdropgrad/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32470879,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"ssl_error","status_checked_at":"2026-04-30T13:12:06.837Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["neural-network","pytorch","regularization"],"created_at":"2026-04-30T16:36:15.535Z","updated_at":"2026-04-30T16:36:16.206Z","avatar_url":"https://github.com/dingo-actual.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DropGrad: A Simple Method for Regularization and Accelerated Optimization of Neural Networks\n\n- [DropGrad: A Simple Method for Regularization and Accelerated Optimization of Neural Networks](#dropgrad-a-simple-method-for-regularization-and-accelerated-optimization-of-neural-networks)\n  - [Installation](#installation)\n    - [Requirements](#requirements)\n    - [Using pip](#using-pip)\n    - [Using git](#using-git)\n  - [Usage](#usage)\n    - [Basic Usage](#basic-usage)\n    - [Use with Learning Rate Schedulers](#use-with-learning-rate-schedulers)\n    - [Varying `drop_rate` per `Parameter`](#varying-drop_rate-per-parameter)\n  - [TODO 🚧](#todo-)\n\nDropGrad is a regularization method for neural networks that works by randomly (and independently) setting gradient values to zero before an optimization step. Similarly to Dropout, it has a single parameter, `drop_rate`, the probability of setting each parameter gradient to zero. In order to de-bias the remaining gradient values, they are divided by `1.0 - drop_rate`.\n\n\u003e To the best of my knowledge DropGrad is an original contribution. However, I have no plans of publishing a paper.\n\u003e If indeed, it is an original method, please feel free to publish a paper about DropGrad. If you do so, all I ask is\n\u003e that you mention me in your publication and cite this repository.\n\n## Installation\n\nThe PyTorch implementation of DropGrad can be installed simply using pip or by cloning the current GitHub repo.\n\n### Requirements\n\nThe only requirement for DropGrad is PyTorch. (Only versions of PyTorch \u003e= 2.0 have been tested, although DropGrad should be compatible with any version of PyTorch)\n\n### Using pip\n\nTo install using pip:\n\n```bash\npip install dropgrad\n```\n\n### Using git\n\n```bash\ngit clone https://github.com/dingo-actual/dropgrad.git\ncd dropgrad\npython -m build\npip install dist/dropgrad-0.1.0-py3-none-any.whl\n```\n\n## Usage\n\n### Basic Usage\n\nTo use DropGrad in your neural network optimization, simply import the `DropGrad` class to wrap your optimizer.\n\n```python\nfrom dropgrad import DropGrad\n```\n\nWrapping an optimizer is similar to using a learning rate scheduler:\n\n```python\nopt_unwrapped = Adam(net.parameters(), lr=1e-3)\nopt = DropGrad(opt_unwrapped, drop_rate=0.1)\n```\n\nDuring training, the application of DropGrad is automatically handled by the wrapper. Simply call `.step()` on\nthe wrapped optimizer to apply DropGrad then `.zero_grad()` to reset the gradients.\n\n```python\nopt.step()\nopt.zero_grad()\n```\n\n### Use with Learning Rate Schedulers\n\nIf you use a learning rate scheduler as well as DropGrad, simply pass the base optimizer to both the DropGrad\nwrapper and the learning rate scheduler:\n\n```python\nopt_unwrapped = Adam(net.parameters(), lr=1e-3)\nlr_scheduler = CosineAnnealingLR(opt_unwrapped, T_max=100)\nopt = DropGrad(opt_unwrapped, drop_rate=0.1)\n```\n\nDuring the training loop, you call `.step()` on the DropGrad wrapper before calling `.step()` on the learning rate\nscheduler, similarly to using an optimizer without DropGrad:\n\n```python\nfor epoch_n in range(n_epochs):\n    for x_batch, y_batch in dataloader:\n        pred_batch = net(x_batch)\n        loss = loss_fn(pred_batch, y_batch)\n\n        loss.backward()\n\n        opt.step()\n        opt.zero_grad()\n\n    lr_scheduler.step()\n```\n\n### Varying `drop_rate` per `Parameter`\n\nDropGrad allows the user to set a different drop rate for each `Parameter` under optimization. To do this, simply\npass a dictionary mapping `Parameters` to drop rates to the `drop_rate` argument of the DropGrad wrapper. If a dictionary\nis passed to DropGrad during initialization, all optimized `Parameter`s that are not present in that dictionary will have\nthe drop rate passed to the DropGrad wrapper at initialization (if `drop_rate=None` then drop grad simply won't be applied\nto `Parameter`s that are not present in the dictionary).\n\nThe example below will apply a `drop_rate` of 0.1 to all optimized weights and a `drop_rate` of 0.01 to all optimized biases,\nwith no DropGrad applied to any other optimized `Parameter`s:\n\n```python\ndrop_rate_weights = 0.1\ndrop_rate_biases = 0.01\n\nparams_weights = [p for name, p in net.named_parameters() if p.requires_grad and 'weight' in name]\nparams_biases = [p for name, p in net.named_parameters() if p.requires_grad and 'bias' in name]\n\nparam_drop_rates = {p: drop_rate_weights for p in params_weights}\nparam_drop_rates.update({p: drop_rate_biases for p in params_biases})\n\nopt_unwrapped = Adam(net.parameters(), lr=1e-3)\nopt = DropGrad(opt_unwrapped, drop_rate=None, params=param_drop_rates)\n```\n\n## TODO 🚧\n\n- [ ] Write analysis of DropGrad\n- [ ] Implement drop rate schedulers\n- [ ] Implement option to apply \"full\" update drop by interrupting `.step()`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdingo-actual%2Fdropgrad","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdingo-actual%2Fdropgrad","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdingo-actual%2Fdropgrad/lists"}