{"id":20972913,"url":"https://github.com/ashishpatel26/regularization-collection-deeplearning","last_synced_at":"2026-01-02T04:29:13.157Z","repository":{"id":107111207,"uuid":"268532662","full_name":"ashishpatel26/Regularization-Collection-Deeplearning","owner":"ashishpatel26","description":"This is Collection of Regularization Deep learning techniques with code and paper","archived":false,"fork":false,"pushed_at":"2020-06-01T13:41:26.000Z","size":2,"stargazers_count":3,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-13T08:35:20.231Z","etag":null,"topics":["alphadropout","deeplearning","dropout","dropout-keras","dropout-probability","dropout-pytorch","l1-normalization","l2normalization","machinelearning","regularization","regularization-hyperparameters","regularization-methods"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ashishpatel26.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-01T13:38:53.000Z","updated_at":"2021-11-08T09:31:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"37fe7835-f63d-4d70-904f-5bbfcd7d7aea","html_url":"https://github.com/ashishpatel26/Regularization-Collection-Deeplearning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ashishpatel26/Regularization-Collection-Deeplearning","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashishpatel26%2FRegularization-Collection-Deeplearning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashishpatel26%2FRegularization-Collection-Deeplearning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashishpatel26%2FRegularization-Collection-Deeplearning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashishpatel26%2FRegularization-Collection-Deeplearning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ashishpatel26","download_url":"https://codeload.github.com/ashishpatel26/Regularization-Collection-Deeplearning/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashishpatel26%2FRegularization-Collection-Deeplearning/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28169003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2026-01-02T02:00:06.235Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alphadropout","deeplearning","dropout","dropout-keras","dropout-probability","dropout-pytorch","l1-normalization","l2normalization","machinelearning","regularization","regularization-hyperparameters","regularization-methods"],"created_at":"2024-11-19T04:10:45.640Z","updated_at":"2026-01-02T04:29:13.131Z","avatar_url":"https://github.com/ashishpatel26.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"## New Regularization Technique for Deep Learning\n\n**Regularization** is a technique used for tuning the function by adding an additional penalty term in the error function. The additional term controls the excessively fluctuating function such that the coefficients don't take extreme values.\n\n---\n\n### **Famous Regularization Technique used By Industry**\n\n**1.L1 \u0026 L2 Regularization:**\n\n![Why do we need L2 \u0026 L1 regularizations ? - Analytics Vidhya - Medium](https://miro.medium.com/fit/c/140/140/1*Ri4zwZ86FM-2eR6ZQXDOaA.gif)\n\n![L1 vs L2 Regularization: The intuitive difference - Analytics ...](https://miro.medium.com/max/550/1*-LydhQEDyg-4yy5hGEj5wA.png)\n\n* **L1 regularization** adds an **L1** penalty equal to the absolute value of the magnitude of coefficients.\n* **L2 regularization** adds an **L2** penalty equal to the square of the magnitude of coefficients. \n\n**2.DropOut**\n\n![Machine Learning: A Newer Version Of A Lean Thinking Tool?](https://miro.medium.com/max/1872/1*-teDpAIho_nzNShRswkfrQ.gif)\n\n![img](https://miro.medium.com/max/2104/1*5Cg6JhNGJI2FXmptDd_RBQ.png)\n\n* **Dropout** is a **regularization** technique patented by Google for reducing overfitting in neural networks by preventing complex co-adaptations on training data.\n* **Eq. 1** shows loss for a regular network and Eq. 2 for a dropout network. In **Eq. 2**, the dropout rate **is 𝛿**, where **𝛿 ~ Bernoulli(*p*)**. This means 𝛿 is equal to **1** with probability *p* and **0** otherwise.\n* **Relationship between Dropout** and **Regularization**, A **Dropout rate of** 0.5 will lead **to** the maximum **regularization**, and. Generalization **of Dropout to** Gaussian-Dropout.\n* **Code:** [**Tensorflow.Keras**](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout) | [**Pytorch**](https://pytorch.org/docs/stable/nn.html?highlight=dropout#torch.nn.Dropout)\n* **Reference:** https://papers.nips.cc/paper/4878-understanding-dropout.pdf\n\n**Dropout Variant:** \n\n1. Alpha Dropout: [**Tensorflow**](https://www.tensorflow.org/api_docs/python/tf/keras/layers/AlphaDropout) | [**Pytorch**](https://pytorch.org/docs/master/generated/torch.nn.AlphaDropout.html) | [**Paper**](https://arxiv.org/abs/1706.02515)\n2. Gaussian Dropout: [**Tensorflow**](https://www.tensorflow.org/api_docs/python/tf/keras/layers/GaussianDropout) | **[Pytorch](https://github.com/j-min/Dropouts/blob/master/Gaussian_Variational_Dropout.ipynb)** | **Paper**\n3. Spatial Dropout: [**Tensorflow**](https://www.tensorflow.org/api_docs/python/tf/keras/layers/SpatialDropout1D) | [**Pytorch**](https://github.com/davidtvs/PyTorch-ENet/blob/master/models/enet.py) | **[Paper](https://arxiv.org/abs/1411.4280)**\n4. State Dropout: [**Tensorflow**](https://www.tensorflow.org/api_docs/python/tf/compat/v1/nn/rnn_cell/DropoutWrapper) | [**Pytorch**](https://discuss.pytorch.org/t/dropout-for-lstm-state-transitions/17112) | **Paper**\n5. Recurrent Dropout: [**Tensorflow**](https://www.tensorflow.org/addons/api_docs/python/tfa/rnn/LayerNormLSTMCell) | **[Pytorch](https://discuss.pytorch.org/t/how-to-use-lstmcell-with-layernorm/47747)** | [**Paper**](https://arxiv.org/abs/1607.06450)\n\n----\n\n***This Markdown is continuously Updated.. Stay Tuned...!!!***","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashishpatel26%2Fregularization-collection-deeplearning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fashishpatel26%2Fregularization-collection-deeplearning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashishpatel26%2Fregularization-collection-deeplearning/lists"}