{"id":29704601,"url":"https://github.com/adaptinfer/dropout_interactions","last_synced_at":"2025-07-23T14:11:39.034Z","repository":{"id":114003354,"uuid":"484792770","full_name":"AdaptInfer/Dropout_Interactions","owner":"AdaptInfer","description":"Code for \"Dropout as a Regularizer of Interaction Effects\"","archived":false,"fork":false,"pushed_at":"2022-04-23T17:42:45.000Z","size":4411,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-04T10:53:29.224Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AdaptInfer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-23T16:01:10.000Z","updated_at":"2024-06-13T17:55:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"e771492f-0019-4a57-8c55-58ac67c5818e","html_url":"https://github.com/AdaptInfer/Dropout_Interactions","commit_stats":null,"previous_names":["lengerichlab/dropout_interactions","adaptinfer/dropout_interactions"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AdaptInfer/Dropout_Interactions","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdaptInfer%2FDropout_Interactions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdaptInfer%2FDropout_Interactions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdaptInfer%2FDropout_Interactions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdaptInfer%2FDropout_Interactions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AdaptInfer","download_url":"https://codeload.github.com/AdaptInfer/Dropout_Interactions/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdaptInfer%2FDropout_Interactions/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266691598,"owners_count":23969189,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-23T14:11:36.961Z","updated_at":"2025-07-23T14:11:39.000Z","avatar_url":"https://github.com/AdaptInfer.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dropout as a Regularizer of Interaction Effects\n\nWe know Dropout works well, but *how* does it work? In [our paper](https://arxiv.org/abs/2007.00823), we propose a simple view of Dropout: Dropout regularizes against high-order interaction effects.\n\nThis can be seen intuitively by the symmetry between the hypothesis space of interaction effects and Dropout regularization:\nGiven \u003cimg src=\"https://render.githubusercontent.com/render/math?math=N\"\u003e variables, there are \u003cimg src=\"https://render.githubusercontent.com/render/math?math={N \\choose k}\"\u003e possible sets of \u003cimg src=\"https://render.githubusercontent.com/render/math?math=k\"\u003e variables (\u003cimg src=\"https://render.githubusercontent.com/render/math?math=N\"\u003e univariate effects, \u003cimg src=\"https://render.githubusercontent.com/render/math?math=\\mathcal{O}(N^2)\"\u003e pairwise interactions, \u003cimg src=\"https://render.githubusercontent.com/render/math?math=\\mathcal{O}(N^3)\"\u003e 3-way interactions); a \u003cimg src=\"https://render.githubusercontent.com/render/math?math=k\"\u003e-order interaction effect survives Dropout at rate \u003cimg src=\"https://render.githubusercontent.com/render/math?math=p\"\u003e with probability \u003cimg src=\"https://render.githubusercontent.com/render/math?math=(1-p)^k\"\u003e. These rates effectively cancel:\n\n![Preview](symmetry.png)\n\nIf you use the ideas in this paper, please cite:\n\n```\n@InProceedings{LengerichDropout2022,\n  title={Dropout as a Regularizer of Interaction Effects},\n  author={Lengerich, Benjamin and Xing, Eric P. and Caruana, Rich},\n  journal={{Proceedings of the Twenty Fifth International Conference on Artificial Intelligence and Statistics}},\n  year={2022},\n  url_Paper = {https://arxiv.org/abs/2007.00823},\n  abstract = {We examine Dropout through the perspective of interactions: effects that require multiple variables. Given $N$ variables, there are ${N \\choose k}$ possible sets of $k$ variables ($N$ univariate effects, $\\mathcal{O}(N^2)$ pairwise interactions, $\\mathcal{O}(N^3)$ 3-way interactions); we can thus imagine that models with large representational capacity could be dominated by high-order interactions. In this paper, we show that Dropout contributes a regularization effect which helps neural networks (NNs) explore functions of lower-order interactions before considering functions of higher-order interactions. Dropout imposes this regularization by reducing the effective learning rate of higher-order interactions. As a result, Dropout encourages models to learn lower-order functions of additive components.\nThis understanding of Dropout has implications for choosing Dropout rates: higher Dropout rates should be used when we need stronger regularization against interactions.\nThis perspective also issues caution against using Dropout to measure term salience because Dropout regularizes against high-order interactions.\nFinally, this view of Dropout as a regularizer of interactions provides insight into the varying effectiveness of Dropout across architectures and datasets.\nWe also compare Dropout to weight decay and early stopping and find that it is difficult to obtain the same regularization with these alternatives.}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadaptinfer%2Fdropout_interactions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadaptinfer%2Fdropout_interactions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadaptinfer%2Fdropout_interactions/lists"}