{"id":26055240,"url":"https://github.com/takashiishida/flooding","last_synced_at":"2025-04-11T02:51:31.824Z","repository":{"id":71025717,"uuid":"239422459","full_name":"takashiishida/flooding","owner":"takashiishida","description":"[ICML 2020] code for the flooding regularizer proposed in \"Do We Need Zero Training Loss After Achieving Zero Training Error?\"","archived":false,"fork":false,"pushed_at":"2022-12-09T10:54:13.000Z","size":443,"stargazers_count":91,"open_issues_count":0,"forks_count":9,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-25T00:40:44.460Z","etag":null,"topics":["deep-learning","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/takashiishida.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-02-10T03:52:07.000Z","updated_at":"2025-02-20T15:17:54.000Z","dependencies_parsed_at":"2023-02-24T16:46:26.428Z","dependency_job_id":null,"html_url":"https://github.com/takashiishida/flooding","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takashiishida%2Fflooding","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takashiishida%2Fflooding/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takashiishida%2Fflooding/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/takashiishida%2Fflooding/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/takashiishida","download_url":"https://codeload.github.com/takashiishida/flooding/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248333112,"owners_count":21086193,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","pytorch"],"created_at":"2025-03-08T10:00:54.547Z","updated_at":"2025-04-11T02:51:31.806Z","avatar_url":"https://github.com/takashiishida.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# flooding\nDemo code for the flooding regularizer proposed in \"[Do We Need Zero Training Loss After Achieving Zero Training Error?](https://arxiv.org/abs/2002.08709)\" (ICML 2020) by Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama.\n\n## Introduction\n\nOverparameterized deep networks have the capacity to memorize training data with zero training error.\nEven after memorization, the training loss continues to approach zero, making the model overconfident and the test performance degraded.\n\n*Flooding* is a direct solution to this issue.\nIt intentionally prevents further reduction of the training loss when it reaches a reasonably small value, which is called the \"flooding level\".\nFlooding makes the loss float around the flooding level by doing\n- (mini-batched) gradient *descent* as usual but\n- gradient *ascent* if the training loss is below the flooding level.\n\nThis can be implemented by adding one line to your code:\n```python\nloss = (loss-b).abs() + b  # b is the flooding level.\n```\nand is compatible with any stochastic optimizer and other regularizers.\n![](fig1.png)\n*Figures from [the paper](https://arxiv.org/abs/2002.08709): (a) shows 3 different concepts related to overfitting. [A] shows the generalization gap increases, while training \u0026 test losses decrease. [B] also shows the increasing gap, but the test loss starts to rise. [C] shows the training loss becoming (near-)zero. We avoid [C] by flooding the bottom area, visualized in (b), which forces the training loss to stay around a constant. This leads to a decreasing test loss once again. The paper presents experiments with CIFAR-10 shown in (c)–(d) supporting these claims.*\n\n## Demo\n\n### Requirements\n- pytorch 1.3.1\n- mlflow 1.5.0\n- matplotlib 3.1.1\n\n### Execution\nRun the following python script for a simple demonstration of flooding with synthetic datasets:\n```bash\npython run_demo_synth.py\n```\nTwo png files will be saved in your directory, which are visualizations of the results with synthetic datasets.\nSee the [paper](https://arxiv.org/abs/2002.08709) for the details of the synthetic datasets.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftakashiishida%2Fflooding","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftakashiishida%2Fflooding","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftakashiishida%2Fflooding/lists"}