https://github.com/takashiishida/flooding
[ICML 2020] code for the flooding regularizer proposed in "Do We Need Zero Training Loss After Achieving Zero Training Error?"
https://github.com/takashiishida/flooding
deep-learning pytorch
Last synced: about 1 year ago
JSON representation
[ICML 2020] code for the flooding regularizer proposed in "Do We Need Zero Training Loss After Achieving Zero Training Error?"
- Host: GitHub
- URL: https://github.com/takashiishida/flooding
- Owner: takashiishida
- License: gpl-3.0
- Created: 2020-02-10T03:52:07.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2022-12-09T10:54:13.000Z (over 3 years ago)
- Last Synced: 2025-03-25T00:40:44.460Z (over 1 year ago)
- Topics: deep-learning, pytorch
- Language: Python
- Homepage:
- Size: 433 KB
- Stars: 91
- Watchers: 4
- Forks: 9
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# flooding
Demo code for the flooding regularizer proposed in "[Do We Need Zero Training Loss After Achieving Zero Training Error?](https://arxiv.org/abs/2002.08709)" (ICML 2020) by Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama.
## Introduction
Overparameterized deep networks have the capacity to memorize training data with zero training error.
Even after memorization, the training loss continues to approach zero, making the model overconfident and the test performance degraded.
*Flooding* is a direct solution to this issue.
It intentionally prevents further reduction of the training loss when it reaches a reasonably small value, which is called the "flooding level".
Flooding makes the loss float around the flooding level by doing
- (mini-batched) gradient *descent* as usual but
- gradient *ascent* if the training loss is below the flooding level.
This can be implemented by adding one line to your code:
```python
loss = (loss-b).abs() + b # b is the flooding level.
```
and is compatible with any stochastic optimizer and other regularizers.
*Figures from [the paper](https://arxiv.org/abs/2002.08709): (a) shows 3 different concepts related to overfitting. [A] shows the generalization gap increases, while training & test losses decrease. [B] also shows the increasing gap, but the test loss starts to rise. [C] shows the training loss becoming (near-)zero. We avoid [C] by flooding the bottom area, visualized in (b), which forces the training loss to stay around a constant. This leads to a decreasing test loss once again. The paper presents experiments with CIFAR-10 shown in (c)–(d) supporting these claims.*
## Demo
### Requirements
- pytorch 1.3.1
- mlflow 1.5.0
- matplotlib 3.1.1
### Execution
Run the following python script for a simple demonstration of flooding with synthetic datasets:
```bash
python run_demo_synth.py
```
Two png files will be saved in your directory, which are visualizations of the results with synthetic datasets.
See the [paper](https://arxiv.org/abs/2002.08709) for the details of the synthetic datasets.