https://github.com/vita-group/alleviate-robust-overfitting
[ICLR 2021] "Robust Overfitting may be mitigated by properly learned smoothening" by Tianlong Chen*, Zhenyu Zhang*, Sijia Liu, Shiyu Chang, Zhangyang Wang
https://github.com/vita-group/alleviate-robust-overfitting
adversarial-robustness generalization-ability overfitting smoothing
Last synced: about 1 year ago
JSON representation
[ICLR 2021] "Robust Overfitting may be mitigated by properly learned smoothening" by Tianlong Chen*, Zhenyu Zhang*, Sijia Liu, Shiyu Chang, Zhangyang Wang
- Host: GitHub
- URL: https://github.com/vita-group/alleviate-robust-overfitting
- Owner: VITA-Group
- License: mit
- Created: 2021-01-13T05:02:13.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2021-12-30T10:32:39.000Z (over 4 years ago)
- Last Synced: 2025-03-29T09:42:01.162Z (about 1 year ago)
- Topics: adversarial-robustness, generalization-ability, overfitting, smoothing
- Language: Python
- Homepage:
- Size: 646 KB
- Stars: 46
- Watchers: 8
- Forks: 5
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Robust Overfitting may be mitigated by properly learned smoothening
[](https://opensource.org/licenses/MIT)
Code for this paper [Robust Overfitting may be mitigated by properly learned smoothing](https://openreview.net/forum?id=qZzy5urZw9)
Tianlong Chen\*, Zhenyu Zhang\*, Sijia Liu, Shiyu Chang, Zhangyang Wang
## Overview
To alleviate the intriguing problem of robust overfitting, we investigate two empirical means to inject more learned smoothening during adversarial training (**AT**): one leveraging knowledge distillation (**KD**) and self-training to smooth the logits, the other performing stochastic weight averaging (**SWA**) to smooth the weights
Highlights:
- **Smoothening mitigates robust overfitting:** After adopting KD and SWA in AT, we mitigated robust overfitting and achieve a better trade-off between standard test accuracy and robustness than early stopping.
- **Rich ablation experiments**: We conducted plenty of ablation experiments and visualizations to investigate the reason why robust overfitting may be mitigated by these smoothening approaches.
## Experiment Results
**Training with KD and SWA to mitigate robust overfitting**
**Flattening the rugged input space**
## Prerequisites
- pytorch 1.5.1
- torchvision 0.6.1
- advertorch 0.2.3
## Usage
**Standard Training:**
```
python -u main_std.py \
--data [dataset direction] \
--dataset cifar10 \
--arch resnet18 \
--save_dir std_cifar10_resnet18
```
**PGD Adversarial Training:**
```
python -u main_adv.py \
--data [dataset direction] \
--dataset cifar10 \
--arch resnet18 \
--save_dir AT_cifar10_resnet18
```
**Adversarial Training with KD&SWA:**
```
python -u main_adv.py \
--data [dataset direction] \
--dataset cifar10 \
--arch resnet18 \
--save_dir KDSWA_cifar10_resnet18 \
--swa \
--lwf \
--t_weight1 pretrained_models/cifar10_resnet18_std_SA_best.pt \
--t_weight2 pretrained_models/cifar10_resnet18_adv_RA_best.pt
```
**Testing under PGD-20 Linf eps=8/255** :
```
python -u main_adv.py \
--data [dataset direction] \
--dataset cifar10 \
--arch resnet18 \
--eval \
--pretrained pretrained_models/**.pt \
--swa #if test with swa_model
```
## Citation
```
@inproceedings{
chen2021robust,
title={Robust Overfitting may be mitigated by properly learned smoothening},
author={Tianlong Chen and Zhenyu Zhang and Sijia Liu and Shiyu Chang and Zhangyang Wang},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=qZzy5urZw9}
}
```