{"id":20663673,"url":"https://github.com/vita-group/alleviate-robust-overfitting","last_synced_at":"2025-04-19T15:55:56.886Z","repository":{"id":107045021,"uuid":"329199045","full_name":"VITA-Group/Alleviate-Robust-Overfitting","owner":"VITA-Group","description":"[ICLR 2021] \"Robust Overfitting may be mitigated by properly learned smoothening\" by Tianlong Chen*, Zhenyu Zhang*, Sijia Liu, Shiyu Chang, Zhangyang Wang","archived":false,"fork":false,"pushed_at":"2021-12-30T10:32:39.000Z","size":661,"stargazers_count":46,"open_issues_count":1,"forks_count":5,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-29T09:42:01.162Z","etag":null,"topics":["adversarial-robustness","generalization-ability","overfitting","smoothing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VITA-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-13T05:02:13.000Z","updated_at":"2024-10-21T06:59:17.000Z","dependencies_parsed_at":"2023-07-12T01:01:41.905Z","dependency_job_id":null,"html_url":"https://github.com/VITA-Group/Alleviate-Robust-Overfitting","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FAlleviate-Robust-Overfitting","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FAlleviate-Robust-Overfitting/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FAlleviate-Robust-Overfitting/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FAlleviate-Robust-Overfitting/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VITA-Group","download_url":"https://codeload.github.com/VITA-Group/Alleviate-Robust-Overfitting/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249731218,"owners_count":21317341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-robustness","generalization-ability","overfitting","smoothing"],"created_at":"2024-11-16T19:19:14.173Z","updated_at":"2025-04-19T15:55:56.880Z","avatar_url":"https://github.com/VITA-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Robust Overfitting may be mitigated by properly learned smoothening\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\nCode for this paper [Robust Overfitting may be mitigated by properly learned smoothing](https://openreview.net/forum?id=qZzy5urZw9)\n\nTianlong Chen\\*, Zhenyu Zhang\\*, Sijia Liu, Shiyu Chang, Zhangyang Wang\n\n## Overview\n\nTo alleviate the intriguing problem of robust overfitting, we investigate two empirical means to inject more learned smoothening during adversarial training (**AT**): one leveraging knowledge distillation (**KD**) and self-training to smooth the logits, the other performing stochastic weight averaging (**SWA**) to smooth the weights \n\nHighlights:\n\n- **Smoothening mitigates robust overfitting:**  After adopting KD and SWA in AT, we mitigated robust overfitting and achieve a better trade-off between standard test accuracy and robustness than early stopping.\n- **Rich ablation experiments**:  We conducted plenty of ablation experiments and visualizations to investigate the reason why robust overfitting may be mitigated by these smoothening approaches.\n\n## Experiment Results\n\n**Training with KD and SWA to mitigate robust overfitting**\n\n![](https://github.com/VITA-Group/Alleviate-Robust-Overfitting/blob/main/Figs/train_curve.png)\n\n**Flattening the rugged input space** \n\n![](https://raw.githubusercontent.com/VITA-Group/Alleviate-Robust-Overfitting/main/Figs/flatness.png)\n\n## Prerequisites\n\n- pytorch 1.5.1\n- torchvision 0.6.1 \n- advertorch 0.2.3\n\n## Usage\n\n**Standard Training:**\n\n```\npython -u main_std.py \\\n\t--data [dataset direction] \\ \n\t--dataset cifar10 \\\n\t--arch resnet18 \\\n\t--save_dir std_cifar10_resnet18 \n```\n\n**PGD Adversarial Training:**\n\n```\npython -u main_adv.py \\\n\t--data [dataset direction] \\ \n\t--dataset cifar10 \\\n\t--arch resnet18 \\\n\t--save_dir AT_cifar10_resnet18 \n```\n\n**Adversarial Training with KD\u0026SWA:**\n\n```\npython -u main_adv.py \\\n\t--data [dataset direction] \\ \n\t--dataset cifar10 \\\n\t--arch resnet18 \\\n\t--save_dir KDSWA_cifar10_resnet18 \\\n\t--swa \\\n\t--lwf \\\n\t--t_weight1 pretrained_models/cifar10_resnet18_std_SA_best.pt \\\n\t--t_weight2 pretrained_models/cifar10_resnet18_adv_RA_best.pt\n```\n\n**Testing under PGD-20 Linf eps=8/255** :\n\n```\npython -u main_adv.py \\\n\t--data [dataset direction] \\\n\t--dataset cifar10 \\\n\t--arch resnet18 \\\n\t--eval \\\n\t--pretrained pretrained_models/**.pt \\\n\t--swa #if test with swa_model\n```\n\n## Citation\n\n```\n@inproceedings{\n\tchen2021robust,\n\ttitle={Robust Overfitting may be mitigated by properly learned smoothening},\n\tauthor={Tianlong Chen and Zhenyu Zhang and Sijia Liu and Shiyu Chang and Zhangyang Wang},\n\tbooktitle={International Conference on Learning Representations},\n\tyear={2021},\n\turl={https://openreview.net/forum?id=qZzy5urZw9}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Falleviate-robust-overfitting","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvita-group%2Falleviate-robust-overfitting","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Falleviate-robust-overfitting/lists"}