{"id":13935830,"url":"https://github.com/LayneH/self-adaptive-training","last_synced_at":"2025-07-19T21:30:44.628Z","repository":{"id":108369566,"uuid":"242276268","full_name":"LayneH/self-adaptive-training","owner":"LayneH","description":"[TPAMI2022 \u0026 NeurIPS2020] Official implementation of Self-Adaptive Training","archived":false,"fork":false,"pushed_at":"2021-10-17T09:31:45.000Z","size":94,"stargazers_count":127,"open_issues_count":1,"forks_count":23,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-27T03:34:52.703Z","etag":null,"topics":["adversarial-robustness","computer-vision","generalization","label-noise","machine-learning","overfitting"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LayneH.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-22T03:59:50.000Z","updated_at":"2024-08-02T00:24:52.000Z","dependencies_parsed_at":"2023-03-06T02:45:34.564Z","dependency_job_id":null,"html_url":"https://github.com/LayneH/self-adaptive-training","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/LayneH/self-adaptive-training","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LayneH%2Fself-adaptive-training","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LayneH%2Fself-adaptive-training/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LayneH%2Fself-adaptive-training/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LayneH%2Fself-adaptive-training/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LayneH","download_url":"https://codeload.github.com/LayneH/self-adaptive-training/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LayneH%2Fself-adaptive-training/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266019657,"owners_count":23864916,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-robustness","computer-vision","generalization","label-noise","machine-learning","overfitting"],"created_at":"2024-08-07T23:02:07.660Z","updated_at":"2025-07-19T21:30:44.205Z","avatar_url":"https://github.com/LayneH.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Self-Adaptive Training\r\nThis is the PyTorch implementation of the \r\n- NeurIPS'2020 paper [Self-Adaptive Training: beyond Empirical Risk Minimization](https://arxiv.org/abs/2002.10319)，\r\n- Journal version [Self-Adaptive Training: Bridging the Supervised and Self-Supervised Learning](https://arxiv.org/abs/2101.08732).\r\n\r\n\r\nSelf-adaptive training significantly improves the generalization of deep networks under noise and enhances the self-supervised representation learning. It also advances the state-of-the-art on *learning with noisy label*, *adversarial training* and the *linear evaluation* on the learned representation.\r\n\r\n## News\r\n- 2021.10: The code of Selective Classification for SAT has been released [here](https://github.com/LayneH/SAT-selective-cls).\r\n- 2021.01: We have released the journal version of Self-Adaptive Training, which is a unified algorithm for both the supervised and self-supervised learning. Code for self-supervised learning will be available soon.\r\n- 2020.09: Our work has been accepted at NeurIPS'2020.\r\n\r\n## Requirements\r\n\r\n- Python \u003e= 3.6\r\n- PyTorch \u003e= 1.0\r\n- CUDA\r\n- Numpy\r\n\r\n## Usage\r\n### Standard training\r\nThe `main.py` contains training and evaluation functions in standard training setting.\r\n#### Runnable scripts\r\n- Training and evaluation using the default parameters\r\n  \r\n  We provide our training scripts in directory `scripts/`. For a concrete example, we can use the command as below to train the default model (i.e., ResNet-34) on CIFAR10 dataset with uniform label noise injected (e.g., 40%):\r\n  ```bash\r\n  $ bash scripts/cifar10/run_sat.sh [TRIAL_NAME]\r\n  ```\r\n  The argument `TRIAL_NAME` is optional, it helps us to identify different trials of the same experiments without modifying the training script. The evaluation is automatically performed when training is finished.\r\n\r\n- Additional arguments\r\n  - `noise-rate`: the percentage of data that being corrupted\r\n  - `noise-type`: type of random corruptions (i.e., corrupted_label, Gaussian,random_pixel, shuffled_pixel)\r\n  - `sat-es`: initial epochs of our approach\r\n  - `sat-alpha`: the momentum term $\\alpha$ of our approach\r\n  - `arch`: the architecture of backbone model, e.g., resnet34/wrn34\r\n\r\n\r\n#### Results on CIFAR datasets under uniform label noise\r\n- Test Accuracy(%) on CIFAR10\r\n\r\n|Noise Rate         |0.2    |0.4    |0.6    |0.8    |\r\n|-------------------|-------|-------|-------|-------|\r\n|ResNet-34          |94.14  | 92.64 |89.23  |78.58  |\r\n|WRN-28-10          |94.84  | 93.23 |89.42  |80.13  |\r\n\r\n\r\n- Test Accuracy(%) on CIFA100\r\n\r\n|Noise Rate         |0.2    |0.4    |0.6    |0.8    |\r\n|-------------------|-------|-------|-------|-------|\r\n|ResNet-34          |75.77  |71.38  |62.69  |38.72  |\r\n|WRN-28-10          |77.71  | 72.60 |64.87  |44.17  |\r\n\r\n\r\n#### Runnable scripts for repreducing double-descent phenomenon\r\nYou can use the command as below to train the default model (i.e., ResNet-18) on CIFAR10 dataset with 16.67% uniform label noise injected (i.e., 15% label *error* rate):\r\n  ```bash\r\n  $ bash scripts/cifar10/run_sat_dd_parallel.sh [TRIAL_NAME]\r\n  $ bash scripts/cifar10/run_ce_dd_parallel.sh [TRIAL_NAME]\r\n  ```\r\n\r\n\r\n#### Double-descent ERM vs. single-descent self-adaptive training\r\n\u003cp align=\"center\"\u003e\r\n    \u003cimg src=\"images/model_dd.png\" width=\"450\"\\\u003e\r\n\u003c/p\u003e\r\n\u003cp align=\"center\"\u003e\r\nDouble-descent ERM vs. single-descent self-adaptive training on the error-capacity curve. The vertical dashed line represents the interpolation threshold.\r\n\u003c/p\u003e\r\n\r\n\u003cp align=\"center\"\u003e\r\n    \u003cimg src=\"images/epoch_dd.png\" width=\"450\"\\\u003e\r\n\u003c/p\u003e\r\n\u003cp align=\"center\"\u003e\r\nDouble-descent ERM vs. single-descent self-adaptive training on the epoch-capacity curve. The dashed vertical line represents the initial epoch E_s of our approach.\r\n\u003c/p\u003e\r\n\r\n\r\n### Adversarial training\r\nWe use state-of-the-art adversarial training algorithm [TRADES](https://github.com/yaodongyu/TRADES) as our baseline. The `main_adv.py` contains training and evaluation functions in adversarial training setting on CIFAR10 dataset.\r\n\r\n#### Training scripts\r\n- Training and evaluation using the default parameters\r\n  \r\n  We provides our training scripts in directory `scripts/cifar10`. For a concrete example, we can use the command as below to train the default model (i.e., WRN34-10) on CIFAR10 dataset with PGD-10 attack ($\\epsilon$=0.031) to generate adversarial examples:\r\n  ```bash\r\n  $ bash scripts/cifar10/run_trades_sat.sh [TRIAL_NAME]\r\n  ```\r\n\r\n- Additional arguments \r\n  - `beta`: hyper-parameter $1/\\lambda$ in TRADES that controls the trade-off between natural accuracy and adversarial robustness\r\n  - `sat-es`: initial epochs of our approach\r\n  - `sat-alpha`: the momentum term $\\alpha$ of our approach\r\n\r\n#### Robust evaluation script\r\nEvaluate robust WRN-34-10 models on CIFAR10 under PGD-20 attack:\r\n```bash\r\n  $ python pgd_attack.py --model-dir \"/path/to/checkpoints\"\r\n```\r\nThis command evaluates 71-st to 100-th checkpoints in the specified path.\r\n\r\n#### Results\r\n\u003cp align=\"center\"\u003e\r\n    \u003cimg src=\"images/robust_acc.png\" width=\"450\"\\\u003e\r\n\u003c/p\u003e\r\n\u003cp align=\"center\"\u003e\r\nSelf-Adaptive Training mitigates the overfitting issue and consistently improves TRADES.\r\n\u003c/p\u003e\r\n\r\n#### Attack TRADES+SAT\r\nWe provide the checkpoint of our best performed model in [Google Drive](https://drive.google.com/file/d/1yX9kVmqxImGQAckSOtlAImarRnZ30nMt/view?usp=sharing) and compare its natural and robust accuracy with [TRADES](https://github.com/yaodongyu/TRADES) as below.\r\n|Attack (submitted by) \\ Method|TRADES|TRADES + SAT|\r\n|------------------------------|:----:|:----------:|\r\n|None (initial entry)          |84.92 |83.48       |\r\n|PGD-20 (initial entry)        |56.68 |58.03       |\r\n|[MultiTargeted-2000](https://arxiv.org/abs/1910.09338) (initial entry)|53.24 |53.46       |\r\n|[Auto-Attack+](https://github.com/fra31/auto-attack) (Francesco Croce)|53.08 |53.29       |\r\n\r\n\r\n## Reference\r\nFor technical details, please check [the conference version](https://arxiv.org/abs/2002.10319) or [the journal version](https://arxiv.org/abs/2101.08732) of our paper.\r\n\r\n```\r\n@inproceedings{huang2020self,\r\n  title={Self-Adaptive Training: beyond Empirical Risk Minimization},\r\n  author={Huang, Lang and Zhang, Chao and Zhang, Hongyang},\r\n  booktitle={Advances in Neural Information Processing Systems},\r\n  volume={33},\r\n  year={2020}\r\n}\r\n\r\n@article{huang2021self,\r\n  title={Self-Adaptive Training: Bridging the Supervised and Self-Supervised Learning},\r\n  author={Huang, Lang and Zhang, Chao and Zhang, Hongyang},\r\n  journal={arXiv preprint arXiv:2101.08732},\r\n  year={2021}\r\n}\r\n```\r\n\r\n## Contact\r\nIf you have any question about this code, feel free to open an issue or contact laynehuang@pku.edu.cn.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLayneH%2Fself-adaptive-training","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FLayneH%2Fself-adaptive-training","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLayneH%2Fself-adaptive-training/lists"}