{"id":18560836,"url":"https://github.com/harry24k/mair","last_synced_at":"2025-04-10T02:31:13.025Z","repository":{"id":205310207,"uuid":"701173637","full_name":"Harry24k/MAIR","owner":"Harry24k","description":"Fantastic Robustness Measures: The Secrets of Robust Generalization [NeurIPS 2023]","archived":false,"fork":false,"pushed_at":"2025-01-13T05:00:19.000Z","size":281,"stargazers_count":37,"open_issues_count":2,"forks_count":6,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-24T15:42:02.690Z","etag":null,"topics":["adversarial-attacks","adversarial-defenses","adversarial-robustness","adversarial-training"],"latest_commit_sha":null,"homepage":"https://openreview.net/forum?id=AGVBqJuL0T","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Harry24k.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-06T04:52:26.000Z","updated_at":"2025-03-24T03:55:21.000Z","dependencies_parsed_at":"2023-12-25T13:05:23.547Z","dependency_job_id":"2642b73e-17ff-476f-aacd-8e1f6a1c813c","html_url":"https://github.com/Harry24k/MAIR","commit_stats":null,"previous_names":["harry24k/mair"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Harry24k%2FMAIR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Harry24k%2FMAIR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Harry24k%2FMAIR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Harry24k%2FMAIR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Harry24k","download_url":"https://codeload.github.com/Harry24k/MAIR/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248144193,"owners_count":21054881,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-attacks","adversarial-defenses","adversarial-robustness","adversarial-training"],"created_at":"2024-11-06T22:04:49.909Z","updated_at":"2025-04-10T02:31:12.363Z","avatar_url":"https://github.com/Harry24k.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/logo_wide.png?raw=true\" width=\"467\" title=\"ART logo\"\u003e\n\u003c/p\u003e\n\n\u003ch2 align=\"center\"\u003eImprove and Evaluate Robustness\u003c/h2\u003e\n\n\u003cp\u003e\n  \u003ca href=\"https://github.com/Harry24k/MAIR/blob/main/LICENSE\"\u003e\u003cimg alt=\"MIT License\" src=\"https://img.shields.io/github/license/Harry24k/MAIR?\u0026color=brightgreen\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/Harry24k/MAIR/releases\"\u003e\u003cimg alt=\"Latest Release\" src=\"https://img.shields.io/github/release/Harry24k/MAIR.svg?\u0026color=blue\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://hits.seeyoufarm.com\"\u003e\u003cimg src=\"https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FHarry24k%2FMAIR\u0026count_bg=%23FFC107\u0026title_bg=%23555555\u0026icon=\u0026icon_color=%23E7E7E7\u0026title=hits\u0026edge_flat=false\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/psf/black\"\u003e\u003cimg alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\n\u003e \"Make your AI Robust.\"\n\n_MAIR_  is a PyTorch-based adversarial training framework. MAIR aims to (1) _provide easy implementation of adversarial training methods_ and (2) _make it easier to evaluate the adversarial robustness of deep learning models_.\n\nAdversarial training has become the de facto standard method for improving the robustness of models against adversarial examples. However, while writing our paper **\"Fantastic Robustness Measures: The Secrets of Robust Generalization\"** [[Paper](https://openreview.net/forum?id=AGVBqJuL0T), [Article](https://trustworthyai.co.kr/article/2024/fantastic-eng/)], we realized that there is no framework integrating adversarial training methods. Therefore, to promote reproducibility and transparency in deep learning, we integrated the algorithms, tools, and pre-trained models. \n\n_Citation:_ If you use this package, please cite the following BibTex ([GoogleScholar](https://scholar.google.com/scholar?hl=ko\u0026as_sdt=0%2C5\u0026q=Fantastic+Robustness+Measures+The+Secrets+of+Robust+Generalization\u0026btnG=)):\n```\n@inproceedings{\n    kim2023fantastic,\n    title={Fantastic Robustness Measures: The Secrets of Robust Generalization},\n    author={Hoki Kim and Jinseong Park and Yujin Choi and Jaewook Lee},\n    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},\n    year={2023},\n    url={https://openreview.net/forum?id=AGVBqJuL0T}\n}\n```\n\n_Benchmarks on adversarially trained models are available at [our notion](https://harry24k.notion.site/harry24k/958ba2d81d194c1fa86accf65c1f6b9e?v=e02792dc2e7e47c697ff6b4a2dfe1a54)._ \n\n\n\n## Installation and usage\n\n### Installation\n\n`pip install git+https://github.com/Harry24k/MAIR.git`\n\n\n### Usage\n\n```python\nimport mair\n```\n\n#### How to train a model?\n**Step1.** Load model as follows:\n\n```python\nmodel = ...\nrmodel = mair.RobModel(model, n_classes=10).cuda()\n```\n\n**Step2.** Set trainer as follows:\n\n```python\nfrom mair.defenses import AT\n# Set adversarial training method: [Strandard, AT, TRADES, MART].\ntrainer = AT(rmodel, eps=EPS, alpha=ALPHA, steps=STEPS)\n# Set recording information.\ntrainer.record_rob(train_loader, val_loader, eps=EPS, alpha=2/255, steps=10, std=0.1)\n# Set detail training methods.\ntrainer.setup(optimizer=\"SGD(lr=0.1, momentum=0.9)\",\n              scheduler=\"Step(milestones=[100, 150], gamma=0.1)\",\n              scheduler_type=\"Epoch\",\n              minimizer=None, # or \"AWP(rho=5e-3)\",\n              n_epochs=200\n             )\n```\n\n**Step3.** Fit model as follows:\n\n```python\ntrainer.fit(train_loader=train_loader,\n            n_epochs=200,\n            save_path='./models/', \n            save_best={\"Clean(Val)\":\"HBO\", \"PGD(Val)\":\"HB\"},\n            # 'save_best': model with high PGD are chosen, \n            # while in similar cases, model with high Clean are selected.\n            save_type=\"Epoch\", \n            save_overwrite=False, \n            record_type=\"Epoch\"\n           )\n```\n\n#### How to evaluate a model?\n**Step1.** Transform model as follows:\n\n```python\nmodel = ...\nrmodel = mair.RobModel(model, n_classes=10).cuda()\n```\n\n**Step2.** Evaluate model as follows:\n\n```python\nrmodel.eval_accuracy(test_loader)  # clean accuracy\nrmodel.eval_rob_accuracy_gn(test_loader)  # gaussian noise accuracy\nrmodel.eval_rob_accuracy_fgsm(test_loader, eps)  # FGSM accuracy\nrmodel.eval_rob_accuracy_pgd(test_loader, eps, alpha, steps)  # PGD accuracy\n```\n\n\n\n#### Please refer to [demo](https://github.com/Harry24k/MAIR/blob/main/demos/Train%20and%20Evaluation.ipynb) for details.\n\n\n\n## Adversarial Benchmarks \u0026 Pre-trained models\n\nHere is our (selected) benchmark on popular techniques in adversarial training frameworks.\n\nNote that all robustness (or robust accuracy) are measured on CIFAR-10 test dataset.\nThrough [our notion](https://harry24k.notion.site/harry24k/958ba2d81d194c1fa86accf65c1f6b9e?v=e02792dc2e7e47c697ff6b4a2dfe1a54), you can check more detailed benchmarks.\n\n### ResNet\n\n| Method | Architecture |  AWP | Extra Data | PGD10 | AutoAttack | **Remark** |\n| :----: | :----------: | :---: | :---------: | ----------: | --------: | :----------: |\n|   AT   |   ResNet18   |  :x: |        :x: | 52.73 | 48.67 |            |\n|  MART  |   ResNet18   |  :x: |        :x: | 54.73 | 47.51 |            |\n| TRADES |   ResNet18   |  :x: |        :x: | 53.47 | 49.45 |            |\n|   AT   |   ResNet18   | ✔️ |        :x: | 55.52 | 50.80 |            |\n|  MART  |   ResNet18   | ✔️ |        :x: | 57.64 | 50.03 |            |\n| TRADES |   ResNet18   | ✔️ |        :x: | 55.91 | **51.62** | :crown: |\n|   AT   |   ResNet18   | ✔️ |      ✔️ | 56.52 | 51.14 |            |\n|  MART  |   ResNet18   | ✔️ |      ✔️ | **57.93** | 50.28 |  |\n| TRADES |   ResNet18   | ✔️ |      ✔️ | 55.51 | 51.08 |            |\n\n### WRN28-10\n\n| Method | Architecture |  AWP | Extra Data |     PGD10 | AutoAttack | **Remark** |\n| :----: | :----------: | :---: | :---------: | ----------: | --------: | :----------: |\n|   AT   |   WRN28-10   |  :x: |        :x: | 56.00 | 52.05 |            |\n|  MART  |   WRN28-10   |  :x: |        :x: | 57.69 | 51.29 |            |\n| TRADES |   WRN28-10   |  :x: |        :x: |           56.81 | 52.81 |            |\n|   AT   |   WRN28-10   | ✔️ |        :x: |           58.70 | 54.3 |            |\n|  MART  |   WRN28-10   | ✔️ |        :x: |           60.39 | 54.46 |            |\n| TRADES |   WRN28-10   | ✔️ |        :x: |           59.50 | 55.85 |            |\n|   AT   |   WRN28-10   | ✔️ |      ✔️ | 62.65 | **58.14** | :crown: |\n|  MART  |   WRN28-10   | ✔️ |      ✔️ | **63.51** | 56.54 |  |\n| TRADES |   WRN28-10   | ✔️ |      ✔️ |           61.81 | 57.44 |            |\n\n### WRN34-10\n\n| Method | Architecture |  AWP | Extra Data | PGD10 | AutoAttack | **Remark** |\n| :----: | :----------: | :---: | :---------: | ----------: | --------: | :----------: |\n|   AT   |   WRN34-10   |  :x: |        :x: |           56.19 | 52.68 |            |\n|  MART  |   WRN34-10   |  :x: |        :x: |           57.56 | 51.89 |            |\n| TRADES |   WRN34-10   |  :x: |        :x: |           56.67 | 53.08 |            |\n|   AT   |   WRN34-10   | ✔️ |        :x: |           59.63 | 55.66 |            |\n|  MART  |   WRN34-10   | ✔️ |        :x: |           10.00 | 10.00 |            |\n| TRADES |   WRN34-10   | ✔️ |        :x: |           59.50 | 56.07 |            |\n|   AT   |   WRN34-10   | ✔️ |      ✔️ |           63.30 | **58.65** | :crown: |\n|  MART  |   WRN34-10   | ✔️ |      ✔️ |       **64.04** | 57.40 |  |\n| TRADES |   WRN34-10   | ✔️ |      ✔️ |           62.07 | 57.75 |            |\n\n\n\nBased on [our notion](https://harry24k.notion.site/harry24k/958ba2d81d194c1fa86accf65c1f6b9e?v=e02792dc2e7e47c697ff6b4a2dfe1a54), we built a hub module to support the direct use of our pretrained models.\n\n```python\nfrom mair.hub import load_pretrained\nrmodel = load_pretrained(\"CIFAR10_ResNet18_AT(eps=8, alpha=2, steps=10)\", flag='Best', save_dir=\"./\")\n```\n\nPlease refer to [demo](https://github.com/Harry24k/MAIR/blob/main/demos/Load%20Pretrained%20Model.ipynb) for details.\n\n\n\nOr you can use [Google-drive](https://drive.google.com/drive/folders/1JoMWfAqXuvROyBbPX1KxPTro7i8LCNLo).\n\nIn each folder, we upload four different files:\n\n* `log.txt`: training log during training.\n* `last.pth`: model at the end of epoch.\n* `init.pth`: model at the start of epoch.\n* `best.pth`: best model selected by the argment `save_best` in `trainer.fit`.\n\nTo load model, \n\n```python\nrmodel.load_dict('./models/.../best.pth')\n```\n\nWe are excited to share modes with the community, but we've run into a storage limitation on Google Drive. Any help would be greatly appreciated!\n\n\n\n## Contribution\n\nWe welcome all contribution to MAIR in many forms :smiley:.\nEspecially, we are looking for diverse adversarial training methods beyond AT, TRADES, and MART.\n\n\n\n\n## Future work\n- [ ] Merge measures.\n- [ ] Generalize attacks gathered from [torchattacks](https://github.com/Harry24k/adversarial-attacks-pytorch).\n- [ ] ...\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharry24k%2Fmair","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharry24k%2Fmair","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharry24k%2Fmair/lists"}