{"id":13737818,"url":"https://github.com/ByungKwanLee/Super-Fast-Adversarial-Training","last_synced_at":"2025-05-08T15:31:33.435Z","repository":{"id":41507458,"uuid":"459513450","full_name":"ByungKwanLee/Super-Fast-Adversarial-Training","owner":"ByungKwanLee","description":"Official PyTorch Implementation Code for Developing Super Fast Adversarial Training with Distributed Data Parallel, Channel Last Memory Format, Mixed Precision Training + Adversarial Attack, Faster Adversarial Training, and Fast Forward Computer Vision (FFCV).","archived":false,"fork":false,"pushed_at":"2023-03-13T12:19:56.000Z","size":67,"stargazers_count":31,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-08-04T03:11:12.603Z","etag":null,"topics":["deep-learning","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ByungKwanLee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-02-15T09:36:03.000Z","updated_at":"2024-04-26T15:44:23.000Z","dependencies_parsed_at":"2024-01-07T17:10:55.454Z","dependency_job_id":"bc585c97-e644-4836-956b-db361c4462f0","html_url":"https://github.com/ByungKwanLee/Super-Fast-Adversarial-Training","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByungKwanLee%2FSuper-Fast-Adversarial-Training","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByungKwanLee%2FSuper-Fast-Adversarial-Training/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByungKwanLee%2FSuper-Fast-Adversarial-Training/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByungKwanLee%2FSuper-Fast-Adversarial-Training/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ByungKwanLee","download_url":"https://codeload.github.com/ByungKwanLee/Super-Fast-Adversarial-Training/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224742399,"owners_count":17362229,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","pytorch"],"created_at":"2024-08-03T03:02:02.087Z","updated_at":"2024-11-15T06:31:03.029Z","avatar_url":"https://github.com/ByungKwanLee.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"![PyTorch](https://img.shields.io/badge/PyTorch-%23EE4C2C.svg?style=for-the-badge\u0026logo=PyTorch\u0026logoColor=white)\r\n![Git](https://img.shields.io/badge/git-%23F05033.svg?style=for-the-badge\u0026logo=git\u0026logoColor=white)\r\n# Super-Fast-Adversarial-Training\r\n[![Generic badge](https://img.shields.io/badge/Library-Pytorch-green.svg)](https://pytorch.org/)\r\n[![Generic badge](https://img.shields.io/badge/Version-alpha-red.svg)](https://shields.io/)\r\n[![Generic badge](https://img.shields.io/badge/Under-Develop-blue.svg)](https://shields.io/)\r\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/ByungKwanLee/Super-Fast-Adversarial-Training/blob/master/LICENSE)\r\n---\r\n\r\nThis is an Official PyTorch Implementation code for developing super fast adversarial training.\r\nThis code is combined with below state-of-the-art technologies for\r\naccelerating adversarial attacks and defenses with Deep Neural Networks\r\non Volta GPU architecture.\r\n\r\n- [x] Distributed Data Parallel [[link]](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html)\r\n- [x] Channel Last Memory Format [[link]](https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html#:~:text=Channels%20last%20memory%20format%20is,pixel%2Dper%2Dpixel)\r\n- [x] Mixed Precision Training [[link]](https://openreview.net/forum?id=r1gs9JgRZ)\r\n- [x] Mixed Precision + Adversarial Attack (based on torchattacks [[link]](https://github.com/Harry24k/adversarial-attacks-pytorch))\r\n- [x] Faster Adversarial Training for Large Dataset [[link]](https://openreview.net/forum?id=BJx040EFvH)\r\n- [x] Fast Forward Computer Vision (FFCV) [[link]](https://github.com/libffcv/ffcv)\r\n\r\n---\r\n\r\n\r\n\r\n## Citation\r\nIf you find this work helpful, please cite it as:\r\n\r\n```\r\n@software{SuperFastAT_ByungKwanLee_2022,\r\n  author = {Byung-Kwan Lee},\r\n  title = {Super Fast Adversarial Training with Distributed Data Parallel and Mixed Precision},\r\n  howpublished = {\\url{https://github.com/ByungKwanLee/Super-Fast-Adversarial-Training}},\r\n  year = {2022}\r\n}\r\n```\r\n---\r\n\r\n## Library for Fast Adversarial Attacks\r\nThis library is developed based on the well-known package of torchattacks [[link]](https://github.com/Harry24k/adversarial-attacks-pytorch) due to its simple scalability.\r\n\r\n**Current Available Attacks Below**\r\n\r\n* Fast Gradient Sign Method ([FGSM](https://arxiv.org/abs/1412.6572))\r\n* Basic Iterative Method ([BIM](https://arxiv.org/abs/1611.01236))\r\n* Projected Gradient Descent ([PGD](https://arxiv.org/abs/1706.06083))\r\n* Momentum Iterative Method ([MIM](https://arxiv.org/abs/1710.06081))\r\n* Carlini \u0026 Wagner ([CW](https://arxiv.org/abs/1608.04644))\r\n* Fast Adaptive Boundary ([FAB](https://arxiv.org/abs/1907.02044))\r\n* Auto-PGD ([AP](https://arxiv.org/abs/2003.01690))\r\n* Difference of Logits Ratio ([DLR](https://arxiv.org/abs/2003.01690))\r\n* Auto-Attack ([AA](https://arxiv.org/abs/2003.01690))\r\n\r\n---\r\n## Environment Setting\r\n\r\n### Please check below settings to successfully run this code. If not, follow step by step during filling the checklist in.\r\n\r\n- [ ] To utilize FFCV [[link]](https://github.com/libffcv/ffcv), you should install it on conda virtual environment.\r\nI use python version 3.8, pytorch 1.7.1, torchvision 0.8.2, and cuda 10.1. For more different version, you can refer to PyTorch official site [[link]](https://pytorch.org/get-started/previous-versions/). \r\n\r\n\u003e conda create -y -n ffcv python=3.8 cupy pkg-config compilers libjpeg-turbo opencv pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.1 numba -c pytorch -c conda-forge\r\n\r\n- [ ] Activate the created environment by conda\r\n\r\n\u003e conda activate ffcv\r\n\r\n- [ ] And, it would be better to install cudnn to more accelerate GPU. (Optional)\r\n\r\n\u003e conda install cudnn -c conda-forge\r\n\r\n- [ ] To install FFCV, you should download it in pip and install torchattacks [[link]](https://github.com/Harry24k/adversarial-attacks-pytorch) to run adversarial attack.\r\n\r\n\u003e pip install ffcv torchattacks==3.1.0\r\n\r\n- [ ] To guarantee the execution of this code, please additionally install library in requirements.txt (matplotlib, tqdm)\r\n\r\n\u003e pip install -r requirements.txt\r\n\r\n---\r\n\r\n## Available Datasets\r\n* [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html)\r\n* [CIFAR-100](https://www.cs.toronto.edu/~kriz/cifar.html)\r\n* [SVHN](http://ufldl.stanford.edu/housenumbers/)\r\n* [Tiny-ImageNet](https://www.kaggle.com/c/tiny-imagenet/overview)\r\n* [ImageNet](https://www.image-net.org/)\r\n\r\n---\r\n\r\n## Available Baseline Models\r\n\r\n* [VGG](https://arxiv.org/abs/1409.1556) *(model/vgg.py)*\r\n* [ResNet](https://arxiv.org/abs/1512.03385) *(model/resnet.py)*\r\n* [WideResNet](https://arxiv.org/abs/1605.07146) *(model/wide.py)*\r\n* [DenseNet](https://arxiv.org/abs/1608.06993) *(model/dense.py)*\r\n---\r\n\r\n## How to run\r\n\r\n### After making completion of environment settings, then you can follow how to run below.\r\n\r\n---\r\n\r\n* First, run `fast_dataset_converter.py` to generate dataset with `.betson` extension, instead of using original dataset [[FFCV]](https://github.com/libffcv/ffcv).\r\n\r\n```python\r\n# Future import build\r\nfrom __future__ import print_function\r\n\r\n# Import built-in module\r\nimport os\r\nimport argparse\r\n\r\n# fetch args\r\nparser = argparse.ArgumentParser()\r\n\r\n# parameter\r\nparser.add_argument('--dataset', default='imagenet', type=str)\r\nparser.add_argument('--gpu', default='0', type=str)\r\nargs = parser.parse_args()\r\n\r\n# GPU configurations\r\nos.environ[\"CUDA_VISIBLE_DEVICES\"]=args.gpu\r\n\r\n# init fast dataloader\r\nfrom utils.fast_data_utils import save_data_for_beton\r\nsave_data_for_beton(dataset=args.dataset)\r\n```\r\n\r\n---\r\n* Second, run `fast_pretrain_standard.py`(Standard Training) or `fast_pretrain_adv.py` (Adversarial Training)\r\n\r\n```python\r\n# model parameter\r\nimport argparse\r\nparser = argparse.ArgumentParser()\r\nparser.add_argument('--dataset', default='imagenet', type=str)\r\nparser.add_argument('--network', default='resnet', type=str)\r\nparser.add_argument('--depth', default=50, type=int)\r\nparser.add_argument('--gpu', default='0,1,2,3,4', type=str)\r\n\r\n# learning parameter\r\nparser.add_argument('--learning_rate', default=0.1, type=float)\r\nparser.add_argument('--weight_decay', default=0.0002, type=float)\r\nparser.add_argument('--batch_size', default=512, type=float)\r\nparser.add_argument('--test_batch_size', default=128, type=float)\r\nparser.add_argument('--epoch', default=100, type=int)\r\n```\r\n\r\nor\r\n\r\n```python\r\n# model parameter\r\nimport argparse\r\nparser = argparse.ArgumentParser()\r\nparser.add_argument('--dataset', default='imagenet', type=str)\r\nparser.add_argument('--network', default='resnet', type=str)\r\nparser.add_argument('--depth', default=18, type=int)\r\nparser.add_argument('--gpu', default='0,1,2,3,4', type=str)\r\n\r\n# learning parameter\r\nparser.add_argument('--learning_rate', default=0.1, type=float)\r\nparser.add_argument('--weight_decay', default=0.0002, type=float)\r\nparser.add_argument('--batch_size', default=1024, type=float)\r\nparser.add_argument('--test_batch_size', default=512, type=float)\r\nparser.add_argument('--epoch', default=60, type=int)\r\n\r\n# attack parameter\r\nparser.add_argument('--attack', default='pgd', type=str)\r\nparser.add_argument('--eps', default=0.03, type=float)\r\nparser.add_argument('--steps', default=10, type=int)\r\n```\r\n---\r\n\r\n## To-do\r\n\r\nI have plans to make a variety of functions to be a standard framework for adversarial training. \r\n\r\n- [x] Many Compatible Adversarial Attacks\r\n- [ ] Many Compatible Adversarial Defenses\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FByungKwanLee%2FSuper-Fast-Adversarial-Training","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FByungKwanLee%2FSuper-Fast-Adversarial-Training","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FByungKwanLee%2FSuper-Fast-Adversarial-Training/lists"}