{"id":16269623,"url":"https://github.com/hiyouga/amp-regularizer","last_synced_at":"2025-09-11T01:30:43.657Z","repository":{"id":37551730,"uuid":"296041057","full_name":"hiyouga/AMP-Regularizer","owner":"hiyouga","description":"Code for our paper \"Regularizing Neural Networks via Adversarial Model Perturbation\", CVPR2021","archived":false,"fork":false,"pushed_at":"2021-05-25T08:41:06.000Z","size":100,"stargazers_count":35,"open_issues_count":1,"forks_count":9,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-29T13:32:21.644Z","etag":null,"topics":["computer-vision","cvpr","cvpr2021","deep-learning","deep-learning-algorithms","image-classification","machine-learning","neural-network","regularization","regularizing-neural-networks"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2010.04925","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hiyouga.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-16T13:28:23.000Z","updated_at":"2024-12-03T20:24:43.000Z","dependencies_parsed_at":"2022-08-27T13:54:20.859Z","dependency_job_id":null,"html_url":"https://github.com/hiyouga/AMP-Regularizer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiyouga%2FAMP-Regularizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiyouga%2FAMP-Regularizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiyouga%2FAMP-Regularizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hiyouga%2FAMP-Regularizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hiyouga","download_url":"https://codeload.github.com/hiyouga/AMP-Regularizer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232581235,"owners_count":18545420,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","cvpr","cvpr2021","deep-learning","deep-learning-algorithms","image-classification","machine-learning","neural-network","regularization","regularizing-neural-networks"],"created_at":"2024-10-10T18:08:42.823Z","updated_at":"2025-01-05T11:48:11.903Z","avatar_url":"https://github.com/hiyouga.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AMP-Regularizer\n\n![GitHub](https://img.shields.io/github/license/hiyouga/amp-regularizer)\n\n![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/regularizing-neural-networks-via-adversarial/image-classification-on-svhn)\n\nA PyTorch implementation for our CVPR 2021 paper \"Regularizing Neural Networks via Adversarial Model Perturbation\".\n\nYou can download the paper via: [[ArXiv]](https://arxiv.org/abs/2010.04925) [[PapersWithCode]](https://paperswithcode.com/paper/regularizing-neural-networks-via-adversarial).\n\nPoster and Slides can be found [here](https://github.com/hiyouga/AMP-Poster-Slides-LaTeX).\n\n## One-Sentence Summary\n\nAdversarial Model Perturbation (AMP) effectively improves the generalization performance of deep models by minimizing an \"AMP loss\" that can find flat local minima via applying a \"worst\" norm-bounded perturbation on the model parameter.\n\n![method](assets/method.jpg)\n\n## Abstract\n\nEffective regularization techniques are highly desired in deep learning for alleviating overfitting and improving generalization. This work proposes a new regularization scheme, based on the understanding that the flat local minima of the empirical risk cause the model to generalize better. This scheme is referred to as adversarial model perturbation (AMP), where instead of directly minimizing the empirical risk, an alternative \"AMP loss\" is minimized via SGD. Specifically, the AMP loss is obtained from the empirical risk by applying the \"worst\" norm-bounded perturbation on each point in the parameter space. Comparing with most existing regularization schemes, AMP has strong theoretical justifications, in that minimizing the AMP loss can be shown theoretically to favour flat local minima of the empirical risk. Extensive experiments on various modern deep architectures establish AMP as a new state of the art among regularization schemes. \n\n## Requirement\n\n- Python \u003e= 3.7\n- Torch \u003e= 1.6.0\n- TorchVision \u003e= 0.7.0\n- NumPy \u003e= 1.18.5\n- Pillow \u003e= 6.1.0\n\n## Preparation\n\n### Clone\n\n```bash\ngit clone https://github.com/hiyouga/AMP-Regularizer.git\n```\n\n### Create an anaconda environment:\n\n```bash\nconda create -n amp python=3.7\nconda activate amp\npip install -r requirements.txt\n```\n\n## Usage\n\nYou can use AMP as a regular optimizer with a `closure` function.\n\n```python\nfrom amp import AMP\n\noptimizer = AMP(model.parameters(), lr=0.1, epsilon=0.5, momentum=0.9)\n\nfor inputs, targets in dataset:\n    def closure():\n        optimizer.zero_grad()\n        outputs = model(inputs)\n        loss = loss_fn(outputs, targets)\n        loss.backward()\n        return outputs, loss\n\n    outputs, loss = optimizer.step(closure)\n```\n\nRun an example:\n\n```bash\npython main.py --dataset cifar100 --model preactresnet18 --epsilon 0.5 --inner_lr 1 --inner_iter 1\n```\n\n## Documentation\n\n\u003e ### **AMP(*params*, *lr*, *epsilon*, *inner_lr=1*, *inner_iter=1*, *base_optimizer=SGD*, *\\*\\*kwargs*)**\n\nImplements adversarial model perturbation.\n\n| Argument                             | Description                                                  |\n| ------------------------------------ | ------------------------------------------------------------ |\n| `params` (*iterable*)                | iterable of trainable parameters                             |\n| `lr` (*float*)                       | learning rate for outer optimization                         |\n| `epsilon` (*float*)                  | perturbation norm ball radius                                |\n| `inner_lr` (*float, optional*)       | learning rate for inner optimization (*default: 1*)          |\n| `inner_iter` (*int, optional*)       | iteration number for inner optimization (*default: 1*)       |\n| `base_optimizer` (*class, optional*) | basic optimizer class (*default: SGD*)                       |\n| `**kwargs`                           | keyword arguments passed to the `__init__` method of `base_optimizer` |\n\n\u003e ### **AMP.step(*closure*)**\n\nPerforms AMP optimization step. Noting that AMP requires a `closure` to perform a optimization step.\n\n| Argument                         | Description                                                  |\n| -------------------------------- | ------------------------------------------------------------ |\n| `closure` (*callable, required*) | a closure-based function that does a full forward-backward pass on the optimized model |\n\n## Results\n\nWe conduct experiment on CIFAR-100 using WideResNet-28-10 with `epsilon=0.5` and `inner_lr=1`. When we adopt `inner_iter=1`, AMP requires two gradient computation in each forward-backward pass. Thus it usually takes 1.8 times longer than ERM training.\n\n| Optimizer            | Test error  |\n| -------------------- | ----------- |\n| SGD + momentum       | 19.17±0.270 |\n| SGD + momentum (AMP) | **17.33±0.110** |\n\n## File Specifications\n\n- **models**: Description for several model architectures.\n- **data_utils.py**: Used functions for data preprocessing.\n- **main.py**: Scripts for training the models.\n- **amp.py**: Implementation for adversarial model perturbation.\n\n## Citation\n\nIf this work is helpful, please cite as:\n\n```bibtex\n@inproceedings{zheng2020regularizing,\n  title={Regularizing Neural Networks via Adversarial Model Perturbation},\n  author={Zheng, Yaowei and Zhang, Richong and Mao, Yongyi},\n  booktitle={{CVPR}},\n  year={2021}\n}\n```\n\n## Acknowledgements\n\nThis work is supported partly by the National Key Research and Development Program of China, by the National Natural Science Foundation of China, by the Beijing Advanced Innovation Center for Big Data and Brain Computing (BDBC), by the Fundamental Research Funds for the Central Universities, by the Beijing S\u0026T Committee and by the State Key Laboratory of Software Development Environment. The authors specially thank Linfang Hou for helpful discussions.\n\n## Contact\n\nhiyouga [AT] buaa [DOT] edu [DOT] cn\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhiyouga%2Famp-regularizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhiyouga%2Famp-regularizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhiyouga%2Famp-regularizer/lists"}