{"id":13738549,"url":"https://github.com/hendrycks/robustness","last_synced_at":"2025-04-08T00:38:23.342Z","repository":{"id":37841228,"uuid":"128312370","full_name":"hendrycks/robustness","owner":"hendrycks","description":"Corruption and Perturbation Robustness (ICLR 2019)","archived":false,"fork":false,"pushed_at":"2022-08-24T14:27:43.000Z","size":76334,"stargazers_count":1060,"open_issues_count":13,"forks_count":149,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-31T23:35:49.469Z","etag":null,"topics":["computer-vision","convolutional-neural-networks","deep-learning","domain-generalization","imagenet","machine-learning","ml-safety","pytorch","robustness"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hendrycks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-06T03:50:57.000Z","updated_at":"2025-03-29T09:02:47.000Z","dependencies_parsed_at":"2022-07-12T19:30:28.957Z","dependency_job_id":null,"html_url":"https://github.com/hendrycks/robustness","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hendrycks%2Frobustness","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hendrycks%2Frobustness/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hendrycks%2Frobustness/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hendrycks%2Frobustness/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hendrycks","download_url":"https://codeload.github.com/hendrycks/robustness/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247755560,"owners_count":20990620,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","convolutional-neural-networks","deep-learning","domain-generalization","imagenet","machine-learning","ml-safety","pytorch","robustness"],"created_at":"2024-08-03T03:02:26.394Z","updated_at":"2025-04-08T00:38:23.314Z","avatar_url":"https://github.com/hendrycks.png","language":"Python","funding_links":[],"categories":["Python","Datasets \u0026 Benchmarks"],"sub_categories":["Robustness / Distribution Shift"],"readme":"# Benchmarking Neural Network Robustness to Common Corruptions and Perturbations\n\nThis repository contains the datasets and some code for the paper [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://arxiv.org/abs/1903.12261) (ICLR 2019) by Dan Hendrycks and Thomas Dietterich.\n\nRequires Python 3+ and PyTorch 0.3+. For evaluation, please download the data from the links below.\n\n## ImageNet-C\n\n\u003cimg align=\"center\" src=\"assets/imagenet-c.png\" width=\"750\"\u003e\n\n[Download ImageNet-C here.](https://zenodo.org/record/2235448) [(Mirror.)](https://drive.google.com/drive/folders/1HDVw6CmX3HiG0ODFtI75iIfBDxSiSz2K?usp=sharing)\n\n[Download Tiny ImageNet-C here.](https://zenodo.org/record/2536630) [(Mirror.)](https://berkeley.box.com/s/6zt1qzwm34hgdzcvi45svsb10zspop8a)\n\nTiny ImageNet-C has 200 classes with images of size 64x64, while ImageNet-C has all 1000 classes where each image is the standard size. For even quicker experimentation, there is [CIFAR-10-C](https://zenodo.org/record/2535967) and [CIFAR-100-C](https://zenodo.org/record/3555552). Evaluation using the JPEGs above is strongly prefered to computing the corruptions in memory, so that evaluation is deterministic and consistent.\n\n## ImageNet-C Leaderboard\n\nImageNet-C Robustness with a ResNet-50 Backbone trained on ImageNet-1K and evaluated on 224x224x3 images.\n\n|                Method               |                              Reference                             | Standalone? |   mCE   |    Clean Error |\n|-------------------------------|------------------------------------------------------------------------|:-------:|:-------:| :-------:|\n| [DeepAugment+AugMix](https://github.com/hendrycks/imagenet-r) | [Hendrycks et al.](https://arxiv.org/abs/2006.16241) | No | 53.6%   |  24.2%\n| [Assemble-ResNet50](https://github.com/clovaai/assembled-cnn) | [Lee et al.](https://arxiv.org/abs/2001.06268) | No | 56.5%   |  17.90%\n| [ANT (3x3)](https://github.com/bethgelab/game-of-noise) | [Rusak and Schott et al.](https://arxiv.org/abs/2001.06057) | Yes | 63%   |  23.9%\n| BlurAfterConv | [Vasconcelos et al.](https://arxiv.org/abs/2011.10675)| Yes | 64.9%   |   21.2%\n| [AugMix](https://github.com/google-research/augmix) | [Hendrycks and Mu et al.](https://arxiv.org/pdf/1912.02781.pdf) (ICLR 2020) | Yes | 65.3%   |  22.47%\n| [Stylized ImageNet](https://github.com/rgeirhos/Stylized-ImageNet) | [Geirhos et al.](https://arxiv.org/pdf/1811.12231.pdf) (ICLR 2019) | Yes | 69.3%   |  25.41%\n| [Group-wise Inhibition+DataAug](https://github.com/LinusWu/TENET_Training) | [Liu et al.](https://openaccess.thecvf.com/content/ICCV2021/papers/Liu_Group-Wise_Inhibition_Based_Feature_Regularization_for_Robust_Classification_ICCV_2021_paper.pdf) (ICCV2021) | No | 69.6% | 22.4% \n| Patch Uniform | [Lopes et al.](https://arxiv.org/abs/1906.02611) | Yes | 74.3%   |  24.5%\n| [Group-wise Inhibition](https://github.com/LinusWu/TENET_Training) | [Liu et al.](https://openaccess.thecvf.com/content/ICCV2021/papers/Liu_Group-Wise_Inhibition_Based_Feature_Regularization_for_Robust_Classification_ICCV_2021_paper.pdf) (ICCV2021) | Yes | 75.3% | 23.1% \n| ResNet-50 Baseline                  |                                                               | N/A    | 76.7%   | 23.85%\n\n\"Standalone\" indicates whether the method is a combination of techniques or a standalone/single method. Combining methods and proposing standalone methods are both valuable but not necessarily commensurable.\n\nBe sure to check each paper for results on all 15 corruptions, as some of these techniques improve robustness on all corruptions, some methods help on some corruptions and hurt on others, and some are exceptional against noise corruptions. Other backbones can obtain better results. For example, a vanilla ResNeXt-101 has an mCE of 62.2%. Note Lopes et al. have a ResNet-50 backbone with an mCE of [80.6](https://openreview.net/pdf?id=S1gmrxHFvB#page=7\u0026zoom=100,144,580), so their improvement is larger than what is immediately suggested by the table.\n\nSubmit a pull request if you beat the state-of-the-art on ImageNet-C with a ResNet-50 backbone.\n\n### UPDATE: New Robustness Benchmarks\n\nFor other distribution shift benchmarks like ImageNet-C, consider datasets like [ImageNet-A](https://github.com/hendrycks/natural-adv-examples) or [ImageNet-R](https://github.com/hendrycks/imagenet-r/).\n\nImageNet-A contains real-world, unmodified natural images that cause model accuracy to substantially degrade. ImageNet-R(endition) has 30,000 renditions of ImageNet classes cocering art, cartoons, deviantart, graffiti, embroidery, graphics, origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, and video games.\n\n\n## Calculating the mCE\n\nThis spreadsheet shows how to calculate the [mean Corruption Error](https://docs.google.com/spreadsheets/d/1RwqofJPHhtdRPG-dDO7wPp-aGn-AmwmU5-rpvTzrMHw/edit?usp=sharing).\n\n## ImageNet-P\n\n\u003cimg align=\"center\" src=\"assets/translate.gif\" width=\"224\"\u003e \u003cimg align=\"center\" src=\"assets/tilt.gif\" width=\"224\"\u003e \u003cimg align=\"center\" src=\"assets/spatter.gif\" width=\"224\"\u003e\n\n\u003csub\u003e\u003csup\u003eImageNet-P sequences are MP4s not GIFs. The spatter perturbation sequence is a validation sequence.\u003c/sup\u003e\u003c/sub\u003e\n\n[Download Tiny ImageNet-P here.](https://zenodo.org/record/2536630) [(Mirror.)](https://berkeley.box.com/s/19m2ppji0xsqgtkrs95329bqftbvncx9)\n\n[Download ImageNet-P here.](https://zenodo.org/record/3565846) [(Mirror.)](https://drive.google.com/drive/folders/1vRrDaWA6-_GaUZqOmovWrr4W34aiSLu7?usp=sharing)\n\n[Download CIFAR-10-P here.](https://drive.google.com/drive/folders/1dY1_zeRyjMKdfmMbQ_uK8O1i0MVI9UbZ?usp=sharing)\n\n[Download CIFAR-100-P here.](https://drive.google.com/drive/folders/1Tpssw4Vn6X_4hmIW8KK5LzkKCZpAaae8?usp=sharing)\n\n## ImageNet-P Leaderboard\n\nImageNet-P Perturbation Robustness with a ResNet-50 Backbone\n\n|                Method               |                              Reference                             |   mFR   |   mT5D   |\n|-------------------------------------|--------------------------------------------------------------------|:-------:|:-------:|\n| [AugMix](https://github.com/google-research/augmix) | [Hendrycks and Mu et al.](https://arxiv.org/pdf/1912.02781.pdf) (ICLR 2020)              |   37.4%  |    |\n| Low Pass Filter Pooling (bin-5)     | [Zhang](https://arxiv.org/abs/1904.11486) (ICML 2019)              | 51.2%   | 71.9%   |\n| ResNet-50 Baseline                  |                                                                    | 58.0%   | 78.4%   |\n\nSubmit a pull request if you beat the state-of-the-art on ImageNet-P.\n\n## Citation\n\nIf you find this useful in your research, please consider citing:\n\n    @article{hendrycks2019robustness,\n      title={Benchmarking Neural Network Robustness to Common Corruptions and Perturbations},\n      author={Dan Hendrycks and Thomas Dietterich},\n      journal={Proceedings of the International Conference on Learning Representations},\n      year={2019}\n    }\n\nPart of the code was contributed by [Tom Brown](https://github.com/nottombrown).\n\n## Icons-50 (From an Older Draft)\n\n\u003cimg align=\"center\" src=\"assets/icons-50.png\" width=\"500\"\u003e\n\nDownload Icons-50 [here](https://berkeley.box.com/s/jcem6ik7rxr6594lg99kmrdo01ue6yjt) or [here.](https://drive.google.com/drive/folders/16_kaFo3uUoS-U8FTDm4nUh6Vo21UVnJX?usp=sharing)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhendrycks%2Frobustness","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhendrycks%2Frobustness","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhendrycks%2Frobustness/lists"}