{"id":13958415,"url":"https://github.com/ytongbai/ViTs-vs-CNNs","last_synced_at":"2025-07-20T23:31:26.760Z","repository":{"id":38364787,"uuid":"421146439","full_name":"ytongbai/ViTs-vs-CNNs","owner":"ytongbai","description":"[NeurIPS 2021]: Are Transformers More Robust Than CNNs? (Pytorch implementation \u0026 checkpoints)","archived":false,"fork":false,"pushed_at":"2021-12-09T18:56:46.000Z","size":2051,"stargazers_count":176,"open_issues_count":1,"forks_count":10,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-11-18T20:03:34.142Z","etag":null,"topics":["robustness","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ytongbai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-25T18:48:12.000Z","updated_at":"2024-11-03T12:21:40.000Z","dependencies_parsed_at":"2022-07-11T04:00:25.357Z","dependency_job_id":null,"html_url":"https://github.com/ytongbai/ViTs-vs-CNNs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ytongbai%2FViTs-vs-CNNs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ytongbai%2FViTs-vs-CNNs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ytongbai%2FViTs-vs-CNNs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ytongbai%2FViTs-vs-CNNs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ytongbai","download_url":"https://codeload.github.com/ytongbai/ViTs-vs-CNNs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226845020,"owners_count":17691143,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["robustness","transformer"],"created_at":"2024-08-08T13:01:33.172Z","updated_at":"2024-11-28T01:32:01.250Z","avatar_url":"https://github.com/ytongbai.png","language":"Python","funding_links":[],"categories":["对象检测、分割"],"sub_categories":["网络服务_其他"],"readme":"# Are Transformers More Robust Than CNNs?\n\nPytorch implementation for NeurIPS 2021 Paper: [Are Transformers More Robust Than CNNs?](https://arxiv.org/pdf/2111.05464.pdf)\n\nOur implementation is based on [DeiT](https://github.com/facebookresearch/deit).\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"resources/teaser.png\"/\u003e\n\u003c/div\u003e\n\n\n## Introduction\nTransformer emerges as a powerful tool for visual recognition. In addition to demonstrating competitive performance on a broad range of visual benchmarks, recent works also argue that Transformers are much more robust than Convolutions Neural Networks (CNNs). Nonetheless, surprisingly, we find these conclusions are drawn from unfair experimental settings, where Transformers and CNNs are compared at different scales and are applied with distinct training frameworks. In this paper, we aim to provide the first fair \u0026 in-depth comparisons between Transformers and CNNs, focusing on robustness evaluations.\n\nWith our unified training setup, we first challenge the previous belief that Transformers outshine CNNs when measuring adversarial robustness. More surprisingly, we find CNNs can easily be as robust as Transformers on defending against adversarial attacks, if they properly adopt Transformers' training recipes. While regarding generalization on out-of-distribution samples, we show pre-training on (external) large-scale datasets is not a fundamental request for enabling Transformers to achieve better performance than CNNs. Moreover, our ablations suggest such stronger generalization is largely benefited by the Transformer's self-attention-like architectures per se, rather than by other training setups. We hope this work can help the community better understand and benchmark the robustness of Transformers and CNNs. \n\n## Pretrained models\n\nWe provide both pretrained vanilla models and adversarially trained models.\n\n### Vanilla Training\n\n#### Main Results\n\n|             |                                          Pretrained Model                                           | ImageNet | ImageNet-A | ImageNet-C | Stylized-ImageNet |\n| ----------- | :-------------------------------------------------------------------------------------------------: | :------: | :--------: | :--------: | :---------------: |\n| Res50-Ori   | [download link](https://drive.google.com/file/d/1iNEvIPKYgi1ivfL4v4dKg8j1QN5bgVpn/view?usp=sharing) |   76.9   |    3.2     |    57.9    |        8.3        |\n| Res50-Align | [download link](https://drive.google.com/file/d/1SE8u1jctOM5dsbhmrcYW9Mwo3yWfNAaw/view?usp=sharing) |   76.3   |    4.5     |    55.6    |        8.2        |\n| Res50-Best  | [download link](https://drive.google.com/file/d/12g6Gwn-KXwLBrscjvWgjrK_kXM0twgiS/view?usp=sharing) |   75.7   |    6.3     |    52.3    |       10.8        |\n| DeiT-Small  | [download link](https://drive.google.com/file/d/1VorHupVJxnBOMS79gYbIhcOaL00h_KTr/view?usp=sharing) |   76.8   |    12.2    |    48.0    |       13.0        |\n\n#### Model Size\n\nResNets:\n\n- ResNets fully aligned (with DeiT's training recipe) model, denoted as `res*`:\n\n|          | Model Size |                                          Pretrained Model                                           | ImageNet | ImageNet-A | ImageNet-C | Stylized-ImageNet |\n| -------- | :--------: | :-------------------------------------------------------------------------------------------------: | :------: | :--------: | :--------: | :---------------: |\n| Res18\\*  |   11.69M   | [download link](https://drive.google.com/file/d/1Q5gj330KoCkNShr_y9mkvFZ5sUcHtfSn/view?usp=sharing) |  67.83   |    1.92    |   64.14    |       7.92        |\n| Res50\\*  |   25.56M   | [download link](https://drive.google.com/file/d/1SE8u1jctOM5dsbhmrcYW9Mwo3yWfNAaw/view?usp=sharing) |  76.28   |    4.53    |   55.62    |       8.17        |\n| Res101\\* |   44.55M   | [download link](https://drive.google.com/file/d/1dk640r6Y504Swhs2lUi-NiPoeNITe6ix/view?usp=sharing) |  77.97   |    8.84    |   49.19    |       11.60       |\n\n- ResNets best model (for Out-of-Distribution (OOD) generalization), denoted as `res-best`:\n\n|             | Model Size |                                          Pretrained Model                                           | ImageNet | ImageNet-A | ImageNet-C | Stylized-ImageNet |\n| ----------- | :--------: | :-------------------------------------------------------------------------------------------------: | :------: | :--------: | :--------: | :---------------: |\n| Res18-best  |   11.69M   | [download link](https://drive.google.com/file/d/16mtiO-04UaIb19BFFKptlThaE7Surac4/view?usp=sharing) |  66.81   |    2.03    |   62.65    |       9.45        |\n| Res50-best  |   25.56M   | [download link](https://drive.google.com/file/d/12g6Gwn-KXwLBrscjvWgjrK_kXM0twgiS/view?usp=sharing) |  75.74   |    6.32    |   52.25    |       10.77       |\n| Res101-best |   44.55M   | [download link](https://drive.google.com/file/d/13HbBPkFVijP8VClBQBCAzdKcGc6o7WZy/view?usp=sharing) |  77.83   |   11.49    |   47.35    |       13.28       |\n\nDeiTs:\n\n|            | Model Size |                                          Pretrained Model                                           | ImageNet | ImageNet-A | ImageNet-C | Stylized-ImageNet |\n| ---------- | :--------: | :-------------------------------------------------------------------------------------------------: | :------: | :--------: | :--------: | :---------------: |\n| DeiT-Mini  |   9.98M    | [download link](https://drive.google.com/file/d/1FzuZP_eH2Vrb0hohIH-NUnSTuI6MG-Eg/view?usp=sharing) |  72.89   |    8.19    |   54.68    |       9.88        |\n| DeiT-Small |   22.05M   | [download link](https://drive.google.com/file/d/1VorHupVJxnBOMS79gYbIhcOaL00h_KTr/view?usp=sharing) |  76.82   |   12.21    |   47.99    |       12.98       |\n\n#### Model Distillation\n\n|         |  Architecture   |                                          Pretrained Model                                           | ImageNet | ImageNet-A | ImageNet-C | Stylized-ImageNet |\n| ------- | :-------------: | :-------------------------------------------------------------------------------------------------: | :------: | :--------: | :--------: | :---------------: |\n| Teacher |   DeiT-Small    | [download link](https://drive.google.com/file/d/1iNEvIPKYgi1ivfL4v4dKg8j1QN5bgVpn/view?usp=sharing) |   76.8 | 12.2 |  48.0 | 13.0        |\n| Student | Res50\\*-Distill | [download link](https://drive.google.com/file/d/1MK0TQXoEAFfrhEC1Edm7FrcnemW8Zp7Y/view?usp=sharing) |   76.7 | 5.2 | 54.2 | 9.8       |\n| Teacher |     Res50\\*     | [download link](https://drive.google.com/file/d/1SE8u1jctOM5dsbhmrcYW9Mwo3yWfNAaw/view?usp=sharing) |   76.3 | 4.5 | 55.6 | 8.2        |\n| Student | DeiT-S-Distill  | [download link](https://drive.google.com/file/d/1IrOowURrFbdZGe_FK87_6UvZWjz2Y-9n/view?usp=sharing) |   76.2 | 10.9 | 49.3 | 11.9       |\n\n### Adversarial Training\n\n|            |                                          Pretrained Model                                           | Clean Acc | PGD-100 | Auto Attack |\n| ---------- | :-------------------------------------------------------------------------------------------------: | :-------: | :-----: | :---------: |\n| Res50-ReLU | [download link](https://drive.google.com/file/d/1q8VxQuMWGVpFeU0OZay989dmDjrPrf3d/view?usp=sharing) |   66.77   |  32.26  |    26.41    |\n| Res50-GELU | [download link](https://drive.google.com/file/d/1IPExDTAAuxIhUSYrmlweTQaKNscTw24-/view?usp=sharing) |   67.38   |  40.27  |    35.51    |\n| DeiT-Small | [download link](https://drive.google.com/file/d/1U5XmAUQkSlw5Q1ZhsVriBEOQEk-bPfFU/view?usp=sharing) |   66.50   |  40.32  |    35.50    |\n\n## Vanilla Training\n\n### Data preparation\n\nDownload and extract ImageNet train and val images from http://image-net.org/.\nThe directory structure is the [standard layout](https://pytorch.org/docs/stable/torchvision/datasets.html#imagefolder) for the torchvision, and the training and validation data is expected to be in the `train` folder and `val` folder respectively:\n\n```\n/path/to/imagenet/\n  train/\n    class1/\n      img1.jpeg\n    class2/\n      img2.jpeg\n  val/\n    class1/\n      img3.jpeg\n    class/2\n      img4.jpeg\n```\n\n### Environment\n\nInstall dependencies:\n\n```bash\npip3 install -r requirements.txt\n```\n\n### Training Scripts\n\nTo train a ResNet model on ImageNet run:\n\n```bash\nbash script/res.sh\n```\n\nTo train a DeiT model on ImageNet run:\n\n```bash\nbash script/deit.sh\n```\n\n## Generalization to Out-of-Distribution Sample\n\n### Data Preparation\n\nDownload and extract [ImageNet-A](https://github.com/hendrycks/natural-adv-examples), [ImageNet-C](https://github.com/hendrycks/robustness), [Stylized-ImageNet](https://github.com/rgeirhos/Stylized-ImageNet) val images:\n\n```\n/path/to/datasets/\n  val/\n    class1/\n      img1.jpeg\n    class/2\n      img2.jpeg\n```\n\n### Evaluation Scripts\n\nTo evaluate pre-trained models, run:\n\n```bash\nbash script/generation_to_ood.sh\n```\nIt is worth noting that for ImageNet-C evaluation, the error rate is calculated based on the Noise, Blur, Weather and Digital categories. \n\n## Adversarial Training\n\nTo perform adversarial training on ResNet run:\n\n```bash\nbash script/advres.sh\n```\n\nTo do adversarial training on DeiT run:\n\n```sh\nbash scripts/advdeit.sh\n```\n\n## Robustness to Adversarial Example\n\n\n### PGD Attack Evaluation\n\nTo evaluate the pre-trained models, run:\n\n```bash\nbash script/eval_advtraining.sh\n```\n\n\n### AutoAttack Evaluation\n\n`./autoattack` contains the [AutoAttack](https://github.com/fra31/auto-attack) public package, with a little modification to best support ImageNet evaluation.\n\n```sh\ncd autoattack/\nbash autoattack.sh\n```\n\n### Patch Attack Evaluation\n\nPlease refer to [PatchAttack](https://github.com/Chenglin-Yang/PatchAttack)\n\n## Citation\n\nIf you use our code, models or wish to refer to our results, please use the following BibTex entry:\n\n```bibtex\n@inproceedings{bai2021transformers,\n  title     = {Are Transformers More Robust Than CNNs?},\n  author    = {Bai, Yutong and Mei, Jieru and Yuille, Alan and Xie, Cihang},\n  booktitle = {Thirty-Fifth Conference on Neural Information Processing Systems},\n  year      = {2021},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fytongbai%2FViTs-vs-CNNs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fytongbai%2FViTs-vs-CNNs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fytongbai%2FViTs-vs-CNNs/lists"}