{"id":15637071,"url":"https://github.com/sayakpaul/robustness-vit","last_synced_at":"2025-10-15T04:33:24.602Z","repository":{"id":41040832,"uuid":"347002404","full_name":"sayakpaul/robustness-vit","owner":"sayakpaul","description":"Contains code for the paper \"Vision Transformers are Robust Learners\" (AAAI 2022).","archived":false,"fork":false,"pushed_at":"2022-12-03T04:54:01.000Z","size":4426,"stargazers_count":125,"open_issues_count":0,"forks_count":19,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-07-11T00:21:16.360Z","etag":null,"topics":["computer-vision","jax","pytorch","robustness","self-attention","tensorflow","transformers"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2105.07581","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sayakpaul.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-12T08:59:12.000Z","updated_at":"2025-06-04T09:58:24.000Z","dependencies_parsed_at":"2022-07-13T18:21:08.401Z","dependency_job_id":null,"html_url":"https://github.com/sayakpaul/robustness-vit","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sayakpaul/robustness-vit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Frobustness-vit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Frobustness-vit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Frobustness-vit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Frobustness-vit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sayakpaul","download_url":"https://codeload.github.com/sayakpaul/robustness-vit/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Frobustness-vit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273790554,"owners_count":25168674,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-05T02:00:09.113Z","response_time":402,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","jax","pytorch","robustness","self-attention","tensorflow","transformers"],"created_at":"2024-10-03T11:09:48.947Z","updated_at":"2025-10-15T04:33:19.583Z","avatar_url":"https://github.com/sayakpaul.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Vision Transformers are Robust Learners\n\nThis repository contains the code for the paper [Vision Transformers are Robust Learners](https://arxiv.org/abs/2105.07581) by\nSayak Paul\u003csup\u003e\\*\u003c/sup\u003e and Pin-Yu Chen\u003csup\u003e\\*\u003c/sup\u003e (AAAI 2022).\n\n\u003csup\u003e\\*\u003c/sup\u003eEqual contribution.\n\n**Update December 2022**: We won the [ML Research Spotlight from Kaggle](https://www.kaggle.com/discussions/general/370095).\n\n**Update July 2022**: The publication is now available as a part of the [AAAI-22 proceedings](https://ojs.aaai.org/index.php/AAAI/article/view/20103). It's also archived in the [IBM Research repository](https://research.ibm.com/publications/vision-transformers-are-robust-learners).\n\n### Abstract\n\nTransformers, composed of multiple self-attention layers, hold strong promises toward a generic learning primitive applicable to\ndifferent data modalities, including the recent breakthroughs in computer vision achieving state-of-the-art (SOTA) standard accuracy with better\nparameter efficiency. Since self-attention helps a model systematically align different components present inside the input data, it leaves grounds\nto investigate its performance under model robustness benchmarks. In this work, we study the robustness of the Vision Transformer (ViT) against common\ncorruptions and perturbations, distribution shifts, and natural adversarial examples. We use six different diverse ImageNet datasets concerning robust\nclassification to conduct a comprehensive performance comparison of ViT models and SOTA convolutional neural networks (CNNs), Big-Transfer. Through a \nseries of six systematically designed experiments, we then present analyses that provide both quantitative and qualitative indications to explain why\nViTs are indeed more robust learners. For example, with fewer parameters and similar dataset and pre-training combinations, ViT gives a top-1 accuracy\nof 28.10% on ImageNet-A which is 4.3x higher than a comparable variant of BiT. Our analyses on image masking, Fourier spectrum sensitivity, and spread\non discrete cosine energy spectrum reveal intriguing properties of ViT attributing to improved robustness. \n\n## Structure and Navigation\n\nAll the results related to the ImageNet datasets (ImageNet-C, ImageNet-P, ImageNet-R, ImageNet-A, ImageNet-O, and ImageNet-9)\ncan be derived from the notebooks contained in the [`imagenet_results/`](https://github.com/sayakpaul/robustness-vit/tree/master/imagenet_results)\ndirectory. Many notebooks inside that directory can be executed with [Google Colab](https://colab.research.google.com/). When that is not the\ncase, we provide execution instructions explicitly. This is followed for the rest of the directories present inside this repository. \n\n[`analysis/`](https://github.com/sayakpaul/robustness-vit/tree/master/analysis) directory contains code used to generate results for Section 4 in the paper. \n\n[`misc/`](https://github.com/sayakpaul/robustness-vit/tree/master/misc) directory contains code for various utilities. \n\nFor any questions, please open an issue and tag @sayakpaul.\n\n## About our dev environment\n\nWe use Python 3.8. As for the hardware setup (when not using Colab), we use [GCP Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench) with\n4 V100s, 60 GBs of RAM with 16 vCPUs (`n1-standard-16` [machine type](https://cloud.google.com/compute/docs/machine-types)).\n\n## Citation\n\n```\n@article{paul2021vision,\n  title={Vision Transformers are Robust Learners},\n  author={Sayak Paul and Pin-Yu Chen},\n  journal={Proceedings of the AAAI Conference on Artificial Intelligence},\n  year={2022}\n}\n```\n\n## Acknowledgements\n\nWe are thankful to the [Google Developers Experts program](https://developers.google.com/programs/experts/) (specifically\nSoonson Kwon and Karl Weinmeister) for providing Google Cloud Platform credits to support the experiments. We also\nthank Justin Gilmer (of Google), Guillermo Ortiz-Jimenez (of EPFL, Switzerland), and Dan Hendrycks (of UC Berkeley)\nfor fruitful discussions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsayakpaul%2Frobustness-vit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsayakpaul%2Frobustness-vit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsayakpaul%2Frobustness-vit/lists"}