{"id":13444425,"url":"https://github.com/alohays/awesome-visual-representation-learning-with-transformers","last_synced_at":"2025-03-20T18:32:30.453Z","repository":{"id":41517826,"uuid":"305061496","full_name":"alohays/awesome-visual-representation-learning-with-transformers","owner":"alohays","description":"Awesome Transformers (self-attention) in Computer Vision","archived":false,"fork":false,"pushed_at":"2021-07-31T08:04:36.000Z","size":75,"stargazers_count":263,"open_issues_count":1,"forks_count":36,"subscribers_count":14,"default_branch":"main","last_synced_at":"2024-05-23T04:00:50.604Z","etag":null,"topics":["awesome-list","computer-vision","representation-learning","self-attention","transformers","vision-transformer"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alohays.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-18T09:02:08.000Z","updated_at":"2024-04-12T04:00:50.000Z","dependencies_parsed_at":"2022-09-01T13:51:14.260Z","dependency_job_id":null,"html_url":"https://github.com/alohays/awesome-visual-representation-learning-with-transformers","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alohays%2Fawesome-visual-representation-learning-with-transformers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alohays%2Fawesome-visual-representation-learning-with-transformers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alohays%2Fawesome-visual-representation-learning-with-transformers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alohays%2Fawesome-visual-representation-learning-with-transformers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alohays","download_url":"https://codeload.github.com/alohays/awesome-visual-representation-learning-with-transformers/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244670566,"owners_count":20491015,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["awesome-list","computer-vision","representation-learning","self-attention","transformers","vision-transformer"],"created_at":"2024-07-31T04:00:22.554Z","updated_at":"2025-03-20T18:32:30.172Z","avatar_url":"https://github.com/alohays.png","language":null,"funding_links":[],"categories":["Uncategorized","Table of Contents","Other Lists"],"sub_categories":["Uncategorized","TeX Lists"],"readme":"# Awesome Visual Representation Learning with Transformers [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\n\nAwesome Transformers (self-attention) in Computer Vision\n\n## About transformers\n- Attention Is All You Need, NeurIPS 2017\n  - Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin\n  - [[paper]](https://arxiv.org/abs/1706.03762) [[official code]](https://github.com/tensorflow/tensor2tensor) [[pytorch implementation]](https://github.com/jadore801120/attention-is-all-you-need-pytorch)\n- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL 2019\n  - Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova\n  - [[paper]](https://arxiv.org/abs/1810.04805) [[offficial code]](https://github.com/google-research/bert) [[huggingface/transformers]](https://github.com/huggingface/transformers)\n- Efficient Transformers: A Survey, arXiv 2020\n  - Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler\n  - [[paper]](https://arxiv.org/abs/2009.06732)\n- A Survey on Visual Transformer, arXiv 2020\n  - Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, Dacheng Tao\n  - [[paper]](https://arxiv.org/abs/2012.12556)\n- Transformers in Vision: A Survey, arXiv 2021\n  - Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, Mubarak Shah\n  - [[paper]](https://arxiv.org/abs/2101.01169)\n\n## Combining CNN with self-attention\n- Attention augmented convolutional networks, ICCV 2019, image classification\n  - Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, Quoc V. Le\n  - [[paper]](https://arxiv.org/abs/1904.09925) [[pytorch implementation]](https://github.com/leaderj1001/Attention-Augmented-Conv2d)\n- Self-Attention Generative Adversarial Networks, ICML 2019, generative model(GANs)\n  - Han Zhang, Ian Goodfellow, Dimitris Metaxas, Augustus Odena\n  - [[paper]](https://arxiv.org/abs/1805.08318) [[official code]](https://github.com/heykeetae/Self-Attention-GAN)\n- Videobert: A joint model for video and language representation learning, ICCV 2019, video processing\n  - Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, Cordelia Schmid\n  - [[paper]](https://arxiv.org/abs/1904.01766)\n- Visual Transformers: Token-based Image Representation and Processing for Computer Vision, arXiv 2020, image classification\n  - Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Masayoshi Tomizuka, Kurt Keutzer, Peter Vajda\n  - [[paper]](https://arxiv.org/abs/2006.03677)\n- Feature Pyramid Transformer, ECCV 2020, detection and segmentation\n  - Dong Zhang, Hanwang Zhang, Jinhui Tang, Meng Wang, Xiansheng Hua, Qianru Sun\n  - [[paper]](http://arxiv.org/abs/2007.09451) [[official code]](https://github.com/ZHANGDONG-NJUST/FPT)\n- Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers, arXiv 2020, depth estimation\n  - Zhaoshuo Li, Xingtong Liu, Francis X. Creighton, Russell H. Taylor, and Mathias Unberath\n  - [[paper]](http://arxiv.org/abs/2011.02910) [[official code]](https://github.com/mli0603/stereo-transformer)\n- End-to-end Lane Shape Prediction with Transformers, arXiv 2020, lane detection\n  - Ruijin Liu, Zejian Yuan, Tie Liu, Zhiliang Xiong\n  - [[paper]](http://arxiv.org/abs/2011.04233) [[official code]](https://github.com/liuruijin17/LSTR)\n- Taming Transformers for High-Resolution Image Synthesis, arXiv 2020, image synthesis\n  - Patrick Esser, Robin Rombach, Bjorn Ommer\n  - [[paper]](http://arxiv.org/abs/2012.09841)[[official code]](https://github.com/CompVis/taming-transformers)\n- TransPose: Towards Explainable Human Pose Estimation by Transformer, arXiv 2020, pose estimation\n  - Sen Yang, Zhibin Quan, Mu Nie, Wankou Yang\n  - [[paper]](https://arxiv.org/abs/2012.14214)\n- End-to-End Video Instance Segmentation with Transformers, arXiv 2020, video instance segmentation\n  - Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, Huaxia Xia\n  - [[paper]](https://arxiv.org/abs/2011.14503)\n- TransTrack: Multiple-Object Tracking with Transformer, arXiv 2020, MOT\n  - Peize Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkun Cao, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, Ping Luo\n  - [[paper]](https://arxiv.org/abs/2012.15460)[[official code]](https://github.com/PeizeSun/TransTrack)\n- TrackFormer: Multi-Object Tracking with Transformers, arXiv 2021, MOT\n  - Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, Christoph Feichtenhofer\n  - [[paper]](https://arxiv.org/abs/2101.02702)\n- Line Segment Detection Using Transformers without Edges, arXiv 2021, line segmentation\n  - Yifan Xu, Weijian Xu, David Cheung, Zhuowen Tu\n  - [[paper]](https://arxiv.org/abs/2101.01909)\n- Segmenting Transparent Object in the Wild with Transformer, arXiv 2021, transparent object segmentation\n  - Enze Xie, Wenjia Wang, Wenhai Wang, Peize Sun, Hang Xu, Ding Liang, Ping Luo\n  - [[paper]](https://arxiv.org/abs/2101.08461)[[official code]](https://github.com/xieenze/Trans2Seg)\n- Bottleneck Transformers for Visual Recognition, arXiv 2021, backbone design\n  - Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, Ashish Vaswani\n  - [[paper]](http://arxiv.org/abs/2101.11605)\n\n### DETR Family\n- End-to-end object detection with transformers, ECCV 2020, object detection\n  - Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko\n  - [[paper]](https://arxiv.org/abs/2005.12872) [[official code]](https://github.com/facebookresearch/detr) [[detectron2 implementation]](https://github.com/poodarchu/DETR.detectron2)\n- Deformable DETR: Deformable Transformers for End-to-End Object Detection, ICLR 2021, object detection\n  - Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai\n  - [[paper]](http://arxiv.org/abs/2010.04159) [[official code]](https://github.com/fundamentalvision/Deformable-DETR)\n- End-to-End Object Detection with Adaptive Clustering Transformer, arXiv 2020, object detection\n  - Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, Hao Dong\n  - [[paper]](http://arxiv.org/abs/2011.09315)\n- UP-DETR: Unsupervised Pre-training for Object Detection with Transformers, arXiv 2020, object detection\n  - Zhigang Dai, Bolun Cai, Yugeng Lin, Junying Chen\n  - [[paper]](http://arxiv.org/abs/2011.09094)\n- DETR for Pedestrian Detection, arXiv 2020, pedestrian detection\n  - Matthieu Lin, Chuming Li, Xingyuan Bu, Ming Sun, Chen Lin, Junjie Yan, Wanli Ouyang, Zhidong Deng\n  - [[paper]](http://arxiv.org/abs/2012.06785)\n\n## Stand-alone transformers for Computer Vision\n### Self-attention only in local neighborhood\n- Image Transformer, ICML 2018\n  - Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, Dustin Tran\n  - [[paper]](https://arxiv.org/abs/1802.05751) [[official code]](https://github.com/tensorflow/tensor2tensor)\n- Stand-alone self-attention in vision models, NeurIPS 2019\n  - Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, Jonathon Shlens\n  - [[paper]](https://arxiv.org/abs/1906.05909) [[official code(underconstruction)]](https://github.com/google-research/google-research/tree/master/standalone_self_attention_in_vision_models)\n- On the relationship between self-attention and convolutional layers, ICLR 2020\n  - Jean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi\n  - [[paper]](https://arxiv.org/abs/1911.03584) [[official code]](https://github.com/epfml/attention-cnn)\n- Exploring self-attention for image recognition, CVPR 2020\n  - Hengshuang Zhao, Jiaya Jia, Vladlen Koltun\n  - [[paper]](https://arxiv.org/abs/2004.13621) [[official code]](https://github.com/hszhao/SAN)\n### Scalable approximations to global self-attention\n- Generating long sequences with sparse transformers, arXiv 2019\n  - Rewon Child, Scott Gray, Alec Radford, Ilya Sutskever\n  - [[paper]](https://arxiv.org/abs/1904.10509) [[official code]](https://github.com/openai/sparse_attention)\n- Scaling autoregressive video models, ICLR 2019\n  - Dirk Weissenborn, Oscar Täckström, Jakob Uszkoreit\n  - [[paper]](https://arxiv.org/abs/1906.02634) \n- Axial attention in multidimensional transformers, arXiv 2019\n  - Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, Tim Salimans\n  - [[paper]](https://arxiv.org/abs/1912.12180) [[pytorch implementation]](https://github.com/lucidrains/axial-attention)\n- Axial-deeplab: Stand-alone axial-attention for panoptic segmentation, ECCV 2020\n  - Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, Liang-Chieh Chen\n  - [[paper]](https://arxiv.org/abs/2003.07853) [[pytorch implementation]](https://github.com/csrhddlam/axial-deeplab)\n- MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers, arXiv 2020\n  - Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, Liang-Chieh Chen\n  - [[paper]](http://arxiv.org/abs/2012.00759)\n### Global self-attention with image preprocessing\n- Generative pretraining from pixels, ICML 2020, iGPT\n  - Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, Ilya Sutskever\n  - [[paper]](https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf) [[official code]](https://github.com/openai/image-gpt)\n- **An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021, ViT**\n  - Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby\n  - [[paper]](https://arxiv.org/abs/2010.11929) [[pytorch implementation]](https://github.com/lucidrains/vit-pytorch)\n- Pre-Trained Image Processing Transformer, arXiv, IPT\n  - Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, Wen Gao\n  - [[paper]](http://arxiv.org/abs/2012.00364)\n- Training data-efficient image transformers \u0026 distillation through attention, arXiv 2020, DeiT\n  - Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Herve Jegou\n  - [[paper]](http://arxiv.org/abs/2012.12877)[[official code]](https://github.com/facebookresearch/deit)\n- Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers, arXiv 2020, SETR\n  - Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, Li Zhang\n  - [[paper]](http://arxiv.org/abs/2012.15840)[[official code]](https://fudan-zvg.github.io/SETR)\n- Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet, arXiv 2021, T2T-ViT\n  - Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, Shuicheng Yan\n  - [[paper]](http://arxiv.org/abs/2101.11986)[[official code]](https://github.com/yitu-opensource/T2T-ViT)\n- TransReID: Transformer-based Object Re-Identification, arXiv 2021\n  - Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, Wei Jiang\n  - [[paper]](http://arxiv.org/abs/2102.04378)\n- Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions\n  - Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao\n  - [[paper]](https://arxiv.org/abs/2102.12122)[[official code]](https://github.com/whai362/PVT)\n### Global self-attention on 3D point clouds\n- Point Transformer, arXiv 2020, points classification + part/semantic segmentation\n  - Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, Vladlen Koltun\n  - [[paper]](http://arxiv.org/abs/2011.00931)\n\n## Unified text-vision tasks\n### Focused on VQA\n- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks, NeurIPS 2019\n  - Jiasen Lu, Dhruv Batra, Devi Parikh, Stefan Lee\n  - [[paper]](https://arxiv.org/abs/1908.02265) [[official code]](https://github.com/facebookresearch/vilbert-multi-task)\n- LXMERT: Learning Cross-Modality Encoder Representations from Transformers, EMNLP 2019\n  - Hao Tan, Mohit Bansal\n  - [[paper]](https://arxiv.org/abs/1908.07490) [[official code]](https://github.com/airsplay/lxmert)\n- VisualBERT: A Simple and Performant Baseline for Vision and Language, arXiv 2019\n  - Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang\n  - [[paper]](https://arxiv.org/abs/1908.03557) [[official code]](https://github.com/uclanlp/visualbert)\n- VL-BERT: Pre-training of Generic Visual-Linguistic Representations, ICLR 2020\n  - Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai\n  - [[paper]](https://arxiv.org/abs/1908.08530) [[official code]](https://github.com/jackroos/VL-BERT)\n- UNITER: UNiversal Image-TExt Representation Learning, ECCV 2020\n  - Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, Jingjing Liu\n  - [[paper]](https://arxiv.org/abs/1909.11740) [[official code]](https://github.com/ChenRocks/UNITER)\n- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers, arXiv 2020\n  - Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, Jianlong Fu\n  - [[paper]](https://arxiv.org/abs/2004.00849)\n- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision, arXiv 2021\n  - Wonjae Kim, Bokyung Son, Ildoo Kim\n  - [[paper]](https://arxiv.org/abs/2102.03334)\n\n### Focused on Image Retrieval\n- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training, AAAI 2020\n  - Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, Ming Zhou\n  - [[paper]](https://arxiv.org/abs/1908.06066) [[official code]](https://github.com/microsoft/Unicoder)\n- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data, arXiv 2020\n  - Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, Arun Sacheti\n  - [[paper]](https://arxiv.org/abs/2001.07966)\n- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks, ECCV 2020\n  - Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao\n  - [[paper]](https://arxiv.org/abs/2004.06165) [[official code]](https://github.com/microsoft/Oscar)\n- Training Vision Transformers for Image Retrieval, arXiv 2021\n  - Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, Herve Jegou\n  - [[paper]](http://arxiv.org/abs/2102.05644)\n### Focused on OCR\n\n- LayoutLM: Pre-training of Text and Layout for Document Image Understanding\n  - Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou\n  - [[paper]](https://arxiv.org/abs/1912.13318) [[official code]](https://github.com/microsoft/unilm/tree/master/layoutlm)\n\n### Focused on Image Captioning\n\n- CPTR: Full Transformer Network for Image Captioning, arXiv 2021\n  - Wei Liu, Sihan Chen, Longteng Guo, Xinxin Zhu, Jing Liu\n  - [[paper]](http://arxiv.org/abs/2101.10804)\n\n\n### Multi-Task\n- 12-in-1: Multi-Task Vision and Language Representation Learning\n  - Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, Stefan Lee\n  - [[paper]](https://arxiv.org/abs/1912.02315) [[official code]](https://github.com/facebookresearch/vilbert-multi-task)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falohays%2Fawesome-visual-representation-learning-with-transformers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falohays%2Fawesome-visual-representation-learning-with-transformers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falohays%2Fawesome-visual-representation-learning-with-transformers/lists"}