https://github.com/vitae-transformer/vitae-transformer

The official repo for [NeurIPS'21] "ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias" and [IJCV'22] "ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond"
https://github.com/vitae-transformer/vitae-transformer

ade20k deep-learning imagenet imagenet-classification mscoco object-detection semantic-segmentation vision-transformer vitae-transformer

Last synced: about 1 year ago
JSON representation

The official repo for [NeurIPS'21] "ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias" and [IJCV'22] "ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond"

• Host: GitHub
• URL: https://github.com/vitae-transformer/vitae-transformer
• Owner: ViTAE-Transformer
• Created: 2022-04-09T07:36:31.000Z (about 4 years ago)
• Default Branch: main
• Last Pushed: 2023-04-05T02:02:47.000Z (over 3 years ago)
• Last Synced: 2025-04-02T08:36:26.148Z (over 1 year ago)
• Topics: ade20k, deep-learning, imagenet, imagenet-classification, mscoco, object-detection, semantic-segmentation, vision-transformer, vitae-transformer
• Language: Python
• Homepage:
• Size: 22.2 MB
• Stars: 268
• Watchers: 5
• Forks: 29
• Open Issues: 11
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

 




  Updates |

  Introduction |

  Statement |



## Current applications

> **Image Classification**: Please see ViTAE-Transformer for image classification;

> **Object Detection**: Please see ViTAE-Transformer for object detection;

> **Sementic Segmentation**: Please see ViTAE-Transformer for semantic segmentation;

> **Animal Pose Estimation**: Please see ViTAE-Transformer for animal pose estimation;

> **Matting**: Please see ViTAE-Transformer for matting;

> **Remote Sensing**: Please see ViTAE-Transformer for Remote Sensing;

## Updates

***09/04/2021***

- The pretrained models for ViTAE on [matting](https://github.com/ViTAE-Transformer/ViTAE-Transformer-Matting) and [remote sensing](https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing) are released! Please try and have fun!

***24/03/2021***

- The pretrained models for both ViTAE and ViTAEv2 are released. The code for downstream tasks are also provided for reference.

***07/12/2021***

- The code is released!

***19/10/2021***

- The paper is accepted by Neurips'2021! The code will be released soon!

  

***06/08/2021***

- The paper is post on arxiv! The code will be made public available once cleaned up.

## Introduction

This repository contains the code, models, test results for the paper ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias. It contains several reduction cells and normal cells to introduce scale-invariance and locality into vision transformers. In ViTAEv2, we explore the usage of window attentions without shift operations to obtain a better balance between memory footprint, speed, and performance. We also stack the proposed RC and NC in a multi-stage manner to faciliate the learning on other vision tasks including detection, segmentation, and pose.



Fig.1 - The details of RC and NC design in ViTAE.



Fig.2 - The multi-stage design of ViTAEv2.

## Statement

This project is for research purpose only. For any other questions please contact [yufei.xu at outlook.com](mailto:yufei.xu@outlook.com) [qmzhangzz at hotmail.com](mailto:qmzhangzz@hotmail.com) .

## Citing ViTAE and ViTAEv2

```

@article{xu2021vitae,

  title={Vitae: Vision transformer advanced by exploring intrinsic inductive bias},

  author={Xu, Yufei and Zhang, Qiming and Zhang, Jing and Tao, Dacheng},

  journal={Advances in Neural Information Processing Systems},

  volume={34},

  year={2021}

}

@article{zhang2022vitaev2,

  title={ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond},

  author={Zhang, Qiming and Xu, Yufei and Zhang, Jing and Tao, Dacheng},

  journal={arXiv preprint arXiv:2202.10108},

  year={2022}

}

```

## Other Links

> **Image Classification**: See [ViTAE for Image Classification](https://github.com/ViTAE-Transformer/ViTAE-Transformer/tree/main/Image-Classification)

> **Object Detection**: See [ViTAE for Object Detection](https://github.com/ViTAE-Transformer/ViTAE-Transformer/tree/main/Object-Detection).

> **Semantic Segmentation**: See [ViTAE for Semantic Segmentation](https://github.com/ViTAE-Transformer/ViTAE-Transformer/tree/main/Semantic-Segmentation).

> **Animal Pose Estimation**: See [ViTAE for Animal Pose Estimation](https://github.com/ViTAE-Transformer/ViTAE-Transformer/tree/main/Animal-Pose-Estimation).

> **Matting**: See [ViTAE for Matting](https://github.com/ViTAE-Transformer/ViTAE-Transformer-Matting).

> **Remote Sensing**: See [ViTAE for Remote Sensing](https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing).