https://github.com/Epiphqny/VisTR
[CVPR2021 Oral] End-to-End Video Instance Segmentation with Transformers
https://github.com/Epiphqny/VisTR
cvpr2021 instance-segmentation transformers video-instance-segmentation
Last synced: 5 months ago
JSON representation
[CVPR2021 Oral] End-to-End Video Instance Segmentation with Transformers
- Host: GitHub
- URL: https://github.com/Epiphqny/VisTR
- Owner: Epiphqny
- License: apache-2.0
- Created: 2021-03-11T11:18:41.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2021-07-15T09:30:52.000Z (almost 4 years ago)
- Last Synced: 2024-08-08T23:21:27.019Z (9 months ago)
- Topics: cvpr2021, instance-segmentation, transformers, video-instance-segmentation
- Language: Python
- Homepage: https://arxiv.org/abs/2011.14503
- Size: 109 KB
- Stars: 739
- Watchers: 11
- Forks: 95
- Open Issues: 25
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## VisTR: End-to-End Video Instance Segmentation with Transformers
This is the official implementation of the [VisTR paper](https://arxiv.org/abs/2011.14503):
### Installation
We provide instructions how to install dependencies via conda.
First, clone the repository locally:
```
git clone https://github.com/Epiphqny/vistr.git
```
Then, install PyTorch 1.6 and torchvision 0.7:
```
conda install pytorch==1.6.0 torchvision==0.7.0
```
Install pycocotools
```
conda install cython scipy
pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'
pip install git+https://github.com/youtubevos/cocoapi.git#"egg=pycocotools&subdirectory=PythonAPI"
```
Compile DCN module(requires GCC>=5.3, cuda>=10.0)
```
cd models/dcn
python setup.py build_ext --inplace
```
### Preparation
Download and extract 2019 version of YoutubeVIS train and val images with annotations from
[CodeLab](https://competitions.codalab.org/competitions/20128#participate-get_data) or [YoutubeVIS](https://youtube-vos.org/dataset/vis/).
We expect the directory structure to be the following:
```
VisTR
├── data
│ ├── train
│ ├── val
│ ├── annotations
│ │ ├── instances_train_sub.json
│ │ ├── instances_val_sub.json
├── models
...
```
Download the pretrained DETR models [Google Drive](https://drive.google.com/drive/folders/1DlN8uWHT2WaKruarGW2_XChhpZeI9MFG?usp=sharing) [BaiduYun](https://pan.baidu.com/s/12omUNDRjhAeGZ5olqQPpHA)(passcode:alge) on COCO and save it to the pretrained path.
### Training
Training of the model requires at least 32g memory GPU, we performed the experiment on 32g V100 card. （As the training resolution is limited by the GPU memory, if you have a larger memory GPU and want to perform the experiment, please contact with me, thanks very much)
To train baseline VisTR on a single node with 8 gpus for 18 epochs, run:
```
python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --backbone resnet101/50 --ytvos_path /path/to/ytvos --masks --pretrained_weights /path/to/pretrained_path
```
### Inference
```
python inference.py --masks --model_path /path/to/model_weights --save_path /path/to/results.json
```
### Models
We provide baseline VisTR models, and plan to include more in future. AP is computed on YouTubeVIS dataset by submitting the result json file to the [CodeLab](https://competitions.codalab.org/competitions/20128#results) system, and inference time is calculated by pure model inference time (without data-loading and post-processing).
name
backbone
FPS
mask AP
model
result json zip
detailed AP
0
VisTR
R50
69.9
36.2
vistr_r50.pth
vistr_r50.zip
1
VisTR
R101
57.7
40.1
vistr_r101.pth
vistr_r101.zip
### License
VisTR is released under the Apache 2.0 license. Please see the [LICENSE](LICENSE) file for more information.
### Acknowledgement
We would like to thank the [DETR](https://github.com/facebookresearch/detr) open-source project for its awesome work, part of the code are modified from its project.
### Citation
Please consider citing our paper in your publications if the project helps your research. BibTeX reference is as follow.
```
@inproceedings{wang2020end,
title={End-to-End Video Instance Segmentation with Transformers},
author={Wang, Yuqing and Xu, Zhaoliang and Wang, Xinlong and Shen, Chunhua and Cheng, Baoshan and Shen, Hao and Xia, Huaxia},
booktitle = {Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)},
year={2021}
}
```