{"id":13935669,"url":"https://github.com/YuqingWang1029/VisTR","last_synced_at":"2025-07-19T20:33:37.585Z","repository":{"id":41345955,"uuid":"346676718","full_name":"Epiphqny/VisTR","owner":"Epiphqny","description":"[CVPR2021 Oral] End-to-End Video Instance Segmentation with Transformers","archived":false,"fork":false,"pushed_at":"2021-07-15T09:30:52.000Z","size":112,"stargazers_count":739,"open_issues_count":25,"forks_count":95,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-08-08T23:21:27.019Z","etag":null,"topics":["cvpr2021","instance-segmentation","transformers","video-instance-segmentation"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2011.14503","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Epiphqny.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-11T11:18:41.000Z","updated_at":"2024-08-08T11:06:52.000Z","dependencies_parsed_at":"2022-08-10T01:54:46.512Z","dependency_job_id":null,"html_url":"https://github.com/Epiphqny/VisTR","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Epiphqny%2FVisTR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Epiphqny%2FVisTR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Epiphqny%2FVisTR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Epiphqny%2FVisTR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Epiphqny","download_url":"https://codeload.github.com/Epiphqny/VisTR/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226677217,"owners_count":17666018,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvpr2021","instance-segmentation","transformers","video-instance-segmentation"],"created_at":"2024-08-07T23:01:58.764Z","updated_at":"2025-07-19T20:33:32.165Z","avatar_url":"https://github.com/Epiphqny.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"## VisTR: End-to-End Video Instance Segmentation with Transformers\n\nThis is the official implementation of the [VisTR paper](https://arxiv.org/abs/2011.14503):\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/16319629/110786946-b99aa080-82a7-11eb-98e4-85478ca4eeac.png\" width=\"600\"\u003e\n\u003c/p\u003e\n\n\n### Installation\nWe provide instructions how to install dependencies via conda.\nFirst, clone the repository locally:\n```\ngit clone https://github.com/Epiphqny/vistr.git\n```\nThen, install PyTorch 1.6 and torchvision 0.7:\n```\nconda install pytorch==1.6.0 torchvision==0.7.0\n```\nInstall pycocotools\n```\nconda install cython scipy\npip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'\npip install git+https://github.com/youtubevos/cocoapi.git#\"egg=pycocotools\u0026subdirectory=PythonAPI\"\n```\nCompile DCN module(requires GCC\u003e=5.3, cuda\u003e=10.0)\n```\ncd models/dcn\npython setup.py build_ext --inplace\n```\n\n### Preparation\n\nDownload and extract 2019 version of YoutubeVIS  train and val images with annotations from\n[CodeLab](https://competitions.codalab.org/competitions/20128#participate-get_data) or [YoutubeVIS](https://youtube-vos.org/dataset/vis/).\nWe expect the directory structure to be the following:\n```\nVisTR\n├── data\n│   ├── train\n│   ├── val\n│   ├── annotations\n│   │   ├── instances_train_sub.json\n│   │   ├── instances_val_sub.json\n├── models\n...\n```\n\nDownload the pretrained DETR models [Google Drive](https://drive.google.com/drive/folders/1DlN8uWHT2WaKruarGW2_XChhpZeI9MFG?usp=sharing) [BaiduYun](https://pan.baidu.com/s/12omUNDRjhAeGZ5olqQPpHA)(passcode:alge) on COCO and save it to the pretrained path.\n\n\n### Training\n\nTraining of the model requires at least 32g memory GPU, we performed the experiment on 32g V100 card. （As the training resolution is limited by the GPU memory, if you have a larger memory GPU and want to perform the experiment, please contact with me, thanks very much)\n\nTo train baseline VisTR on a single node with 8 gpus for 18 epochs, run:\n```\npython -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --backbone resnet101/50 --ytvos_path /path/to/ytvos --masks --pretrained_weights /path/to/pretrained_path\n```\n\n### Inference\n\n```\npython inference.py --masks --model_path /path/to/model_weights --save_path /path/to/results.json\n```\n\n### Models\n\nWe provide baseline VisTR models, and plan to include more in future. AP is computed on YouTubeVIS dataset by submitting the result json file to the [CodeLab](https://competitions.codalab.org/competitions/20128#results) system, and inference time is calculated by pure model inference time (without data-loading and post-processing).\n\n   \u003ctable\u003e\n     \u003cthead\u003e\n       \u003ctr style=\"text-align: right;\"\u003e\n         \u003cth\u003e\u003c/th\u003e\n         \u003cth\u003ename\u003c/th\u003e\n         \u003cth\u003ebackbone\u003c/th\u003e\n         \u003cth\u003eFPS\u003c/th\u003e\n         \u003cth\u003emask AP\u003c/th\u003e\n         \u003cth\u003emodel\u003c/th\u003e\n         \u003cth\u003eresult json zip\u003c/th\u003e\n         \u003cth\u003edetailed AP \u003c/th\u003e\n       \u003c/tr\u003e\n     \u003c/thead\u003e\n     \u003ctbody\u003e\n       \u003ctr\u003e\n         \u003cth\u003e0\u003c/th\u003e\n         \u003ctd\u003eVisTR\u003c/td\u003e\n         \u003ctd\u003eR50\u003c/td\u003e\n         \u003ctd\u003e69.9\u003c/td\u003e\n         \u003ctd\u003e36.2\u003c/td\u003e\n         \u003ctd\u003e\u003ca href=\"https://drive.google.com/file/d/10lfe_QJSoZJzcJKxWoxt67QbJG35X55e/view?usp=sharing\"\u003evistr_r50.pth \u003c/a\u003e\u003c/td\u003e\n         \u003ctd\u003e\u003ca href=\"https://drive.google.com/file/d/1vnZvxFR94EQ5TsrWixe368WMnqJ2KHIv/view?usp=sharing\"\u003evistr_r50.zip\u003c/a\u003e\u003c/td\u003e\n       \u003ctd\u003e\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/16319629/115868905-b4686e00-a46f-11eb-9fe0-c9170026fca9.png\" width=\"100\"\u003e\n\n\n\u003c/p\u003e\u003c/td\u003e\n       \u003c/tr\u003e\n       \u003ctr\u003e\n         \u003cth\u003e1\u003c/th\u003e\n         \u003ctd\u003eVisTR\u003c/td\u003e\n         \u003ctd\u003eR101\u003c/td\u003e\n         \u003ctd\u003e57.7\u003c/td\u003e\n         \u003ctd\u003e40.1\u003c/td\u003e\n         \u003ctd\u003e\u003ca href=\"https://drive.google.com/file/d/1WTkrpbITPsjVQESaetgI-nPyjQybRc2M/view?usp=sharing\"\u003evistr_r101.pth \u003c/a\u003e\u003c/td\u003e\n         \u003ctd\u003e\u003ca href=\"https://drive.google.com/file/d/1bfZO3MNF9e0aO0W8vwnALsf_m84lCFfb/view?usp=sharing\"\u003evistr_r101.zip\u003c/a\u003e\u003c/td\u003e\n       \u003ctd\u003e\n          \u003cp align=\"center\"\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/16319629/115869052-e974c080-a46f-11eb-92e3-1778aad83c71.png\" width=\"100\"\u003e\n\u003c/p\u003e\n\u003c/td\u003e\n       \u003c/tr\u003e\n   \u003c/table\u003e\n\n\n### License\n\nVisTR is released under the Apache 2.0 license. Please see the [LICENSE](LICENSE) file for more information.\n\n### Acknowledgement\nWe would like to thank the [DETR](https://github.com/facebookresearch/detr) open-source project for its awesome work, part of the code are modified from its project.\n\n### Citation\n\nPlease consider citing our paper in your publications if the project helps your research. BibTeX reference is as follow.\n\n```\n@inproceedings{wang2020end,\n  title={End-to-End Video Instance Segmentation with Transformers},\n  author={Wang, Yuqing and Xu, Zhaoliang and Wang, Xinlong and Shen, Chunhua and Cheng, Baoshan and Shen, Hao and Xia, Huaxia},\n  booktitle =  {Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)},\n  year={2021}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYuqingWang1029%2FVisTR","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FYuqingWang1029%2FVisTR","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYuqingWang1029%2FVisTR/lists"}