{"id":19216983,"url":"https://github.com/hustvl/queryinst","last_synced_at":"2025-04-05T13:06:56.980Z","repository":{"id":41282190,"uuid":"364738981","full_name":"hustvl/QueryInst","owner":"hustvl","description":"[ICCV 2021] Instances as Queries ","archived":false,"fork":false,"pushed_at":"2023-10-20T16:51:39.000Z","size":7643,"stargazers_count":407,"open_issues_count":16,"forks_count":55,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-03-29T12:05:51.811Z","etag":null,"topics":["computer-vision","instance-segmentation","mmdetection","video-instance-segmentation"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2105.01928","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hustvl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-06T00:18:21.000Z","updated_at":"2025-03-14T09:49:48.000Z","dependencies_parsed_at":"2024-11-09T14:29:54.572Z","dependency_job_id":null,"html_url":"https://github.com/hustvl/QueryInst","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hustvl%2FQueryInst","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hustvl%2FQueryInst/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hustvl%2FQueryInst/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hustvl%2FQueryInst/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hustvl","download_url":"https://codeload.github.com/hustvl/QueryInst/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247339155,"owners_count":20923014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","instance-segmentation","mmdetection","video-instance-segmentation"],"created_at":"2024-11-09T14:19:43.340Z","updated_at":"2025-04-05T13:06:56.963Z","avatar_url":"https://github.com/hustvl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e   \n\n# Instances as Queries\n\u003c/div\u003e\n\n\u003c!-- [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/queryinst-parallelly-supervised-mask-query/instance-segmentation-on-coco-minival)](https://paperswithcode.com/sota/instance-segmentation-on-coco-minival?p=queryinst-parallelly-supervised-mask-query)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/queryinst-parallelly-supervised-mask-query/instance-segmentation-on-coco)](https://paperswithcode.com/sota/instance-segmentation-on-coco?p=queryinst-parallelly-supervised-mask-query)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/queryinst-parallelly-supervised-mask-query/object-detection-on-coco-minival)](https://paperswithcode.com/sota/object-detection-on-coco-minival?p=queryinst-parallelly-supervised-mask-query)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/queryinst-parallelly-supervised-mask-query/object-detection-on-coco)](https://paperswithcode.com/sota/object-detection-on-coco?p=queryinst-parallelly-supervised-mask-query) --\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"100%\" alt=\"QueryInst-VIS Demo\" src=\"https://user-images.githubusercontent.com/45201863/120617230-7d34a600-c48c-11eb-8a43-d61689a050be.gif\"\u003e\n\u003c/div\u003e\n\n\n* **[News]**\n  * **`Apr, 2022`:** If you like `QueryInst` for instance segmentation, you might also like `TeViT` (CVPR 2022, oral, [paper](https://arxiv.org/abs/2204.08412) / [code \u0026 models](https://github.com/hustvl/TeViT)) for high-performance video instance segmentation!.\n  * **`Oct, 2021`:** `QueryInst (ICCV 2021)` is now officially included by `mmdetection` library, with new checkpoints, corresponding logs, and augmented training settings. We suggest you to use the newest `QueryInst` implementation in `mmdetection`, meanwhile this repo will be maintained too. Issues are welcomed if you have problems using `QueryInst` to reproduce the COCO AP reported in our paper.\n\n\n* **TL;DR:** **QueryInst (Instances as Queries)** is a simple and effective query based instance segmentation method driven by parallel supervision on dynamic mask heads, which outperforms previous arts in terms of both accuracy and speed.\n\n\n* Our QueryTrack (_i.e., Tracking Instances as Queries,_ [tech report](https://arxiv.org/abs/2106.11963)) based on QueryInst won [**the 2nd place** `(AP = 52.3 @ test set, AP = 54.3 @ val set)`](https://competitions.codalab.org/competitions/28988#results) in video instance segmentation (VIS) track with **single online end-to-end model, single scale testing \u0026 without using extra video training data** in the [3rd Large-scale Video Object Segmentation Challenge, CVPR 2021](https://youtube-vos.org/challenge/2021/). \n\n\n* For the first time, we demonstrate that an end-to-end query based framework driven by parallel supervision is competitive with well-established and highly-optimized methods in a wide range of instance-level recognition tasks ([object detection](https://paperswithcode.com/sota/object-detection-on-coco), [instance segmentation](https://paperswithcode.com/sota/instance-segmentation-on-coco) and video instance segmentation).\n\n#\n\n\u003e [**Instances as Queries**](https://openaccess.thecvf.com/content/ICCV2021/papers/Fang_Instances_As_Queries_ICCV_2021_paper.pdf)\n\u003e\n\u003e by [Yuxin Fang\\*](https://scholar.google.com/citations?user=_Lk0-fQAAAAJ\u0026hl=en), [Shusheng Yang\\*](https://scholar.google.com/citations?hl=zh-CN\u0026user=v6dmW5cntoMC\u0026view_op=list_works\u0026citft=1\u0026email_for_op=2yuxinfang%40gmail.com\u0026gmla=AJsN-F53CnxYBtSUBs91e_N7uL7139t5ufTWFZ-r8k5oNe1haqf_6f8AE0uyoqnVBPqNG8MGOPH_ep6k_-gMW9KmflOUalJPYu1VTaE2IVjNVn1k-lDjzMEN_oN_a7MySKPieyFEPwMfabczLcR4Qg14seBM3mx6QXUu9Hj5QMZrg9jbKDOGQxxeVX0DJtjiWCGr2ukQgSIR4VVetSaGei48SNUkO8zol-8hApyNYZcUBLD6n9FvTEeE94iLiIbFbNP0m59fh3_z), [Xinggang Wang†](https://xwcv.github.io/), [Yu Li](http://yu-li.github.io), [Chen Fang](https://scholar.google.com/citations?hl=en\u0026user=Vu1OqIsAAAAJ\u0026view_op=list_works\u0026citft=1\u0026email_for_op=2yuxinfang%40gmail.com\u0026gmla=AJsN-F5phq2a5UjdoNudoavuaCbem43ptau5cM8rWScWoxkUm0xFgCl6q49r-6MAWh-X9FVZCv9GuLk8D4u-ka0hVjKEWibox_kN9B346lA80Mrl4bUyDHBjwmbvsAfEBZ56neZ0D9p5neQBX8dBp8dD1I3248R0n0vVvzlfILA9oVpcn7xy6P0MQHUY-g0VT2g7sV6LJSPB7ZGyJFGqUk2SJ4MHRxG8U7Hz28WGuobOz-lrTnehfz5wsbwAaLETSZbP3vEMQ3Hc), [Ying Shan](https://scholar.google.com/citations?user=4oXBp9UAAAAJ\u0026hl=en), Bin Feng, [Wenyu Liu](http://eic.hust.edu.cn/professor/liuwenyu/).\n\u003e\n\u003e (\\*) equal contribution, (†) corresponding author.\n\u003e \n\u003e *[ICCV2021 Paper](https://openaccess.thecvf.com/content/ICCV2021/papers/Fang_Instances_As_Queries_ICCV_2021_paper.pdf)*\n\n![QueryInst](resources/QueryInst.png)\n\n* This repo serves as the official implementation for [QueryInst](http://arxiv.org/abs/2105.01928), based on [mmdetection](https://github.com/open-mmlab/mmdetection) and built upon [Sparse R-CNN](https://github.com/PeizeSun/SparseR-CNN) \u0026 [DETR](https://github.com/facebookresearch/detr). Implantations based on [Detectron2 ](https://github.com/facebookresearch/detectron2) will be released in the near future.\n\n* This project is under active development, we will extend [QueryInst](http://arxiv.org/abs/2105.01928) to a wide range of instance-level recognition tasks.\n\n#\n\n### Main Results on COCO test-dev\n\n|                            Configs                            |        Aug.         | Weights | Box AP | Mask AP |\n| :----------------------------------------------------------: | :----------------: | :-----: | :----: | :-----: |\n|              [QueryInst_Swin_L_300_queries (single scale testing)](configs/queryinst/queryinst_swin_large_patch4_window7_fpn_300_proposals_crop_mstrain_400-1200_50e_coco.py)               | 400 ~ 1200, w/ Crop |    [baidu](https://pan.baidu.com/s/1c-5A_XS1L79pBw5J0OlF9w) / [google](https://drive.google.com/file/d/1tqkpaArF0a0WVEolsCC8yrvgoydY7_Ha/view?usp=sharing)    |  [56.1](https://gist.github.com/Yuxin-CV/f477cb2a310e2db2b26112ae0f167baf)  |  [49.1](https://gist.github.com/Yuxin-CV/0e93ec9ab4c2d05be2d8a6cc61cd2f6b)   |\n\n\n### Main Results on COCO val\n\n|                            Configs                            |        Aug.         | Weights | Box AP | Mask AP |\n| :----------------------------------------------------------: | :----------------: | :-----: | :----: | :-----: | \n| [QueryInst\\_R50\\_3x\\_300_queries](configs/queryinst/queryinst_r50_fpn_300_proposals_crop_mstrain_480-800_3x_coco.py) | 480 ~ 800, w/ Crop |    [baidu](https://pan.baidu.com/s/1_WtTSVLLfWzKK7PAvHSylQ) / [google](https://drive.google.com/file/d/1D4Goiwb8BrDBVKkC35xm4ihUpSuF_tCF/view?usp=sharing)    |  46.9  |  41.4   | \n| [QueryInst\\_R101\\_3x\\_300_queries](configs/queryinst/queryinst_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco.py) | 480 ~ 800, w/ Crop |    [baidu](https://pan.baidu.com/s/1upDlN8SEaTpXyOXWAc9hWg) / [google](https://drive.google.com/file/d/1EYFdoKfMt99uVL2z4hTcSIVA__Z04NKE/view?usp=sharing)    |  48.0  |  42.4   |\n|              QueryInst_X101-DCN_3x_300_queries               | 480 ~ 800, w/ Crop |    -    |  50.3  |  44.2   | \n|              [QueryInst_Swin_L_300_queries (single scale testing)](configs/queryinst/queryinst_swin_large_patch4_window7_fpn_300_proposals_crop_mstrain_400-1200_50e_coco.py)          | 400 ~ 1200, w/ Crop |    [baidu](https://pan.baidu.com/s/1c-5A_XS1L79pBw5J0OlF9w) / [google](https://drive.google.com/file/d/1tqkpaArF0a0WVEolsCC8yrvgoydY7_Ha/view?usp=sharing)     |  56.1  |  48.9   |\n\nNotes:\n* Accesscode for ```baidu``` is ```QIst```.\n\n### Getting Started\n\n* Our project is mainly developed on [mmdetection toolbox `(931d96)`](https://github.com/open-mmlab/mmdetection), please refer to the [mmdetection official installation](https://github.com/open-mmlab/mmdetection/blob/master/docs/get_started.md).\n* Install `QueryInst` by:\n\n```bash\npython setup.py develop\n```\n\n* Prepare datasets:\n\n```bash\nmkdir data \u0026\u0026 cd data\nln -s /path/to/coco coco\n```\n\n* Training QueryInst with single GPU:\n\n```bash\npython tools/train.py configs/queryinst/queryinst_r50_fpn_1x_coco.py\n```\n\n* Training QueryInst with multi GPUs:\n\n```bash\n./tools/dist_train.sh configs/queryinst/queryinst_r50_fpn_1x_coco.py 8\n```\n\n* Test QueryInst on COCO val set with single GPU:\n\n```bash\npython tools/test.py configs/queryinst/queryinst_r50_fpn_1x_coco.py PATH/TO/CKPT.pth --eval bbox segm\n```\n\n* Test QueryInst on COCO val set with multi GPUs:\n\n```bash\n./tools/dist_test.sh configs/queryinst/queryinst_r50_fpn_1x_coco.py PATH/TO/CKPT.pth 8 --eval bbox segm\n```\n\n### Citation\n\nIf you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil: :\n\n```BibTeX\n@InProceedings{Fang_2021_ICCV,\n    author    = {Fang, Yuxin and Yang, Shusheng and Wang, Xinggang and Li, Yu and Fang, Chen and Shan, Ying and Feng, Bin and Liu, Wenyu},\n    title     = {Instances As Queries},\n    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},\n    month     = {October},\n    year      = {2021},\n    pages     = {6910-6919}\n}\n```\n\n```BibTeX\n@article{QueryTrack,\n  title={Tracking Instances as Queries},\n  author={Yang, Shusheng and Fang, Yuxin and Wang, Xinggang and Li, Yu and Shan, Ying and Feng, Bin and Liu, Wenyu},\n  journal={arXiv preprint arXiv:2106.11963},\n  year={2021}\n}\n```\n\n### TODO\n\n- [x] QueryInst training and inference code.\n- [x] QueryInst with Swin-Transformer and Test-Time-Augmentation.\n- [ ] QueryInst configurations for Cityscapes and YouTube-VIS.\n- [x] QueryInst pretrain weights.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhustvl%2Fqueryinst","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhustvl%2Fqueryinst","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhustvl%2Fqueryinst/lists"}