{"id":29219703,"url":"https://github.com/dvlab-research/tagclip","last_synced_at":"2025-07-03T02:06:44.235Z","repository":{"id":248481792,"uuid":"827158255","full_name":"dvlab-research/TagCLIP","owner":"dvlab-research","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-15T07:10:38.000Z","size":2543,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2024-07-15T08:37:10.472Z","etag":null,"topics":["clip","image-text","segmentation","zero-shot"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2304.07547","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dvlab-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-11T05:48:49.000Z","updated_at":"2024-07-15T08:37:23.887Z","dependencies_parsed_at":"2024-07-15T08:54:59.686Z","dependency_job_id":null,"html_url":"https://github.com/dvlab-research/TagCLIP","commit_stats":null,"previous_names":["dvlab-research/tagclip"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dvlab-research/TagCLIP","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FTagCLIP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FTagCLIP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FTagCLIP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FTagCLIP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dvlab-research","download_url":"https://codeload.github.com/dvlab-research/TagCLIP/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FTagCLIP/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263245317,"owners_count":23436514,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","image-text","segmentation","zero-shot"],"created_at":"2025-07-03T02:06:41.899Z","updated_at":"2025-07-03T02:06:44.220Z","avatar_url":"https://github.com/dvlab-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TagCLIP\nThis is the official repo of TagCLIP: Improving Discrimination Ability of Zero-Shot Semantic Segmentation\nOur paper has been accepted by TPAMI.\n\n## Overview\nContrastive Language-Image Pre-training (CLIP) has recently shown great promise in pixel-level zero-shot learning tasks. However, existing approaches utilizing CLIP's text and patch embeddings to generate semantic masks often misidentify input pixels from unseen classes, leading to confusion between novel classes and semantically similar ones. In this work, we propose a novel approach, **TagCLIP**~(**T**rusty-**a**ware **g**uided CLIP), to address this issue. We disentangle the ill-posed optimization problem into two parallel processes: semantic matching performed individually and reliability judgment for improving discrimination ability. Building on the idea of special tokens in language modeling representing sentence-level embeddings, we introduce a trusty token that enables distinguishing novel classes from known ones in prediction. To evaluate our approach, we conduct experiments on two benchmark datasets, PASCAL VOC 2012 and COCO-Stuff 164K. Our results show that TagCLIP improves the Intersection over Union (IoU) of unseen classes by 7.4% and 1.7%, respectively, with negligible overheads.\n\n![Alt text](imgs/framework.png)\n\n## Preparation\n### Environment\n```bash\npip install -r requirements.txt\n```\n### Dataset\nThe data preparation document is [here](https://github.com/open-mmlab/mmsegmentation/blob/master/docs/en/dataset_prepare.md)\n\n### Pretrained CLIP model\n```\nmkdir pretrained \u0026 cd pretrained\nwget https://openaipublic.azureedge.net/clip/models/5806e77cd80f8b59890b7e101eabd078d9fb84e6937f9e85e4ecb61988df416f/ViT-B-16.pt\n```\n\n## Train\n### Inductive\n```bash\ndataset=voc12 # support voc12, coco\n# multi gpu training\nbash src/dist_train.sh src/configs/$dataset/inductive.py output/$dataset/inductive\n# single gpu training\npython src/train.py src/configs/$dataset/inductive.py --work-dir=output/$dataset/inductive\n```\n### Transductive\n```bash\n# coco multi gpu training\nbash src/dist_train.sh src/configs/coco/transductive.py output/coco/transductive --load-from=output/coco/inductive/iter_40000.pth\n# voc12 multi gpu training\nbash src/dist_train.sh src/configs/voc12/transductive.py output/voc12/transductive --load-from=output/voc12/transductive/iter_10000.pth\n# coco single gpu training\npython src/train.py src/configs/coco/transductive.py output/coco/transductive --load-from=output/coco/inductive/iter_40000.pth\n# voc single gpu training\npython src/train.py src/configs/voc12/transductive.py output/voc12/transductive --load-from=output/voc12/transductive/iter_10000.pth\n```\n\n## Inference\n```bash\n# intra dataset\npython src/test.py $config $ckpt --eval=mIoU\n# cross dataset\npython src/test.py src/configs/cross_dataset/coco-to-voc.py output/coco/inductive/iter_80000.pth --eval=mIoU\npython src/test.py src/configs/cross_dataset/coco-to-context.py output/coco/inductive/iter_80000.pth --eval=mIoU\n```\n\n## Acknowledgement\nOur work is closely related to the following assets that inspire our implementation. We gratefully thank the authors. \n\n - [CLIP](https://github.com/openai/CLIP)\n - [Maskformer](https://bowenc0221.github.io/maskformer)\n - [Zegformer](https://github.com/dingjiansw101/ZegFormer)\n - [zsseg](https://github.com/MendelXu/zsseg.baseline)\n - [MaskCLIP](https://github.com/chongzhou96/MaskCLIP)\n - [SegViT](https://github.com/zbwxp/SegVit)\n - [ZegCLIP](https://github.com/ZiqinZhou66/ZegCLIP)\n\n## Citation\n```\nIf you find this project useful, please consider citing:\n@misc{tagclip,\n      title={TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentation}, \n      author={Jingyao Li and Pengguang Chen and Shengju Qian and Jiaya Jia},\n      year={2023},\n      eprint={2304.07547},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2304.07547}, \n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdvlab-research%2Ftagclip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdvlab-research%2Ftagclip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdvlab-research%2Ftagclip/lists"}