{"id":22067894,"url":"https://github.com/chenxi52/CMPF","last_synced_at":"2025-07-24T05:31:04.703Z","repository":{"id":255665985,"uuid":"773253618","full_name":"chenxi52/CMPF","owner":"chenxi52","description":"Open-Vocabulary Panoptic Segmentation","archived":false,"fork":false,"pushed_at":"2025-06-15T04:19:11.000Z","size":1179,"stargazers_count":24,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-15T05:22:11.377Z","etag":null,"topics":["clip","instance-segmentation","multi-modal-learning","open-vocabulary","open-vocabulary-segmentation","open-vocabulary-semantic-segmentation","panoptic-segmentation","segment-anything","segmentation","vision-and-language","zero-shot"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chenxi52.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-03-17T06:38:53.000Z","updated_at":"2025-06-15T04:19:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"c5ef6a73-b253-4fa2-a164-114d4bde2c80","html_url":"https://github.com/chenxi52/CMPF","commit_stats":null,"previous_names":["chenxi52/frozenseg","chenxi52/cmpf"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/chenxi52/CMPF","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenxi52%2FCMPF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenxi52%2FCMPF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenxi52%2FCMPF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenxi52%2FCMPF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chenxi52","download_url":"https://codeload.github.com/chenxi52/CMPF/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenxi52%2FCMPF/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266796832,"owners_count":23985483,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","instance-segmentation","multi-modal-learning","open-vocabulary","open-vocabulary-segmentation","open-vocabulary-semantic-segmentation","panoptic-segmentation","segment-anything","segmentation","vision-and-language","zero-shot"],"created_at":"2024-11-30T20:03:35.984Z","updated_at":"2025-07-24T05:31:04.673Z","avatar_url":"https://github.com/chenxi52.png","language":"Python","funding_links":[],"categories":["Paper List"],"sub_categories":["Follow-up Papers"],"readme":"# CMPF: Harmonizing Cross-Model Prior Fusion for Open-Vocabulary Segmentation\n\n\n\n\n## Abstract\n\n\u003eOpen-vocabulary segmentation poses significant challenges, as it requires segmenting and recognizing objects across an open set of categories in unconstrained environments. Building on the success of powerful vision-language (ViL) foundation models, such as CLIP, recent efforts sought to harness their zero-shot capabilities to recognize unseen categories. Despite notable performance improvements, these models still encounter the critical issue of generating and recognizing precise mask proposals for unseen categories and scenarios, resulting in inferior segmentation performance eventually. To address this challenge, we introduce a novel Cross-Model Prior Fusion (CMPF) framework, an innovative framework that fuses visual knowledge from a localization foundation model (e.g., SAM) and text knowledge from a ViL model (e.g., CLIP), leveraging their complementary knowledge priors to overcome inherent limitations in mask proposal generation. Taking the ViL model’s visual encoder as the feature backbone, we propose Query Injector and Feature Injector to inject the visual localization feature into the learnable queries and CLIP features respectively, within a transformer decoder. In addition, an OpenSeg Ensemble strategy is designed to further improve mask quality by incorporating SAM’s universal segmentation masks during inference. To fully exploit pre-trained knowledge while minimizing training overhead, we freeze both foundation models, focusing optimization efforts solely on a lightweight transformer decoder for mask proposal generation – the performance bottleneck. Extensive experiments demonstrate that CMPF advances state-of-the-art results across various segmentation benchmarks, trained exclusively on COCO panoptic data, and tested in a zero-shot manner.\n\n![FrozenSeg design](images/frozenseg.png)\n\n## Dependencies and Installation\nSee [installation instructions](INSTALL.md).\n\n## Getting Started\nSee [Preparing Datasets](datasets/README.md).\n\nSee [Getting Started](GETTING_STARTED.md).\n\n\n## Models\n\u003ctable\u003e\n\u003cthead\u003e\n  \u003ctr\u003e\n    \u003cth align=\"center\"\u003e\u003c/th\u003e\n    \u003cth align=\"center\" style=\"text-align:center\" colspan=\"4\"\u003e\u003ca href=\"logs/testing/ade20k.log\"\u003eADE20K(A-150)\u003c/th\u003e\n    \u003cth align=\"center\" style=\"text-align:center\" colspan=\"3\"\u003e\u003ca href=\"logs/testing/cityscapes.log\"\u003eCityscapes\u003c/th\u003e\n    \u003cth align=\"center\" style=\"text-align:center\" colspan=\"2\"\u003e\u003ca href=\"logs/testing/mapillary_vistas.log\"\u003eMapillary Vistas\u003c/th\u003e\n    \u003cth align=\"center\" style=\"text-align:center\" colspan=\"2\"\u003e\u003ca href=\"logs/testing/bdd100k.log\"\u003eBDD 100K\u003c/th\u003e\n    \u003cth align=\"center\" style=\"text-align:center\" colspan=\"2\"\u003e\u003ca href=\"logs/testing/a-847.log\"\u003e A-847 \u003c/th\u003e\n    \u003cth align=\"center\" style=\"text-align:center\" colspan=\"2\"\u003e\u003ca href=\"logs/testing/pc-459.log\"\u003e PC-459 \u003c/th\u003e\n    \u003cth align=\"center\" style=\"text-align:center\" colspan=\"2\"\u003e\u003ca href=\"logs/testing/pas-21.log\"\u003ePAS-21 \u003c/th\u003e\n    \u003cth align=\"center\" style=\"text-align:center\" \u003e\u003ca href=\"logs/testing/lvis.log\"\u003eLvis \u003c/th\u003e\n    \u003cth align=\"center\" style=\"text-align:center\" colspan=\"3\"\u003e\u003ca href=\"logs/testing/coco.log\"\u003eCOCO \u003cbr\u003e (training dataset)\u003c/th\u003e\n    \u003cth align=\"center\" style=\"text-align:center\"\u003edownload \u003c/th\u003e\n  \u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003ePQ\u003c/td\u003e\n    \u003ctd align=\"center\"\u003emAP\u003c/td\u003e\n    \u003ctd align=\"center\"\u003emIoU\u003c/td\u003e\n    \u003ctd align=\"center\"\u003eFWIoU\u003c/td\u003e\n    \u003ctd align=\"center\"\u003ePQ\u003c/td\u003e\n    \u003ctd align=\"center\"\u003emAP\u003c/td\u003e\n    \u003ctd align=\"center\"\u003emIoU\u003c/td\u003e\n    \u003ctd align=\"center\"\u003ePQ\u003c/td\u003e\n    \u003ctd align=\"center\"\u003emIoU\u003c/td\u003e\n    \u003ctd align=\"center\"\u003ePQ\u003c/td\u003e\n    \u003ctd align=\"center\"\u003emIoU\u003c/td\u003e\n    \u003ctd align=\"center\"\u003emIoU\u003c/td\u003e\n    \u003ctd align=\"center\"\u003eFWIoU\u003c/td\u003e\n    \u003ctd align=\"center\"\u003emIoU\u003c/td\u003e\n    \u003ctd align=\"center\"\u003eFWIoU\u003c/td\u003e\n    \u003ctd align=\"center\"\u003emIoU\u003c/td\u003e\n    \u003ctd align=\"center\"\u003eFWIoU\u003c/td\u003e\n    \u003ctd align=\"center\"\u003eAPr\u003c/td\u003e\n    \u003ctd align=\"center\"\u003ePQ\u003c/td\u003e\n    \u003ctd align=\"center\"\u003emAP\u003c/td\u003e\n    \u003ctd align=\"center\"\u003emIoU\u003c/td\u003e\n    \u003ctd\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"configs/coco/frozenseg/r50x64_eval_ade20k.yaml\"\u003e CMPF (ResNet50x64) \u003c/a\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e23.1\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e13.5\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e30.7\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e56.6\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e45.2\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e28.9\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e56.0\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e18.1\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e27.7\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e12.9\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e46.2\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e11.8\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e52.8\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e18.7\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e60.1\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e82.3\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e92.1\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e23.5\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e55.7\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e47.4\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e65.4\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1wiPKPczTXrH1cPrDbf-l9NGdOkMUUikw/view?usp=drive_link\"\u003e checkpoint \u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"configs/coco/frozenseg/convnext_large_eval_ade20k.yaml\"\u003e CMPF (ConvNeXt-Large) \u003c/a\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e25.9\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e16.5\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e34.4\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e59.9\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e45.8\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e28.4\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e56.8\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e18.5\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e27.3\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e19.3\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e52.3\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e14.8\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e51.4\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e19.7\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e60.2\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e82.5\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e92.1\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e25.6\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e56.2\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e47.3\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e65.5\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1ThjVgY7nawm1AAP1LhrmGVlI3zr1EYMG/view?usp=drive_link\"\u003e checkpoint \u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\n## !!Note: \n\nThis repository serves as the official implementation for both **CMPF** and **FrozenSeg**, which are essentially the same work presented under different names.\n\u003e[FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary Segmentation](https://arxiv.org/abs/2409.03525)\n\n##  Acknowledgement\n[Detectron2](https://github.com/facebookresearch/detectron2), [Mask2Former](https://github.com/facebookresearch/Mask2Former), [Segment Anything](https://github.com/facebookresearch/segment-anything), [OpenCLIP](https://github.com/mlfoundations/open_clip) and [FC-CLIP](https://github.com/bytedance/fc-clip/tree/main).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenxi52%2FCMPF","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchenxi52%2FCMPF","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenxi52%2FCMPF/lists"}