{"id":21091775,"url":"https://github.com/pengtaojiang/segment-anything-clip","last_synced_at":"2025-04-04T21:08:41.038Z","repository":{"id":152028362,"uuid":"625147985","full_name":"PengtaoJiang/Segment-Anything-CLIP","owner":"PengtaoJiang","description":"Connecting segment-anything's output masks with the CLIP model; Awesome-Segment-Anything-Works","archived":false,"fork":false,"pushed_at":"2024-10-13T03:12:50.000Z","size":23107,"stargazers_count":190,"open_issues_count":1,"forks_count":14,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-28T20:08:05.923Z","etag":null,"topics":["classification","clip","segment-anything","semantic-segmentation"],"latest_commit_sha":null,"homepage":"https://github.com/PengtaoJiang/Segment-Anything-CLIP","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PengtaoJiang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-08T07:58:07.000Z","updated_at":"2025-03-26T02:03:26.000Z","dependencies_parsed_at":"2024-12-14T11:11:00.577Z","dependency_job_id":"eb8e09ae-5f6e-4a64-a88d-6b28c40e0fd3","html_url":"https://github.com/PengtaoJiang/Segment-Anything-CLIP","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PengtaoJiang%2FSegment-Anything-CLIP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PengtaoJiang%2FSegment-Anything-CLIP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PengtaoJiang%2FSegment-Anything-CLIP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PengtaoJiang%2FSegment-Anything-CLIP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PengtaoJiang","download_url":"https://codeload.github.com/PengtaoJiang/Segment-Anything-CLIP/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247249526,"owners_count":20908212,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["classification","clip","segment-anything","semantic-segmentation"],"created_at":"2024-11-19T21:47:14.253Z","updated_at":"2025-04-04T21:08:41.019Z","avatar_url":"https://github.com/PengtaoJiang.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Conect Segment-Anything with CLIP\nWe aim to classify the output masks of [segment-anything](https://github.com/facebookresearch/segment-anything) with the off-the-shelf [CLIP](https://github.com/openai/CLIP) models. The cropped image corresponding to each mask is sent to the CLIP model.\n\u003cimg src=\"https://github.com/PengtaoJiang/SAM-CLIP/blob/main/imgs/pipeline.png\" width=\"100%\" height=\"50%\"\u003e\n\n## Other Nice Works\n\n#### Editing-Related Works\n1. [sail-sg/EditAnything](https://github.com/sail-sg/EditAnything) \n2. [IDEA-Research/Grounded-Segment-Anything](https://github.com/IDEA-Research/Grounded-Segment-Anything) \n3. [geekyutao/Inpaint-Anything](https://github.com/geekyutao/Inpaint-Anything)\n4. [Luodian/RelateAnything](https://github.com/Luodian/RelateAnything)\n\n#### Nerf-Related Works \n1. [ashawkey/Segment-Anything-NeRF](https://github.com/ashawkey/Segment-Anything-NeRF)\n2. [Anything-of-anything/Anything-3D](https://github.com/Anything-of-anything/Anything-3D)\n3. [Jun-CEN/SegmentAnyRGBD](https://github.com/Jun-CEN/SegmentAnyRGBD)\n4. [Pointcept/SegmentAnything3D](https://github.com/Pointcept/SegmentAnything3D)\n\n#### Segmentation-Related Works\n1. [maxi-w/CLIP-SAM](https://github.com/maxi-w/CLIP-SAM)  \n2. [Curt-Park/segment-anything-with-clip](https://github.com/Curt-Park/segment-anything-with-clip)  \n3. [kadirnar/segment-anything-video](https://github.com/kadirnar/segment-anything-video) \n4. [fudan-zvg/Semantic-Segment-Anything](https://github.com/fudan-zvg/Semantic-Segment-Anything) \n5. [continue-revolution/sd-webui-segment-anything](https://github.com/continue-revolution/sd-webui-segment-anything) \n6. [RockeyCoss/Prompt-Segment-Anything](https://github.com/RockeyCoss/Prompt-Segment-Anything) \n7. [ttengwang/Caption-Anything](https://github.com/ttengwang/Caption-Anything)  \n8. [ngthanhtin/owlvit_segment_anything](https://github.com/ngthanhtin/owlvit_segment_anything)   \n9. [lang-segment-anything](https://github.com/luca-medeiros/lang-segment-anything)\n10. [helblazer811/RefSAM](https://github.com/helblazer811/RefSAM)  \n11. [Hedlen/awesome-segment-anything](https://github.com/Hedlen/awesome-segment-anything)  \n12. [ziqi-jin/finetune-anythin](https://github.com/ziqi-jin/finetune-anything)\n13. [ylqi/Count-Anything](https://github.com/ylqi/Count-Anything)\n14. [xmed-lab/CLIP_Surgery](https://github.com/xmed-lab/CLIP_Surgery)\n15. [RockeyCoss/Prompt-Segment-Anything](https://github.com/RockeyCoss/Prompt-Segment-Anything)\n16. [segments-ai/panoptic-segment-anything](https://github.com/segments-ai/panoptic-segment-anything)\n17. [Cheems-Seminar/grounded-segment-any-parts](https://github.com/Cheems-Seminar/grounded-segment-any-parts)\n18. [aim-uofa/Matcher](https://github.com/aim-uofa/Matcher)\n19. [SysCV/sam-hq](https://github.com/SysCV/sam-hq)\n20. [CASIA-IVA-Lab/FastSAM](https://github.com/CASIA-IVA-Lab/FastSAM)\n21. [ChaoningZhang/MobileSAM](https://github.com/ChaoningZhang/MobileSAM)\n22. [JamesQFreeman/Sam_LoRA](https://github.com/JamesQFreeman/Sam_LoRA)\n23. [UX-Decoder/Semantic-SAM](https://github.com/UX-Decoder/Semantic-SAM)\n24. [cskyl/SAM_WSSS](https://github.com/cskyl/SAM_WSSS)\n25. [ggsDing/SAM-CD](https://github.com/ggsDing/SAM-CD)\n26. [yformer/EfficientSAM](https://github.com/yformer/EfficientSAM)\n27. [XiaRho/SEMat](https://github.com/XiaRho/SEMat)\n\n#### Labelling-Related Works\n1. [vietanhdev/anylabeling](https://github.com/vietanhdev/anylabeling)\n2. [anuragxel/salt](https://github.com/anuragxel/salt)\n\n\n#### Tracking-Related Works\n1. [gaomingqi/track-anything](https://github.com/gaomingqi/track-anything) \n2. [z-x-yang/Segment-and-Track-Anything](https://github.com/z-x-yang/Segment-and-Track-Anything)\n3. [achalddave/segment-any-moving](https://github.com/achalddave/segment-any-moving)\n\n#### Medical-Related Works\n1. [bowang-lab/medsam](https://github.com/bowang-lab/medsam)\n2. [hitachinsk/SAMed](https://github.com/hitachinsk/SAMed)\n3. [cchen-cc/MA-SAM](https://github.com/cchen-cc/MA-SAM#ma-sam-modality-agnostic-sam-adaptation-for-3d-medical-image-segmentation)\n4. [OpenGVLab/SAM-Med2D](https://github.com/OpenGVLab/SAM-Med2D)\n\n## Todo\n1. We plan to connect segment-anything with [MaskCLIP](https://github.com/chongzhou96/MaskCLIP).  \n2. We plan to finetune on the COCO and LVIS datasets.  \n\n\n## Run Demo\nDownload the [sam_vit_h_4b8939.pth](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth) model from the SAM repository and put it at ```./SAM-CLIP/```. Follow the instructions to install segment-anything and clip packages using the following command. \n```\ncd SAM-CLIP; pip install -e .\npip install git+https://github.com/openai/CLIP.git\n```\nThen run the following script:\n```\nsh run.sh\n```\n\n## Example \nInput an example image and a point (250, 250) to the SAM model. The input image and output three masks as follows:\n\u003ccenter\u003e\u003cimg src=\"https://github.com/PengtaoJiang/SAM-CLIP/blob/main/imgs/ADE_val_00000001.jpg\" width=\"50%\" height=\"50%\"\u003e\u003c/center\u003e\n\nThe three masks and corresponding predicted category are as follows:\n\u003ccenter\u003e\n\u003cimg src=\"https://github.com/PengtaoJiang/SAM-CLIP/blob/main/outs/ADE_val_00000001/outs.png\" width=\"100%\" height=\"50%\"\u003e \n\u003c/center\u003e\n\u003ccenter\u003e\n\u003cimg src=\"https://github.com/PengtaoJiang/SAM-CLIP/blob/main/outs/ADE_val_00000001/logits.png\" width=\"100%\" height=\"50%\"\u003e \n\u003c/center\u003e\n\nYou can change the point location at L273-274 of ```scripts/amp_points.py```.\n```\n## input points \ninput_points_list = [[250, 250]]\nlabel_list = [1]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpengtaojiang%2Fsegment-anything-clip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpengtaojiang%2Fsegment-anything-clip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpengtaojiang%2Fsegment-anything-clip/lists"}