{"id":30298022,"url":"https://github.com/vivocameraresearch/sdmatte","last_synced_at":"2026-03-02T10:31:02.701Z","repository":{"id":307624947,"uuid":"1008782014","full_name":"vivoCameraResearch/SDMatte","owner":"vivoCameraResearch","description":"Official code for our ICCV2025 paper \"SDMatte: Grafting Diffusion Models for Interactive Matting\"","archived":false,"fork":false,"pushed_at":"2025-08-15T14:29:13.000Z","size":2894,"stargazers_count":82,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-15T16:25:18.926Z","etag":null,"topics":["alpha-matting","interactive-matting","matting","matting-algorithms","object-segmentation","sdmatte","stable-diffusion"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vivoCameraResearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-26T05:01:34.000Z","updated_at":"2025-08-15T14:29:16.000Z","dependencies_parsed_at":"2025-08-01T09:54:01.760Z","dependency_job_id":null,"html_url":"https://github.com/vivoCameraResearch/SDMatte","commit_stats":null,"previous_names":["vivocameraresearch/sdmatte"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vivoCameraResearch/SDMatte","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vivoCameraResearch%2FSDMatte","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vivoCameraResearch%2FSDMatte/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vivoCameraResearch%2FSDMatte/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vivoCameraResearch%2FSDMatte/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vivoCameraResearch","download_url":"https://codeload.github.com/vivoCameraResearch/SDMatte/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vivoCameraResearch%2FSDMatte/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270803621,"owners_count":24648707,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-17T02:00:09.016Z","response_time":129,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alpha-matting","interactive-matting","matting","matting-algorithms","object-segmentation","sdmatte","stable-diffusion"],"created_at":"2025-08-17T04:01:28.652Z","updated_at":"2025-08-17T04:03:32.460Z","avatar_url":"https://github.com/vivoCameraResearch.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cp align=\"center\"\u003e\n    \u003cimg src=\"./assets/logo.png\" alt=\"SDMatte Logo\" height=\"52\"\u003e\n  \u003c/p\u003e\n\n  \u003ch1 align=\"center\"\u003eGrafting Diffusion Models for Interactive Matting\u003c/h1\u003e\n\n  \u003cp align=\"center\"\u003e\n    \u003cb\u003e\n      Longfei Huang\u003csup\u003e1,2*\u003c/sup\u003e, Yu Liang\u003csup\u003e2*\u003c/sup\u003e, Hao Zhang\u003csup\u003e2\u003c/sup\u003e,\n      Jinwei Chen\u003csup\u003e2\u003c/sup\u003e, Wei Dong\u003csup\u003e2\u003c/sup\u003e,\u003cbr\u003e\n      Lunde Chen\u003csup\u003e1\u003c/sup\u003e, Wanyu Liu\u003csup\u003e1\u003c/sup\u003e, Bo Li\u003csup\u003e2\u003c/sup\u003e,\n      Peng-Tao Jiang\u003csup\u003e2†\u003c/sup\u003e\n    \u003c/b\u003e\u003cbr\u003e\n    \u003csup\u003e1\u003c/sup\u003e Shanghai University \u0026nbsp;\u0026nbsp;\u0026nbsp;\n    \u003csup\u003e2\u003c/sup\u003e vivo Mobile Communication Co., Ltd.\u003cbr\u003e\n    \u003csup\u003e*\u003c/sup\u003e Equal contribution \u0026nbsp;\u0026nbsp;\u0026nbsp;\n    \u003csup\u003e†\u003c/sup\u003e Corresponding author\n  \u003c/p\u003e\n\n  [![project page](https://img.shields.io/badge/🌐-Project%20Page-3977dd)](https://github.com/vivoCameraResearch/SDMatte)\n  [![arxiv paper](https://img.shields.io/badge/arxiv-2508.00443-orange)](https://arxiv.org/abs/2508.00443)\n  [![license](https://img.shields.io/badge/license-MIT-A7D8DE)](https://opensource.org/license/MIT)\n  [![huggingface](https://img.shields.io/badge/🤗-HuggingFace%20Space-green)](https://huggingface.co/LongfeiHuang/SDMatte)\n  [![by vivoCRL](https://img.shields.io/badge/by-vivoCRL-blue)](https://github.com/vivoCameraResearch)\n\n  \u003cstrong\u003e\u003ci\u003eSDMatte\u003c/i\u003e is an interactive image matting method based on stable diffusion, which supports three types of visual prompts (points, boxes, and masks) for accurately extracting target objects from natural images.\u003c/strong\u003e\n\n  \u003cdiv style=\"width: 100%; text-align: center; margin:auto;\"\u003e\n      \u003cimg style=\"width:100%\" src=\"assets/result_show.png\"\u003e\n  \u003c/div\u003e\n\u003c/div\u003e\n\n## 📢 News\n- [2025.08.16] Released the LiteSDMatte weights.  \n- [2025.08.05] Released the SDMatte and SDMatte* weights.\n- [2025.08.04] Released evaluation codes. \n- [2025.08.01] Published the [arXiv preprint](https://arxiv.org/abs/2508.00443).  \n- [2025.07.31] This repo is created. \n- [2025.06.26] Paper accepted by ICCV 2025.\n\n## 🤝 Community Works\nIf your work has improved **SDMatte** and you would like more people to see it, please inform us.\n- [ComfyUI-SDMatte](https://github.com/flybirdxx/ComfyUI-SDMatte), a ComfyUI custom node built on SDMatte, offers interactive high-precision image matting with refined edge detail preservation and optimized VRAM efficiency.\n\n## 📖 Overview\n### Abstract\n\nRecent interactive matting methods have demonstrated satisfactory performance in capturing the primary regions of objects, but they fall short in extracting fine-grained details in edge regions. Diffusion models trained on billions of image-text pairs, demonstrate exceptional capability in modeling highly complex data distributions and synthesizing realistic texture details, while exhibiting robust text-driven interaction capabilities, making them an attractive solution for interactive matting. \nTo this end, we propose SDMatte, a diffusion-driven interactive matting model, with three key contributions.\nFirst, we exploit the powerful priors of the pre-trained U-Net within diffusion models and transform the text-driven interaction mechanism into a visual prompt-driven interaction mechanism to enable interactive matting.\nSecond, we integrate coordinate embeddings of visual prompts and opacity embeddings of objects into U-Net, enhancing SDMatte's sensitivity to spatial position information and opacity information.\nThird, we propose a masked self-attention mechanism and a visual prompt-driven interaction mechanism that enable the model to focus on areas specified by visual prompts, leading to better performance.\nExtensive experiments on multiple datasets demonstrate the superior performance of our method, validating its effectiveness in interactive matting.\n\n### Architecture\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"assets/pipeline.png\" width=\"100%\" height=\"100%\"/\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n### Visualization\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"assets/compare.png\" width=\"100%\" height=\"100%\"/\u003e\n\u003c/div\u003e\n\n## 🛠️ Installation\n\n1. Create a conda virtual env and activate it.\n\n    ```\n    conda create -n SDMatte python==3.10\n    conda activate SDMatte\n    ```\n2. Install packages.\n    ```\n    cd path/to/SDMatte\n    pip install -r requirements.txt\n    ```\n3. Install [detectron2](https://github.com/facebookresearch/detectron2) , follow its [documentation](https://detectron2.readthedocs.io/en/latest/).\n  For SDMatte, we recommend to build it from latest source code.\n    ```\n    python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'\n    ```\n\n## 📂 Dataset\n\n* To train **SDMatte** and **LiteSDMatte**, please prepare the following datasets: [Composition-1K](https://sites.google.com/view/deepimagematting), [DIS-646](https://github.com/yuhaoliu7456/CVPR2020-HAttMatting), [AM-2K](https://github.com/JizhiziLi/GFM), [UHRSD](https://github.com/iCVTEAM/PGNet), [RefMatte](https://github.com/jizhiziLi/rim), and [BG-20K](https://github.com/JizhiziLi/GFM).\n\n* To train **SDMatte\u003csup\u003e*\u003c/sup\u003e**, please prepare the following datasets: [Composition-1K](https://sites.google.com/view/deepimagematting), [DIS-646](https://github.com/yuhaoliu7456/CVPR2020-HAttMatting), [AM-2K](https://github.com/JizhiziLi/GFM), [COCO-Matte](https://github.com/XiaRho/SEMat), and [BG-20K](https://github.com/JizhiziLi/GFM).\n\n* To evaluate **SDMatte**, please prepare the following test datasets: [AIM-500](https://github.com/JizhiziLi/AIM), [AM-2K](https://github.com/JizhiziLi/GFM), [P3M-500](https://github.com/JizhiziLi/P3M), and [RefMatte-RW-100](https://github.com/jizhiziLi/rim).\n\n* Check lines 15–56 and 529–530 in `SDMatte/data/dataset.py` to modify the data path to your data path.\n\n## 📦 Model Zoo\n\n* Download the weights and configurations of **SDMatte** and **SDMatte\u003csup\u003e*\u003c/sup\u003e**,** from the Hugging Face repository [LongfeiHuang/SDMatte](https://huggingface.co/LongfeiHuang/SDMatte)\n* Download the weight and configuration of **LiteSDMatte** from the Hugging Face repository [LongfeiHuang/LiteSDMatte](https://huggingface.co/LongfeiHuang/LiteSDMatte)\n\n## 🔍 Test\n\n* Modify the `pretrained_model_name_or_path` field in `configs/SDMatte.py` to the directory path containing the configuration files, so that the model can be properly initialized.\n\n* Modify the `CKPT_DIR` parameter in `script/test_SDMatte.sh` or `script/test_SDMatte.sh` to the specific path of the downloaded weight file.\n\n* Run the following command to evaluate **SDMatte** or **LiteSDMatte**.\n\n    ```\n    bash script/test_SDMatte.sh\n    bash script/test_LiteSDMatte.sh\n    ```\n\n\n## 📋 TodoList\n\n- [x] Publish paper on arXiv  \n- [x] Release source code for SDMatte  \n- [x] Release evaluation codes  \n- [x] Release pretrained weights for SDMatte and SDMatte*  \n- [x] Release source code for LiteSDMatte  \n- [x] Release pretrained weights for LiteSDMatte  \n- [ ] Release training code\n- [ ] Deploy interactive demo using Gradio or on Hugging Face Spaces  \n\n## 📝 License\n\nThis project is licensed under \u003ca rel=\"license\" href=\"./LICENSE\"\u003eMIT\u003c/a\u003e. Redistribution and use should follow this license.\n\n## ✏️ Cite Us\nIf you find this repository or our work useful, please consider citing us:\n```\n@inproceedings{huang2025sdmatte,\n  title={SDMatte: Grafting Diffusion Models for Interactive Matting},\n  author={Huang, Longfei and Liang, Yu and Zhang, Hao and Chen, Jinwei and Dong, Wei and Chen, Lunde and Liu, Wanyu and Li, Bo and Jiang, Peng-Tao},\n  booktitle={Proceedings of the IEEE/CVF international conference on computer vision},\n  year={2025}\n}\n```\n\n## 👏 Acknowledgement\n\nOur repo is built upon [Stable Diffusion 2](https://github.com/Stability-AI/stablediffusion), [TAESD](https://github.com/madebyollin/taesd), and [BK-SDM](https://github.com/Nota-NetsPresso/BK-SDM). We sincerely thank the authors for their contributions to the community.\n\n## 📧 Contact\n\nIf you have any questions, please feel free to reach us at `2946399650fly@shu.edu.cn` or ` pt.jiang@vivo.com`. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvivocameraresearch%2Fsdmatte","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvivocameraresearch%2Fsdmatte","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvivocameraresearch%2Fsdmatte/lists"}