https://github.com/vivocameraresearch/sdmatte
Official code for our ICCV2025 paper "SDMatte: Grafting Diffusion Models for Interactive Matting"
https://github.com/vivocameraresearch/sdmatte
alpha-matting interactive-matting matting matting-algorithms object-segmentation sdmatte stable-diffusion
Last synced: about 2 months ago
JSON representation
Official code for our ICCV2025 paper "SDMatte: Grafting Diffusion Models for Interactive Matting"
- Host: GitHub
- URL: https://github.com/vivocameraresearch/sdmatte
- Owner: vivoCameraResearch
- License: mit
- Created: 2025-06-26T05:01:34.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2025-08-15T14:29:13.000Z (about 2 months ago)
- Last Synced: 2025-08-15T16:25:18.926Z (about 2 months ago)
- Topics: alpha-matting, interactive-matting, matting, matting-algorithms, object-segmentation, sdmatte, stable-diffusion
- Language: Python
- Homepage:
- Size: 2.76 MB
- Stars: 82
- Watchers: 2
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
Grafting Diffusion Models for Interactive Matting
Longfei Huang1,2*, Yu Liang2*, Hao Zhang2,
Jinwei Chen2, Wei Dong2,
Lunde Chen1, Wanyu Liu1, Bo Li2,
Peng-Tao Jiang2†
1 Shanghai University
2 vivo Mobile Communication Co., Ltd.
* Equal contribution
† Corresponding author
[](https://github.com/vivoCameraResearch/SDMatte)
[](https://arxiv.org/abs/2508.00443)
[](https://opensource.org/license/MIT)
[](https://huggingface.co/LongfeiHuang/SDMatte)
[](https://github.com/vivoCameraResearch)
SDMatte is an interactive image matting method based on stable diffusion, which supports three types of visual prompts (points, boxes, and masks) for accurately extracting target objects from natural images.
## 📢 News
- [2025.08.16] Released the LiteSDMatte weights.
- [2025.08.05] Released the SDMatte and SDMatte* weights.
- [2025.08.04] Released evaluation codes.
- [2025.08.01] Published the [arXiv preprint](https://arxiv.org/abs/2508.00443).
- [2025.07.31] This repo is created.
- [2025.06.26] Paper accepted by ICCV 2025.
## 🤝 Community Works
If your work has improved **SDMatte** and you would like more people to see it, please inform us.
- [ComfyUI-SDMatte](https://github.com/flybirdxx/ComfyUI-SDMatte), a ComfyUI custom node built on SDMatte, offers interactive high-precision image matting with refined edge detail preservation and optimized VRAM efficiency.
## 📖 Overview
### Abstract
Recent interactive matting methods have demonstrated satisfactory performance in capturing the primary regions of objects, but they fall short in extracting fine-grained details in edge regions. Diffusion models trained on billions of image-text pairs, demonstrate exceptional capability in modeling highly complex data distributions and synthesizing realistic texture details, while exhibiting robust text-driven interaction capabilities, making them an attractive solution for interactive matting.
To this end, we propose SDMatte, a diffusion-driven interactive matting model, with three key contributions.
First, we exploit the powerful priors of the pre-trained U-Net within diffusion models and transform the text-driven interaction mechanism into a visual prompt-driven interaction mechanism to enable interactive matting.
Second, we integrate coordinate embeddings of visual prompts and opacity embeddings of objects into U-Net, enhancing SDMatte's sensitivity to spatial position information and opacity information.
Third, we propose a masked self-attention mechanism and a visual prompt-driven interaction mechanism that enable the model to focus on areas specified by visual prompts, leading to better performance.
Extensive experiments on multiple datasets demonstrate the superior performance of our method, validating its effectiveness in interactive matting.
### Architecture
### Visualization
## 🛠️ Installation
1. Create a conda virtual env and activate it.
```
conda create -n SDMatte python==3.10
conda activate SDMatte
```
2. Install packages.
```
cd path/to/SDMatte
pip install -r requirements.txt
```
3. Install [detectron2](https://github.com/facebookresearch/detectron2) , follow its [documentation](https://detectron2.readthedocs.io/en/latest/).
For SDMatte, we recommend to build it from latest source code.
```
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
```
## 📂 Dataset
* To train **SDMatte** and **LiteSDMatte**, please prepare the following datasets: [Composition-1K](https://sites.google.com/view/deepimagematting), [DIS-646](https://github.com/yuhaoliu7456/CVPR2020-HAttMatting), [AM-2K](https://github.com/JizhiziLi/GFM), [UHRSD](https://github.com/iCVTEAM/PGNet), [RefMatte](https://github.com/jizhiziLi/rim), and [BG-20K](https://github.com/JizhiziLi/GFM).
* To train **SDMatte***, please prepare the following datasets: [Composition-1K](https://sites.google.com/view/deepimagematting), [DIS-646](https://github.com/yuhaoliu7456/CVPR2020-HAttMatting), [AM-2K](https://github.com/JizhiziLi/GFM), [COCO-Matte](https://github.com/XiaRho/SEMat), and [BG-20K](https://github.com/JizhiziLi/GFM).
* To evaluate **SDMatte**, please prepare the following test datasets: [AIM-500](https://github.com/JizhiziLi/AIM), [AM-2K](https://github.com/JizhiziLi/GFM), [P3M-500](https://github.com/JizhiziLi/P3M), and [RefMatte-RW-100](https://github.com/jizhiziLi/rim).
* Check lines 15–56 and 529–530 in `SDMatte/data/dataset.py` to modify the data path to your data path.
## 📦 Model Zoo
* Download the weights and configurations of **SDMatte** and **SDMatte***,** from the Hugging Face repository [LongfeiHuang/SDMatte](https://huggingface.co/LongfeiHuang/SDMatte)
* Download the weight and configuration of **LiteSDMatte** from the Hugging Face repository [LongfeiHuang/LiteSDMatte](https://huggingface.co/LongfeiHuang/LiteSDMatte)
## 🔍 Test
* Modify the `pretrained_model_name_or_path` field in `configs/SDMatte.py` to the directory path containing the configuration files, so that the model can be properly initialized.
* Modify the `CKPT_DIR` parameter in `script/test_SDMatte.sh` or `script/test_SDMatte.sh` to the specific path of the downloaded weight file.
* Run the following command to evaluate **SDMatte** or **LiteSDMatte**.
```
bash script/test_SDMatte.sh
bash script/test_LiteSDMatte.sh
```
## 📋 TodoList
- [x] Publish paper on arXiv
- [x] Release source code for SDMatte
- [x] Release evaluation codes
- [x] Release pretrained weights for SDMatte and SDMatte*
- [x] Release source code for LiteSDMatte
- [x] Release pretrained weights for LiteSDMatte
- [ ] Release training code
- [ ] Deploy interactive demo using Gradio or on Hugging Face Spaces
## 📝 License
This project is licensed under MIT. Redistribution and use should follow this license.
## ✏️ Cite Us
If you find this repository or our work useful, please consider citing us:
```
@inproceedings{huang2025sdmatte,
title={SDMatte: Grafting Diffusion Models for Interactive Matting},
author={Huang, Longfei and Liang, Yu and Zhang, Hao and Chen, Jinwei and Dong, Wei and Chen, Lunde and Liu, Wanyu and Li, Bo and Jiang, Peng-Tao},
booktitle={Proceedings of the IEEE/CVF international conference on computer vision},
year={2025}
}
```
## 👏 Acknowledgement
Our repo is built upon [Stable Diffusion 2](https://github.com/Stability-AI/stablediffusion), [TAESD](https://github.com/madebyollin/taesd), and [BK-SDM](https://github.com/Nota-NetsPresso/BK-SDM). We sincerely thank the authors for their contributions to the community.
## 📧 Contact
If you have any questions, please feel free to reach us at `2946399650fly@shu.edu.cn` or ` pt.jiang@vivo.com`.