https://github.com/ZrrSkywalker/Personalize-SAM
Personalize Segment Anything Model (SAM) with 1 shot in 10 seconds
https://github.com/ZrrSkywalker/Personalize-SAM
Last synced: 23 days ago
JSON representation
Personalize Segment Anything Model (SAM) with 1 shot in 10 seconds
- Host: GitHub
- URL: https://github.com/ZrrSkywalker/Personalize-SAM
- Owner: ZrrSkywalker
- License: mit
- Created: 2023-05-04T18:22:19.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-07-22T15:13:43.000Z (9 months ago)
- Last Synced: 2024-10-29T15:29:22.083Z (6 months ago)
- Language: Python
- Homepage:
- Size: 65 MB
- Stars: 1,512
- Watchers: 27
- Forks: 101
- Open Issues: 38
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
- awesome-diffusion-categorized - [Code
- awesome-segment-anything-extensions - Repo
- Awesome-Segment-Anything - [code
README
# Personalize Segment Anything with 1 Shot in 10 Seconds
[](https://paperswithcode.com/sota/personalized-segmentation-on-perseg?p=personalize-segment-anything-model-with-one)
Official implementation of ['Personalize Segment Anything Model with One Shot'](https://arxiv.org/pdf/2305.03048.pdf).
💥 Try out the [web demo](https://huggingface.co/spaces/justin-zk/Personalize-SAM) 🤗 of PerSAM and PerSAM-F: [](https://huggingface.co/spaces/justin-zk/Personalize-SAM)
🎉 Try out the [tutorial notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/PerSAM) in colab for your own dataset. Great thanks to [@NielsRogge](https://github.com/NielsRogge)!
🎆 Try out the online web demo of PerSAM in OpenXLab :
[](https://openxlab.org.cn/apps/detail/RenRuiZhang/Personalize-SAM)
## News
* Support [MobileSAM](https://github.com/ChaoningZhang/MobileSAM) 🔥 with significant efficiency improvement. Thanks for their wonderful work!
* **TODO**: Release the PerSAM-assisted [Dreambooth](https://arxiv.org/pdf/2208.12242.pdf) for better fine-tuning [Stable Diffusion](https://github.com/CompVis/stable-diffusion) 📌.
* We release the code of PerSAM and PerSAM-F 🔥. Check our [video](https://www.youtube.com/watch?v=QlunvXpYQXM) here!
* We release a new dataset for personalized segmentation, [PerSeg](https://drive.google.com/file/d/18TbrwhZtAPY5dlaoEqkPa5h08G9Rjcio/view?usp=sharing) 🔥.
## Introduction
*How to customize SAM to automatically segment your pet dog in a photo album?*
In this project, we propose a training-free **Per**sonalization approach for [Segment Anything Model (SAM)](https://ai.facebook.com/research/publications/segment-anything/), termed as **PerSAM**. Given only a single image with a reference mask, PerSAM can segment specific visual concepts, e.g., your pet dog, within other images or videos without any training.
For better performance, we further present an efficient one-shot fine-tuning variant, **PerSAM-F**. We freeze the entire SAM and introduce two learnable mask weights, which only trains **2 parameters** within **10 seconds**.
Besides, our approach can be utilized to assist [DreamBooth](https://arxiv.org/pdf/2208.12242.pdf) in fine-tuning better [Stable Diffusion](https://github.com/CompVis/stable-diffusion) for personalized image synthesis. We adopt PerSAM to segment the target object in the user-provided few-shot images, which eliminates the **background disturbance** and benefits the target representation learning.
## Requirements
### Installation
Clone the repo and create a conda environment:
```bash
git clone https://github.com/ZrrSkywalker/Personalize-SAM.git
cd Personalize-SAM
conda create -n persam python=3.8
conda activate persam
pip install -r requirements.txt
```
Similar to Segment Anything, our code requires `pytorch>=1.7` and `torchvision>=0.8`. Please follow the instructions [here](https://pytorch.org/get-started/locally/) to install both PyTorch and TorchVision dependencies.
### Preparation
Please download our constructed dataset **PerSeg** for personalized segmentation from [Google Drive](https://drive.google.com/file/d/18TbrwhZtAPY5dlaoEqkPa5h08G9Rjcio/view?usp=sharing) or [Baidu Yun](https://pan.baidu.com/s/1X-czD-FYW0ELlk2x90eTLg) (code `222k`), and the pre-trained weights of SAM from [here](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth). Then, unzip the dataset file and organize them as
```
data/
|–– Annotations/
|–– Images/
sam_vit_h_4b8939.pth
```
Please download 480p [TrainVal](https://data.vision.ee.ethz.ch/csergi/share/davis/DAVIS-2017-trainval-480p.zip) split of DAVIS 2017. Then decompress the file to `DAVIS/2017` and organize them as
```
DAVIS/
|––2017/
|–– Annotations/
|–– ImageSets/
|–– JPEGImages/
```
## Getting Started
### Personalized Segmentation
For the training-free 🧊 **PerSAM**, just run:
```bash
python persam.py --outdir
```
For 10-second fine-tuning of 🚀 **PerSAM-F**, just run:
```bash
python persam_f.py --outdir
```
For [MobileSAM](https://github.com/ChaoningZhang/MobileSAM) with higher efficiency, just add `--sam_type vit_t`:
```bash
python persam.py/persam_f.py --outdir --sam_type vit_t
```
For **Multi-Object** segmentation of the same category by PerSAM-F (Great thanks to [@mlzoo](https://github.com/mlzoo)), just run:
```bash
python persam_f_multi_obj.py --sam_type --outdir
```
After running, the output masks and visualizations will be stored at `outputs/`.
### Evaluation
Then, for mIoU evaluation, please run:
```bash
python eval_miou.py --pred_path
```
### Personalized Segmentation On Video
For the training-free and evaluation of 🧊 **PerSAM** on video, just run:
```bash
python persam_video.py --output_path
```
For 10-second fine-tuning and evaluation of 🚀 **PerSAM-F** on video, just run:
```bash
python persam_video_f.py --output_path
```
### Personalized Stable Diffusion
Our approach can enhance DreamBooth to better personalize Stable Diffusion for text-to-image generation.
Coming soon.
## Citation
```bash
@article{zhang2023personalize,
title={Personalize Segment Anything Model with One Shot},
author={Zhang, Renrui and Jiang, Zhengkai and Guo, Ziyu and Yan, Shilin and Pan, Junting and Dong, Hao and Gao, Peng and Li, Hongsheng},
journal={arXiv preprint arXiv:2305.03048},
year={2023}
}
```
## Acknowledgement
This repo benefits from [Segment Anything](https://github.com/facebookresearch/segment-anything) and [DreamBooth](https://github.com/XavierXiao/Dreambooth-Stable-Diffusion). Thanks for their wonderful works.
## Contact
If you have any question about this project, please feel free to contact [email protected].