https://github.com/PaulCouairon/DiffCut

Official code for "DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut", NeurIPS 2024
https://github.com/PaulCouairon/DiffCut

computer-vision image-segmentation normalized-cuts stable-diffusion unsupervised-segmentation zero-shot-segmentation

Last synced: 3 months ago
JSON representation

Official code for "DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut", NeurIPS 2024

• Host: GitHub
• URL: https://github.com/PaulCouairon/DiffCut
• Owner: PaulCouairon
• License: mit
• Created: 2024-05-21T22:53:47.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-10-15T20:32:25.000Z (8 months ago)
• Last Synced: 2024-10-17T07:04:49.232Z (8 months ago)
• Topics: computer-vision, image-segmentation, normalized-cuts, stable-diffusion, unsupervised-segmentation, zero-shot-segmentation
• Language: Jupyter Notebook
• Homepage: https://diffcut-segmentation.github.io
• Size: 17.9 MB
• Stars: 20
• Watchers: 1
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-diffusion-categorized - [Code

README

        




DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut








NeurIPS 2024







Paul Couairon 

Mustafa Shukor 


Jean-Emmanuel Haugeard 

Matthieu Cord 

Nicolas Thome 









![main_figure.png](./assets/main_figure.png)



## Environment

```

conda create -n diffcut python=3.10

conda activate diffcut

pip install -r requirements.txt

```

For evaluation, install [detectron2](https://detectron2.readthedocs.io/en/latest/tutorials/install.html)

```

python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'

```

## Demo

Try our __DiffCut__ method by running the notebook ``diffcut.ipynb``

Visualize the __semantic coherence__ of vision encoders (SD, CLIP, DINO...) with ``semantic_coherence.ipynb``

## Evaluation

### Datasets Preparation

In the paper, we evaluate DiffCut on 6 benchmarks: PASCAL VOC (20 classes + background), PASCAL Context (59 classes + background), COCO-Object (80 classes + background), COCO-Stuff (27 classes), Cityscapes (27 classes) and ADE20k (150 classes). See [Preparing Datasets for DiffCut](datasets/README.md).

### Run Evaluation

```

python eval_diffcut.py --dataset_name Cityscapes --tau 0.5 --alpha 10 --refinement

python eval_diffcut_openvoc.py --dataset_name VOC20 --tau 0.5 --alpha 10 --refinement

```

## Citation

```

@inproceedings{

    couairon2024diffcut,

    title={DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut},

    author={Paul Couairon and Mustafa Shukor and Jean-Emmanuel HAUGEARD and Matthieu Cord and Nicolas THOME},

    booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},

    year={2024},

    url={https://openreview.net/forum?id=N0xNf9Qqmc}

}

```

## Acknowledgements

This repo relies on the following projects:

[Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion](https://github.com/google/diffseg)

[Emergent Correspondence from Image Diffusion](https://github.com/Tsingularity/dift)

[Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning](https://arxiv.org/abs/2212.04994)

[Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://github.com/bytedance/fc-clip)

[Cut and Learn for Unsupervised Image & Video Object Detection and Instance Segmentation](https://github.com/facebookresearch/CutLER)