https://github.com/picsart-ai-research/zero-painter
🔥 [CVPR 2024] The official repo for Zero-Painter!
https://github.com/picsart-ai-research/zero-painter
computer-vision cvpr2024 generative-ai zero-painter
Last synced: about 1 year ago
JSON representation
🔥 [CVPR 2024] The official repo for Zero-Painter!
- Host: GitHub
- URL: https://github.com/picsart-ai-research/zero-painter
- Owner: Picsart-AI-Research
- License: mit
- Created: 2024-03-29T13:10:39.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-08T10:16:45.000Z (about 2 years ago)
- Last Synced: 2025-03-26T04:41:25.934Z (over 1 year ago)
- Topics: computer-vision, cvpr2024, generative-ai, zero-painter
- Language: Python
- Homepage: http://arxiv.org/abs/2406.04032
- Size: 3.43 MB
- Stars: 66
- Watchers: 2
- Forks: 4
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis [CVPR 2024]
This repository is the official implementation of [Zero-Painter](https://arxiv.org/abs/2406.04032).
**[Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis](https://arxiv.org/abs/2406.04032)**
Marianna Ohanyan*,
Hayk Manukyan*,
Zhangyang Wang,
Shant Navasardyan,
[Humphrey Shi](https://www.humphreyshi.com)
[Arxiv](https://arxiv.org/abs/2406.04032)
We present Zero-Painter , a novel training-free framework for layout-conditional text-to-image synthesis that facilitates the creation of detailed and controlled imagery from textual prompts. Our method utilizes object masks and individual descriptions, coupled with a global text prompt, to generate images with high fidelity. Zero-Painter employs a two-stage process involving our novel Prompt-Adjusted Cross-Attention (PACA) and Region-Grouped Cross-Attention (ReGCA) blocks, ensuring precise alignment of generated objects with textual prompts and mask shapes. Our extensive experiments demonstrate that Zero-Painter surpasses current state-of-the-art methods in preserving textual details and adhering to mask shapes.
## 🔥 News
- [2024.06.6] ZeroPainter paper and code is released.
- [2024.02.27] Paper is accepted to CVPR 2024.
## ⚒️ Installation
Install with `pip`:
```bash
pip3 install -r requirements.txt
```
## 💃 Inference: Generate images with Zero-Painter
1. Download [models](https://huggingface.co/PAIR/Zero-Painter) and put them in the `models` folder.
2. You can use the following script to perform inference on the given mask and prompts pair:
```
python zero_painter.py \
--mask-path data/masks/1_rgb.png \
--metadata data/metadata/1.json \
--output-dir data/outputs/
```
`meatadata` sould be in the following format
```
[{
"prompt": "Brown gift box beside red candle.",
"color_context_dict": {
"(244, 54, 32)": "Brown gift box",
"(54, 245, 32)": "red candle"
}
}]
```
## Method
---
## 🎓 Citation
If you use our work in your research, please cite our publication:
```
@article{Zeropainter,
title={Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis},
url={http://arxiv.org/abs/2406.04032},
publisher={arXiv},
author={Ohanyan, Marianna and Manukyan, Hayk and Wang, Zhangyang and Navasardyan, Shant and Shi, Humphrey},
year={2024}}
```