https://github.com/genforce/freecontrol
Official implementation of CVPR 2024 paper: "FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition"
https://github.com/genforce/freecontrol
Last synced: about 2 months ago
JSON representation
Official implementation of CVPR 2024 paper: "FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition"
- Host: GitHub
- URL: https://github.com/genforce/freecontrol
- Owner: genforce
- Created: 2023-11-27T23:42:07.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-10-21T08:45:56.000Z (7 months ago)
- Last Synced: 2024-10-21T12:11:50.269Z (7 months ago)
- Language: Python
- Homepage:
- Size: 34.4 MB
- Stars: 434
- Watchers: 26
- Forks: 14
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-diffusion-categorized - [Code
README
> ## FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition
> ### [[Paper]](https://arxiv.org/abs/2312.07536) [[Project Page]](https://genforce.github.io/freecontrol/)
> [Sicheng Mo](https://sichengmo.github.io/)1*, [Fangzhou Mu](https://pages.cs.wisc.edu/~fmu/)2*,
> [Kuan Heng Lin](https://kuanhenglin.github.io)1, [Yanli Liu](https://scholar.google.ca/citations?user=YzXIxCwAAAAJ&hl=en)3,
> Bochen Guan3, [Yin Li](https://www.biostat.wisc.edu/~yli/)2, [Bolei Zhou](https://boleizhou.github.io/)1
> 1 UCLA, 2 University of Wisconsin-Madison, 3 Innopeak Technology, Inc
> * Equal contribution
> Computer Vision and Pattern Recognition (CVPR), 2024
## Overview
This is the official implementation of FreeControl, a Generative AI algorithm for controllable text-to-image generation using pre-trained Diffusion Models.
## Changelog
* 10/21/2024: Added SDXL pipeline (thanks to @shirleyzhu233).
* 02/19/2024: Initial code release. The paper is accepted to CVPR 2024.
## Getting Started
**Environment Setup**
- We provide a [conda env file](environment.yml) for environment setup.
```bash
conda env create -f environment.yml
conda activate freecontrol
pip install -U diffusers
pip install -U gradio
```
**Sample Semantic Bases**
- We provide three sample scripts in the [scripts](scripts) folder (one for each base model) to showcase how to compute target semantic bases.
- You may also download pre-computed bases from [google drive](https://drive.google.com/file/d/1o1BcIBANukeJ2pCG064-eNH9hbQoB24Z/view?usp=sharing). Put them in the [dataset](dataset) folder and launch the gradio demo.
**Gradio demo**
- We provide a graphical user interface (GUI) for users to try out FreeControl. Run the following command to start the demo.
```python
python gradio_app.py
```
## Galley:
We are building a gallery of images generated with FreeControl. You are welcome to share your generated images with us.
## Contact
[Sicheng Mo](https://sichengmo.github.io/) ([email protected])
## Reference
```
@article{mo2023freecontrol,
title={FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition},
author={Mo, Sicheng and Mu, Fangzhou and Lin, Kuan Heng and Liu, Yanli and Guan, Bochen and Li, Yin and Zhou, Bolei},
journal={arXiv preprint arXiv:2312.07536},
year={2023}
}
```