https://github.com/Liuxinyv/HiPrompt
https://github.com/Liuxinyv/HiPrompt
Last synced: about 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/Liuxinyv/HiPrompt
- Owner: Liuxinyv
- Created: 2024-09-04T10:16:28.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-28T12:38:53.000Z (8 months ago)
- Last Synced: 2025-02-28T18:56:03.477Z (8 months ago)
- Language: Python
- Size: 30.7 MB
- Stars: 25
- Watchers: 6
- Forks: 2
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-diffusion-categorized - [Code
README
# HiPrompt: Tuning-free Higher-Resolution Generation with Hierarchical MLLM Prompts
Xinyu Liu1,
Yingqing He2,
Lanqing Guo2,
Xiang Li2,
Bu Jin3,
Peng Li1,
Yan Li1,
Chi-Min Chan3,
Qifeng Chen1,
Wei Xue1,
Wenhan Luo1,
Qingfeng Liu1,
QiYike Guo1
1Hong Kong University of Science and Technology
2Nanyang Technological University
3Tsinghua University
4University of Chinese Academy of Sciences
## 🔆 Abstract
> The potential for higher-resolution image generation using pretrained diffusion models is immense, yet these models often struggle with issues of object repetition and structural artifacts especially when scaling to 4K resolution and higher. We figure out that the problem is caused by that, a single prompt for the generation of multiple scales provides insufficient efficacy. In response, we propose HiPrompt, a new tuning-free solution that tackles the above problems by introducing hierarchical prompts. The hierarchical prompts offer both global and local guidance. Specifically, the global guidance comes from the user input that describes the overall content, while the local guidance utilizes patch-wise descriptions from MLLMs to elaborately guide the regional structure and texture generation. Furthermore, during the inverse denoising process, the generated noise is decomposed into low- and high-frequency spatial components. These components are conditioned on multiple prompt levels, including detailed patch-wise descriptions and broader image-level prompts, facilitating prompt-guided denoising under hierarchical semantic guidance. It further allows the generation to focus more on local spatial regions and ensures the generated images maintain coherent local and global semantics, structures, and textures with high definition. Extensive experiments demonstrate that HiPrompt outperforms state-of-the-art works in higher-resolution image generation, significantly reducing object repetition and enhancing structural quality.
## ⚙️ Setup:
```
conda create -n HiPrompt python=3.9
conda activate HiPrompt
pip install -r requirements.txt
```
---
## 💫 Inference
Without Noise Decomposition
```
python hiprompt_llava.py \
--height 4096 \
--width 4096 \
--model_ckpt="stabilityai/stable-diffusion-xl-base-1.0" \
--validation_prompt "Astronaut on Mars During sunset." \
--llava true \
--scale true \
--cosine_scale_3 0.8 \
--guidance_scale_2 10.0 \
--logging_dir ${your-logging-dir} \
```
```
python hiprompt_share.py \
--height 4096 \
--width 4096 \
--model_ckpt="stabilityai/stable-diffusion-xl-base-1.0" \
--validation_prompt "A cute corgi on the lawn." \
--share true \
--scale true \
--cosine_scale_3 0.8 \
--guidance_scale_2 10.0 \
--logging_dir ${your-logging-dir} \
```
With Noise Decomposition
```
python hiprompt_llava.py \
--height 4096 \
--width 4096 \
--model_ckpt="stabilityai/stable-diffusion-xl-base-1.0" \
--validation_prompt "Astronaut on Mars During sunset." \
--llava true \
--scale true \
--cosine_scale_3 0.8 \
--noise_decom true \
--reduction sum \
--view_args 2.0 2.0 \
--views_type low_pass high_pass \
--guidance_scale_2 10.0 \
--logging_dir ${your-logging-dir} \
```
```
python hiprompt_share.py \
--height 4096 \
--width 4096 \
--model_ckpt="stabilityai/stable-diffusion-xl-base-1.0" \
--validation_prompt "A cute corgi on the lawn." \
--share true \
--scale true \
--cosine_scale_3 0.8 \
--noise_decom true \
--reduction sum \
--view_args 2.0 2.0 \
--views_type low_pass high_pass \
--guidance_scale_2 10.0 \
--logging_dir ${your-logging-dir} \
```
## 📝 Changelog
- __[2024.09]__: 🔥 Release paper.
- __[2025.02]__: 🔥 Release source code.
## ⏳ TODO
- [ ] Hugging Face Gradio demo
## 😉 Citation
```bib
@article{liu2024hiprompt,
title={Hiprompt: Tuning-free higher-resolution generation with hierarchical mllm prompts},
author={Liu, Xinyu and He, Yingqing and Guo, Lanqing and Li, Xiang and Jin, Bu and Li, Peng and Li, Yan and Chan, Chi-Min and Chen, Qifeng and Xue, Wei and others},
journal={arXiv preprint arXiv:2409.02919},
year={2024}
}
```