Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/Sainzerjj/Free-Guidance-Diffusion
A novel method that provides greater control over generated images by guiding the internal representations of the pre-trained Stable Diffusion.
https://github.com/Sainzerjj/Free-Guidance-Diffusion
Last synced: 2 months ago
JSON representation
A novel method that provides greater control over generated images by guiding the internal representations of the pre-trained Stable Diffusion.
- Host: GitHub
- URL: https://github.com/Sainzerjj/Free-Guidance-Diffusion
- Owner: Sainzerjj
- License: mit
- Created: 2023-10-11T16:05:53.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2024-02-03T10:27:13.000Z (11 months ago)
- Last Synced: 2024-08-01T18:35:05.503Z (5 months ago)
- Language: Jupyter Notebook
- Size: 8.7 MB
- Stars: 30
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-diffusion-categorized - [Code
README
# Free-Guidance-Diffusion
A method that provides greater control over generated images by guiding the internal representations of the pre-trained Stable Diffusion (SDv1.5). This idea is inspired by the paper [Diffusion Self-Guidance for Controllable Image Generation](https://arxiv.org/pdf/2306.00986.pdf) (NIPS 2023).
The method allows for various modifications, including changing the position or size of specific objects, combining the appearance of an object from one image with the layout of another image, and merging objects from multiple images into a single image.
The core implementation is **StableDiffusionFreeGuidancePipeline** written based on [🧨diffusers](https://huggingface.co/docs/diffusers/index), which is defined in `free_guidance.py`. The class inherits from the [StableDiffusionAttendAndExcitePipeline](https://huggingface.co/spaces/AttendAndExcite/Attend-and-Excite) and can be easily used. The file `experiments.ipynb` provides some visualization attempts as a reference for improvement. All guidance functions are located in `./utils/guidance_function.py`. All visualization methods are defined in `./utils/vis_utils.py`.
The biggest challenege is the guidance weights are very sensitive and the method performs worse as the prompts get more complex — subjects of the image interact, and it becomes harder to isolate the attention of specific tokens.
Â