https://github.com/shiimizu/ComfyUI-TiledDiffusion
Tiled Diffusion, MultiDiffusion, Mixture of Diffusers, and optimized VAE
https://github.com/shiimizu/ComfyUI-TiledDiffusion
comfyui image-generation large-image multidiffusion stable-diffusion vramsaving
Last synced: 5 months ago
JSON representation
Tiled Diffusion, MultiDiffusion, Mixture of Diffusers, and optimized VAE
- Host: GitHub
- URL: https://github.com/shiimizu/ComfyUI-TiledDiffusion
- Owner: shiimizu
- Created: 2024-01-07T10:57:58.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-12-01T21:53:41.000Z (6 months ago)
- Last Synced: 2024-12-01T22:32:36.764Z (6 months ago)
- Topics: comfyui, image-generation, large-image, multidiffusion, stable-diffusion, vramsaving
- Language: Python
- Homepage:
- Size: 101 KB
- Stars: 331
- Watchers: 9
- Forks: 23
- Open Issues: 27
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-comfyui - **Tiled Diffusion & VAE for ComfyUI**
README
# Tiled Diffusion & VAE for ComfyUI
Check out the [SD-WebUI extension](https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111/) for more information.
This extension enables **large image drawing & upscaling with limited VRAM** via the following techniques:
- Reproduced SOTA Tiled Diffusion methods
- [MultiDiffusion](https://github.com/omerbt/MultiDiffusion) 
- [Mixture of Diffusers](https://github.com/albarji/mixture-of-diffusers)
- pkuliyi2015 & Kahsolt's Tiled VAE algorithm
- ~~pkuliyi2015 & Kahsolt's TIled Noise Inversion method~~
> [!NOTE]
> Sizes/dimensions are in pixels and then converted to latent-space sizes.
## Features
- [x] Supported models
- [x] SD1.x, SD2.x, SDXL, SD3
- [x] FLUX
- [x] ControlNet support
- [ ] ~~StableSR support~~
- [ ] ~~Tiled Noise Inversion~~
- [x] Tiled VAE
- [ ] Regional Prompt Control
- [x] Img2img upscale
- [x] Ultra-Large image generation
## Tiled Diffusion
> [!TIP]
> * Set `tile_overlap` to 0 and `denoise` to 1 to see the tile seams and then adjust the options to your needs.
> * Increase `tile_batch_size` to increase speed (if your machine can handle it).
> * Use the [colorfix node](https://github.com/gameltb/Comfyui-StableSR) if your colors look off.
### Options
| Name | Description |
|-------------------|--------------------------------------------------------------|
| `method` | Tiling [strategy](https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111/blob/fbb24736c9bc374c7f098f82b575fcd14a73936a/scripts/tilediffusion.py#L39-L46). |
| `tile_width` | Tile's width |
| `tile_height` | Tile's height |
| `tile_overlap` | Tile's overlap |
| `tile_batch_size` | The number of tiles to process in a batch |
### How can I specify the tiles' arrangement?
If you have the [Math Expression](https://github.com/pythongosssss/ComfyUI-Custom-Scripts#math-expression) node (or something similar), you can use that to pass in the latent that's passed in your KSampler and divide the `tile_height`/`tile_width` by the number of rows/columns you want.
`C` = number of columns you want
`R` = number of rows you want
`pixel width of input image or latent // C` = `tile_width`
`pixel height of input image or latent // R` = `tile_height`
### SpotDiffusion
[Paper](https://arxiv.org/abs/2407.15507)
A tiling algorithm that attempts to eliminate seams by randomly shifting the denoise window per timestep. It is mainly used for fast inferences by setting `tile_overlap` to 0; otherwise, it's better to stick with the other tiling strategies as they produce better outputs.
This additional feature is experimental, in testing, and subject to change.
## Tiled VAE
The recommended tile sizes are given upon the creation of the node based on the available VRAM.
> [!NOTE]
> Enabling `fast` for the decoder may produce images with slightly higher contrast and brightness.
### Options
| Name | Description |
|-------------|----------------------------------------------------------------------------------------------------------------------------------------------|
| `tile_size` |
The image is split into tiles, which are then padded with 11/32 pixels' in the decoder/encoder.
|
| `fast` |
When Fast Mode is disabled:
- The original VAE forward is decomposed into a task queue and a task worker, which starts to process each tile.
- When GroupNorm is needed, it suspends, stores current GroupNorm mean and var, send everything to RAM, and turns to the next tile.
- After all GroupNorm means and vars are summarized, it applies group norm to tiles and continues.
- A zigzag execution order is used to reduce unnecessary data transfer.
When Fast Mode is enabled:
- The original input is downsampled and passed to a separate task queue.
- Its group norm parameters are recorded and used by all tiles' task queues.
- Each tile is separately processed without any RAM-VRAM data transfer.
After all tiles are processed, tiles are written to a result buffer and returned.
|
| `color_fix` | Only estimate GroupNorm before downsampling, i.e., run in a semi-fast mode.
Only for the encoder. Can restore colors if tiles are too small.
|
## Workflows
The following images can be loaded in ComfyUI.
Simple upscale.
4x upscale. 3 passes.
## License
Great thanks to all the contributors! 🎉🎉🎉
The implementation of MultiDiffusion, Mixture of Diffusers, and Tiled VAE code is currently under [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-nc-sa/4.0/) since it was borrowed from the wonderful [SD-WebUI extension](https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111/). Anything else GPLv3.
## Citation
```bibtex
@article{jimenez2023mixtureofdiffusers,
title={Mixture of Diffusers for scene composition and high resolution image generation},
author={Álvaro Barbero Jiménez},
journal={arXiv preprint arXiv:2302.02412},
year={2023}
}
```
```bibtex
@article{bar2023multidiffusion,
title={MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation},
author={Bar-Tal, Omer and Yariv, Lior and Lipman, Yaron and Dekel, Tali},
journal={arXiv preprint arXiv:2302.08113},
year={2023}
}
```