https://github.com/againstentropy/streamdiffusionio
https://github.com/againstentropy/streamdiffusionio
Last synced: 5 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/againstentropy/streamdiffusionio
- Owner: AgainstEntropy
- License: apache-2.0
- Created: 2024-02-26T04:12:46.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-10-04T20:25:25.000Z (almost 2 years ago)
- Last Synced: 2025-08-27T03:15:59.704Z (11 months ago)
- Language: Python
- Size: 40 KB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# StreamDiffusionIO
StreamDiffusionIO's pipeline design is based on [StreamDiffusion](https://github.com/cumulo-autumn/StreamDiffusion), but especially allows using different text prompt on different samples in the denoising batch respectively but consistently.
A natural application of StreamDiffusionIO is to render text streams into image streams, as in [Streaming Kanji](https://github.com/AgainstEntropy/kanji).
## Features
- [ ] Streaming with LDM
- [x] Streaming with LCM
## Installation
### Create the Env
```shell
conda create -n StreamDiffusionIO python=3.10
conda activate StreamDiffusionIO
```
### Install StreamDiffusionIO
#### For Users
```shell
ppip install StreamDiffusionIO
```
#### For Developers
```shell
git clone https://github.com/AgainstEntropy/StreamDiffusionIO.git
pip install --editable ./StreamDiffusionIO/
```
### (Optional) Accelaration with `xformers`
```shell
# For user
pip install StreamDiffusionIO[xformers]
# For dev
pip install -e '.[xformers]'
```
Or install `xformers` manually:
```shell
pip install xformers --index-url https://download.pytorch.org/whl/cu124
```
## Quick Start
StreamDiffusionIO is very similar to StreamDiffusion, but even more lightweight. One can use the pipeline with only a few lines of codes.
```python
import torch
from StreamDiffusionIO import LatentConsistencyModelStreamIO
device = "cuda" if torch.cuda.is_available() else "cpu"
model_id_or_path = "/path/to/stable-diffusion-v1-5"
lora_path = "/path/to/lora/pytorch_lora_weights.safetensors"
lcm_lora_path = "/path/to/lcm-lora/pytorch_lora_weights.safetensors"
stream = LatentConsistencyModelStreamIO(
model_id_or_path=model_id_or_path,
lcm_lora_path=lcm_lora_path,
lora_dict={lora_path: 1},
resolution=128,
device=device,
)
text = "Today I saw a beautiful sunset and it made me feel so happy."
prompt_list = text.split()
# to simulate a text stream
for prompt in prompt_list:
image, text = stream(prompt) # stream returns None during warmup
if image is not None:
print(text)
display(image)
# Continue to display the remaining images in the stream
while True:
image, text = stream(prompt)
print(text)
display(image)
if stream.stop():
break
```
Note the `text` returnded from the `stream` is the corresponding text prompt used to generating the returned `image`.
Please follow the Jupyter notebooks in [examples](./examples/) to see details.
## Acknowledgements & References
- [StreamDiffusion](https://github.com/cumulo-autumn/StreamDiffusion)
- [Latent Consistency Models](https://github.com/huggingface/diffusers/tree/main/examples/consistency_distillation)
- [Latent Diffision Models](https://github.com/CompVis/latent-diffusion/tree/main)