Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/radames/real-time-latent-consistency-model
App showcasing multiple real-time diffusion models pipelines with Diffusers
https://github.com/radames/real-time-latent-consistency-model
diffusers diffusion-models latent-consistency-model machine-learning mjpeg mjpeg-stream real-time stable-diffusion
Last synced: 4 days ago
JSON representation
App showcasing multiple real-time diffusion models pipelines with Diffusers
- Host: GitHub
- URL: https://github.com/radames/real-time-latent-consistency-model
- Owner: radames
- License: apache-2.0
- Created: 2023-10-28T21:52:18.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-06-21T06:00:33.000Z (5 months ago)
- Last Synced: 2024-11-09T21:02:09.649Z (11 days ago)
- Topics: diffusers, diffusion-models, latent-consistency-model, machine-learning, mjpeg, mjpeg-stream, real-time, stable-diffusion
- Language: Python
- Homepage: https://huggingface.co/spaces/radames/Real-Time-Latent-Consistency-Model
- Size: 306 KB
- Stars: 872
- Watchers: 18
- Forks: 102
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
---
title: Real-Time Latent Consistency Model Image-to-Image ControlNet
emoji: 🖼️🖼️
colorFrom: gray
colorTo: indigo
sdk: docker
pinned: false
suggested_hardware: a10g-small
disable_embedding: true
---
# Real-Time Latent Consistency Model
This demo showcases [Latent Consistency Model (LCM)](https://latent-consistency-models.github.io/) using [Diffusers](https://huggingface.co/docs/diffusers/using-diffusers/lcm) with a MJPEG stream server. You can read more about LCM + LoRAs with diffusers [here](https://huggingface.co/blog/lcm_lora).
You need a webcam to run this demo. 🤗
See a collecting with live demos [here](https://huggingface.co/collections/latent-consistency/latent-consistency-model-demos-654e90c52adb0688a0acbe6f)
## Running Locally
You need CUDA and Python 3.10, Node > 19, Mac with an M1/M2/M3 chip or Intel Arc GPU
## Install
```bash
python -m venv venv
source venv/bin/activate
pip3 install -r server/requirements.txt
cd frontend && npm install && npm run build && cd ..
python server/main.py --reload --pipeline img2imgSDTurbo
```
Don't forget to fuild the frontend!!!
```bash
cd frontend && npm install && npm run build && cd ..
```
# Pipelines
You can build your own pipeline following examples here [here](pipelines),
# LCM
### Image to Image
```bash
python server/main.py --reload --pipeline img2img
```
# LCM
### Text to Image
```bash
python server/main.py --reload --pipeline txt2img
```
### Image to Image ControlNet Canny
```bash
python server/main.py --reload --pipeline controlnet
```
# LCM + LoRa
Using LCM-LoRA, giving it the super power of doing inference in as little as 4 steps. [Learn more here](https://huggingface.co/blog/lcm_lora) or [technical report](https://huggingface.co/papers/2311.05556)
### Image to Image ControlNet Canny LoRa
```bash
python server/main.py --reload --pipeline controlnetLoraSD15
```
or SDXL, note that SDXL is slower than SD15 since the inference runs on 1024x1024 images
```bash
python server/main.py --reload --pipeline controlnetLoraSDXL
```
### Text to Image
```bash
python server/main.py --reload --pipeline txt2imgLora
```
```bash
python server/main.py --reload --pipeline txt2imgLoraSDXL
```
# Available Pipelines
#### [LCM](https://huggingface.co/SimianLuo/LCM_Dreamshaper_v7)
`img2img`
`txt2img`
`controlnet`
`txt2imgLora`
`controlnetLoraSD15`
#### [SD15](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)
`controlnetLoraSDXL`
`txt2imgLoraSDXL`
#### [SDXL Turbo](https://huggingface.co/stabilityai/sd-xl-turbo)
`img2imgSDXLTurbo`
`controlnetSDXLTurbo`
#### [SDTurbo](https://huggingface.co/stabilityai/sd-turbo)
`img2imgSDTurbo`
`controlnetSDTurbo`
#### [Segmind-Vega](https://huggingface.co/segmind/Segmind-Vega)
`controlnetSegmindVegaRT`
`img2imgSegmindVegaRT`
### Setting environment variables
* `--host`: Host address (default: 0.0.0.0)
* `--port`: Port number (default: 7860)
* `--reload`: Reload code on change
* `--max-queue-size`: Maximum queue size (optional)
* `--timeout`: Timeout period (optional)
* `--safety-checker`: Enable Safety Checker (optional)
* `--torch-compile`: Use Torch Compile
* `--use-taesd` / `--no-taesd`: Use Tiny Autoencoder
* `--pipeline`: Pipeline to use (default: "txt2img")
* `--ssl-certfile`: SSL Certificate File (optional)
* `--ssl-keyfile`: SSL Key File (optional)
* `--debug`: Print Inference time
* `--compel`: Compel option
* `--sfast`: Enable Stable Fast
* `--onediff`: Enable OneDiff
If you run using `bash build-run.sh` you can set `PIPELINE` variables to choose the pipeline you want to run
```bash
PIPELINE=txt2imgLoraSDXL bash build-run.sh
```
and setting environment variables
```bash
TIMEOUT=120 SAFETY_CHECKER=True MAX_QUEUE_SIZE=4 python server/main.py --reload --pipeline txt2imgLoraSDXL
```
If you're running locally and want to test it on Mobile Safari, the webserver needs to be served over HTTPS, or follow this instruction on my [comment](https://github.com/radames/Real-Time-Latent-Consistency-Model/issues/17#issuecomment-1811957196)
```bash
openssl req -newkey rsa:4096 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
python server/main.py --reload --ssl-certfile=certificate.pem --ssl-keyfile=key.pem
```
## Docker
You need NVIDIA Container Toolkit for Docker, defaults to `controlnet``
```bash
docker build -t lcm-live .
docker run -ti -p 7860:7860 --gpus all lcm-live
```
reuse models data from host to avoid downloading them again, you can change `~/.cache/huggingface` to any other directory, but if you use hugingface-cli locally, you can share the same cache
```bash
docker run -ti -p 7860:7860 -e HF_HOME=/data -v ~/.cache/huggingface:/data --gpus all lcm-live
```
or with environment variables
```bash
docker run -ti -e PIPELINE=txt2imgLoraSDXL -p 7860:7860 --gpus all lcm-live
```
# Demo on Hugging Face
* [radames/Real-Time-Latent-Consistency-Model](https://huggingface.co/spaces/radames/Real-Time-Latent-Consistency-Model)
* [radames/Real-Time-SD-Turbo](https://huggingface.co/spaces/radames/Real-Time-SD-Turbo)
* [latent-consistency/Real-Time-LCM-ControlNet-Lora-SD1.5](https://huggingface.co/spaces/latent-consistency/Real-Time-LCM-ControlNet-Lora-SD1.5)
* [latent-consistency/Real-Time-LCM-Text-to-Image-Lora-SD1.5](https://huggingface.co/spaces/latent-consistency/Real-Time-LCM-Text-to-Image-Lora-SD1.5)
* [radames/Real-Time-Latent-Consistency-Model-Text-To-Image](https://huggingface.co/spaces/radames/Real-Time-Latent-Consistency-Model-Text-To-Image)
https://github.com/radames/Real-Time-Latent-Consistency-Model/assets/102277/c4003ac5-e7ff-44c0-97d3-464bb659de70