Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/okotaku/sdxlengine
Stable Diffusion XL Diffusers training with mmengine
https://github.com/okotaku/sdxlengine
Last synced: 24 days ago
JSON representation
Stable Diffusion XL Diffusers training with mmengine
- Host: GitHub
- URL: https://github.com/okotaku/sdxlengine
- Owner: okotaku
- License: apache-2.0
- Created: 2024-02-26T08:33:32.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2024-03-15T23:58:07.000Z (8 months ago)
- Last Synced: 2024-03-16T00:39:10.639Z (8 months ago)
- Language: Python
- Size: 214 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# DiffEngine
[](https://github.com/okotaku/sdxlengine/actions/workflows/build.yml)
[](https://sdxlengine.readthedocs.io/en/latest/)
[](https://github.com/okotaku/sdxlengine/blob/main/LICENSE)
[](https://github.com/okotaku/sdxlengine/issues)
[](https://github.com/astral-sh/ruff)
[](https://mypy-lang.org/)
[📘 Documentation](https://sdxlengine0.readthedocs.io/en/latest/) |
[🤔 Reporting Issues](https://github.com/okotaku/sdxlengine/issues/new/choose)
## 📄 Table of Contents
- [DiffEngine](#diffengine)
- [📄 Table of Contents](#-table-of-contents)
- [📖 Introduction 🔝](#-introduction-)
- [🛠️ Installation 🔝](#️-installation-)
- [🐳 Docker](#-docker)
- [📦 Devcontainer](#-devcontainer)
- [👨🏫 Get Started 🔝](#-get-started-)
- [📘 Documentation 🔝](#-documentation-)
- [📊 Model Zoo 🔝](#-model-zoo-)
- [🙌 Contributing 🔝](#-contributing-)
- [🎫 License 🔝](#-license-)
- [🖊️ Citation 🔝](#️-citation-)
- [🤝 Acknowledgement 🔝](#-acknowledgement-)
## 📖 Introduction [🔝](#-table-of-contents)
DiffEngine is an open-source toolbox designed for training state-of-the-art Diffusion Models. Packed with advanced features including stable-fast, diffusers and MMEngine, DiffEngine empowers those seasoned experts and newcomers in the field to efficiently create and enhance diffusion models. Stay at the forefront of innovation with our cutting-edge platform, accelerating your journey in Diffusion Models training.
1. **Training state-of-the-art Diffusion Models**: Empower your projects with state-of-the-art Diffusion Models. Explore options like Stable Diffusion XL, DreamBooth, and LoRA.
2. **Unified Config System and Module Designs**: Thanks to MMEngine, our platform boasts a unified configuration system and modular designs. Easily customize hyperparameters, loss functions, and other crucial settings while maintaining a structured and organized project environment.
3. **Inference with diffusers.pipeline**: Seamlessly transition from training to real-world application. Effortlessly deploy your trained Diffusion Models for inference tasks, enabling quick and efficient decision-making based on model insights.
4. **Optimized training speed**: Our platform is designed to accelerate training speed. We utilize the Apex, Nvidia NGC Container, `torch.compile`. You can achieve high-quality results in less time, accelerating your project timeline and enhancing your productivity.
## 🛠️ Installation [🔝](#-table-of-contents)
#### 🐳 Docker
Below are the quick steps for installing and running dreambooth training using Docker:
```bash
git clone https://github.com/okotaku/template
cd sdxlengine
docker compose up -d
docker compose exec template diffengine train stable_diffusion_xl_dreambooth_lora_dog
```
#### 📦 Devcontainer
You can also utilize the devcontainer to develop the DiffEngine. The devcontainer is a pre-configured development environment that runs in a Docker container. It includes all the necessary tools and dependencies for developing, building, and testing the DiffEngine.
1. Clone repository:
```
git clone https://github.com/okotaku/template
```
2. Open the cloned repository in Visual Studio Code.
3. Click on the "Reopen in Container" button located in the bottom right corner of the window. This action will open the repository within a devcontainer.
4. Run the following command to start training with the selected config:
```bash
diffengine train stable_diffusion_xl_dreambooth_lora_dog
```
## 👨🏫 Get Started [🔝](#-table-of-contents)
DiffEngine makes training easy through its pre-defined configs. These configs provide a streamlined way to start your training process. Here's how you can get started using one of the pre-defined configs:
1. **Choose a config**: You can find various pre-defined configs in the [`configs`](diffengine/configs/) directory of the DiffEngine repository. For example, if you wish to train a DreamBooth model using the Stable Diffusion algorithm, you can use the [`configs/dreambooth/stable_diffusion_xl_dreambooth_lora_dog.py`](diffengine/configs/dreambooth/stable_diffusion_xl_dreambooth_lora_dog.py).
2. **Start Training**: Open a terminal and run the following command to start training with the selected config:
```bash
diffengine train stable_diffusion_xl_dreambooth_lora_dog
```
3. **Monitor Progress and get results**: The training process will begin, and you can track its progress. The outputs of the training will be located in the `work_dirs/stable_diffusion_xl_dreambooth_lora_dog` directory, specifically when using the `stable_diffusion_xl_dreambooth_lora_dog` config.
```
work_dirs/stable_diffusion_xl_dreambooth_lora_dog
├── 20230802_033741
| ├── 20230802_033741.log # log file
| └── vis_data
| ├── 20230802_033741.json # log json file
| ├── config.py # config file for each experiment
| └── vis_image # visualized image from each step
├── step999/unet
| ├── adapter_config.json # adapter conrfig file
| └── adapter_model.bin # weight for inferencing with diffusers.pipeline
├── iter_1000.pth # checkpoint from each step
├── last_checkpoint # last checkpoint, it can be used for resuming
└── stable_diffusion_xl_dreambooth_lora_dog.py # latest config file
```
An illustrative output example is provided below:
4. **Inference with diffusers.pipeline**: Once you have trained a model, simply specify the path to the saved model and inference by the `diffusers.pipeline` module.
```py
from pathlib import Path
import torch
from diffusers import DiffusionPipeline, AutoencoderKL
from peft import PeftModel
checkpoint = Path('work_dirs/stable_diffusion_xl_dreambooth_lora_dog/step999')
prompt = 'A photo of sks dog in a bucket'
vae = AutoencoderKL.from_pretrained(
'madebyollin/sdxl-vae-fp16-fix',
torch_dtype=torch.float16,
)
pipe = DiffusionPipeline.from_pretrained(
'stabilityai/stable-diffusion-xl-base-1.0', vae=vae, torch_dtype=torch.float16)
pipe.to('cuda')
pipe.unet = PeftModel.from_pretrained(pipe.unet, checkpoint / "unet", adapter_name="default")
image = pipe(
prompt,
num_inference_steps=50,
height=1024,
width=1024,
).images[0]
image.save('demo.png')
```
## 📘 Documentation [🔝](#-table-of-contents)
For detailed user guides and advanced guides, please refer to our [Documentation](https://sdxlengine.readthedocs.io/en/latest/):
- [Get Started](https://sdxlengine.readthedocs.io/en/latest/get_started.html) for get started.
Run Guides
- [Run Stable Diffusion](https://sdxlengine.readthedocs.io/en/latest/run_guides/run.html)
- [Run DreamBooth](https://sdxlengine.readthedocs.io/en/latest/run_guides/run_dreambooth.html)
- [Run LoRA](https://sdxlengine.readthedocs.io/en/latest/run_guides/run_lora.html)
- [Run ControlNet](https://sdxlengine.readthedocs.io/en/latest/run_guides/run_controlnet.html)
- [Run Inpaint](https://sdxlengine.readthedocs.io/en/latest/run_guides/run_inpaint.html)
User Guides
- [Learn About Config](https://sdxlengine.readthedocs.io/en/latest/user_guides/config.html)
- [Prepare Dataset](https://sdxlengine.readthedocs.io/en/latest/user_guides/dataset_prepare.html)
## 📊 Model Zoo [🔝](#-table-of-contents)
Supported algorithms
Stable Diffusion XLs
Others
- Min-SNR Loss (ICCV'2023)
- DeBias Estimation Loss (2023)
- Offset Noise (2023)
- Pyramid Noise (2023)
- Input Perturbation (2023)
- Time Steps Bias (2023)
- V Prediction (ICLR'2022)
## 🙌 Contributing [🔝](#-table-of-contents)
We appreciate all contributions to improve clshub. Please refer to [CONTRIBUTING.md](https://github.com/open-mmlab/mmpretrain/blob/main/CONTRIBUTING.md) for the contributing guideline.
## 🎫 License [🔝](#-table-of-contents)
This project is released under the [Apache 2.0 license](LICENSE).
## 🖊️ Citation [🔝](#-table-of-contents)
If DiffEngine is helpful to your research, please cite it as below.
```
@misc{diffengine2023,
title = {{DiffEngine}: diffusers training toolbox with mmengine},
author = {{DiffEngine Contributors}},
howpublished = {\url{https://github.com/okotaku/diffengine}},
year = {2023}
}
```
## 🤝 Acknowledgement [🔝](#-table-of-contents)
This repo borrows the architecture design and part of the code from [mmengine](https://github.com/open-mmlab/mmengine) and [diffusers](https://github.com/huggingface/diffusers).
Also, please check the following openmmlab and huggingface projects and the corresponding Documentation.
- [OpenMMLab](https://openmmlab.com/)
- [HuggingFace](https://huggingface.co/)
```
@article{mmengine2022,
title = {{MMEngine}: OpenMMLab Foundational Library for Training Deep Learning Models},
author = {MMEngine Contributors},
howpublished = {\url{https://github.com/open-mmlab/mmengine}},
year={2022}
}
```
```
@misc{von-platen-etal-2022-diffusers,
author = {Patrick von Platen and Suraj Patil and Anton Lozhkov and Pedro Cuenca and Nathan Lambert and Kashif Rasul and Mishig Davaadorj and Thomas Wolf},
title = {Diffusers: State-of-the-art diffusion models},
year = {2022},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/huggingface/diffusers}}
}
```
```
@Misc{peft,
title = {PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods},
author = {Sourab Mangrulkar and Sylvain Gugger and Lysandre Debut and Younes Belkada and Sayak Paul and Benjamin Bossan},
howpublished = {\url{https://github.com/huggingface/peft}},
year = {2022}
}
```