Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/magic-research/magic-animate
[CVPR 2024] MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
https://github.com/magic-research/magic-animate
Last synced: 3 days ago
JSON representation
[CVPR 2024] MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
- Host: GitHub
- URL: https://github.com/magic-research/magic-animate
- Owner: magic-research
- License: bsd-3-clause
- Created: 2023-11-21T08:33:54.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-06-21T20:23:43.000Z (6 months ago)
- Last Synced: 2024-12-12T02:00:08.338Z (10 days ago)
- Language: Python
- Homepage: https://showlab.github.io/magicanimate/
- Size: 24.3 MB
- Stars: 10,591
- Watchers: 104
- Forks: 1,082
- Open Issues: 105
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-diffusion-categorized - [Code
- AiTreasureBox - magic-research/magic-animate - 12-20_10612_2](https://img.shields.io/github/stars/magic-research/magic-animate.svg)|MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model| (Repos)
- StarryDivineSky - magic-research/magic-animate
README
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
Zhongcong Xu
·
Jianfeng Zhang
·
Jun Hao Liew
·
Hanshu Yan
·
Jia-Wei Liu
·
Chenxu Zhang
·
Jiashi Feng
·
Mike Zheng Shou
National University of Singapore | ByteDance
## 📢 News
* **[2023.12.4]** Release inference code and gradio demo. We are working to improve MagicAnimate, stay tuned!
* **[2023.11.23]** Release MagicAnimate paper and project page.
## 🏃♂️ Getting Started
Download the pretrained base models for [StableDiffusion V1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5) and [MSE-finetuned VAE](https://huggingface.co/stabilityai/sd-vae-ft-mse).
Download our MagicAnimate [checkpoints](https://huggingface.co/zcxu-eric/MagicAnimate).
Please follow the huggingface download instructions to download the above models and checkpoints, `git lfs` is recommended.
Place the based models and checkpoints as follows:
```bash
magic-animate
|----pretrained_models
|----MagicAnimate
|----appearance_encoder
|----diffusion_pytorch_model.safetensors
|----config.json
|----densepose_controlnet
|----diffusion_pytorch_model.safetensors
|----config.json
|----temporal_attention
|----temporal_attention.ckpt
|----sd-vae-ft-mse
|----config.json
|----diffusion_pytorch_model.safetensors
|----stable-diffusion-v1-5
|----scheduler
|----scheduler_config.json
|----text_encoder
|----config.json
|----pytorch_model.bin
|----tokenizer (all)
|----unet
|----diffusion_pytorch_model.bin
|----config.json
|----v1-5-pruned-emaonly.safetensors
|----...
```
## ⚒️ Installation
prerequisites: `python>=3.8`, `CUDA>=11.3`, and `ffmpeg`.
Install with `conda`:
```bash
conda env create -f environment.yaml
conda activate manimate
```
or `pip`:
```bash
pip3 install -r requirements.txt
```
## 💃 Inference
Run inference on single GPU:
```bash
bash scripts/animate.sh
```
Run inference with multiple GPUs:
```bash
bash scripts/animate_dist.sh
```
## 🎨 Gradio Demo
#### Online Gradio Demo:
Try our [online gradio demo](https://huggingface.co/spaces/zcxu-eric/magicanimate) quickly.
#### Local Gradio Demo:
Launch local gradio demo on single GPU:
```bash
python3 -m demo.gradio_animate
```
Launch local gradio demo if you have multiple GPUs:
```bash
python3 -m demo.gradio_animate_dist
```
Then open gradio demo in local browser.
## 🙏 Acknowledgements
We would like to thank [AK(@_akhaliq)](https://twitter.com/_akhaliq?lang=en) and huggingface team for the help of setting up oneline gradio demo.
## 🎓 Citation
If you find this codebase useful for your research, please use the following entry.
```BibTeX
@inproceedings{xu2023magicanimate,
author = {Xu, Zhongcong and Zhang, Jianfeng and Liew, Jun Hao and Yan, Hanshu and Liu, Jia-Wei and Zhang, Chenxu and Feng, Jiashi and Shou, Mike Zheng},
title = {MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model},
booktitle = {arXiv},
year = {2023}
}
```