Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/magic-research/magic-animate

[CVPR 2024] MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
https://github.com/magic-research/magic-animate

Last synced: 11 days ago
JSON representation

[CVPR 2024] MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

Lists

README 

        






  
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model


  


    Zhongcong Xu

    ·

    Jianfeng Zhang

    ·

    Jun Hao Liew

    ·

    Hanshu Yan

    ·

    Jia-Wei Liu

    ·

    Chenxu Zhang

    ·

    Jiashi Feng

    ·

    Mike Zheng Shou

    


    


        Paper PDF

        Project Page

        

    


    National University of Singapore   |    ByteDance

  


  

  

    

    

      

    

    

      

    

    

  



## 📢 News

* **[2023.12.4]** Release inference code and gradio demo. We are working to improve MagicAnimate, stay tuned!

* **[2023.11.23]** Release MagicAnimate paper and project page.



## 🏃‍♂️ Getting Started

Download the pretrained base models for [StableDiffusion V1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5) and [MSE-finetuned VAE](https://huggingface.co/stabilityai/sd-vae-ft-mse).



Download our MagicAnimate [checkpoints](https://huggingface.co/zcxu-eric/MagicAnimate).



Please follow the huggingface download instructions to download the above models and checkpoints, `git lfs` is recommended.



Place the based models and checkpoints as follows:

```bash

magic-animate

|----pretrained_models

  |----MagicAnimate

    |----appearance_encoder

      |----diffusion_pytorch_model.safetensors

      |----config.json

    |----densepose_controlnet

      |----diffusion_pytorch_model.safetensors

      |----config.json

    |----temporal_attention

      |----temporal_attention.ckpt

  |----sd-vae-ft-mse

    |----config.json

    |----diffusion_pytorch_model.safetensors

  |----stable-diffusion-v1-5

    |----scheduler

       |----scheduler_config.json

    |----text_encoder

       |----config.json

       |----pytorch_model.bin

    |----tokenizer (all)

    |----unet

       |----diffusion_pytorch_model.bin

       |----config.json

    |----v1-5-pruned-emaonly.safetensors

|----...

```



## ⚒️ Installation

prerequisites: `python>=3.8`, `CUDA>=11.3`, and `ffmpeg`.



Install with `conda`: 

```bash

conda env create -f environment.yaml

conda activate manimate

```

or `pip`:

```bash

pip3 install -r requirements.txt

```



## 💃 Inference

Run inference on single GPU:

```bash

bash scripts/animate.sh

```

Run inference with multiple GPUs:

```bash

bash scripts/animate_dist.sh

```



## 🎨 Gradio Demo 



#### Online Gradio Demo:

Try our [online gradio demo](https://huggingface.co/spaces/zcxu-eric/magicanimate) quickly.



#### Local Gradio Demo:

Launch local gradio demo on single GPU:

```bash

python3 -m demo.gradio_animate

```

Launch local gradio demo if you have multiple GPUs:

```bash

python3 -m demo.gradio_animate_dist

```

Then open gradio demo in local browser.



## 🙏 Acknowledgements

We would like to thank [AK(@_akhaliq)](https://twitter.com/_akhaliq?lang=en) and huggingface team for the help of setting up oneline gradio demo.



## 🎓 Citation

If you find this codebase useful for your research, please use the following entry.

```BibTeX

@inproceedings{xu2023magicanimate,

    author    = {Xu, Zhongcong and Zhang, Jianfeng and Liew, Jun Hao and Yan, Hanshu and Liu, Jia-Wei and Zhang, Chenxu and Feng, Jiashi and Shou, Mike Zheng},

    title     = {MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model},

    booktitle = {arXiv},

    year      = {2023}

}

```