https://github.com/bytedance/X-Portrait

Source code for the SIGGRAPH 2024 paper "X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention"
https://github.com/bytedance/X-Portrait

research

Last synced: 6 months ago
JSON representation

Source code for the SIGGRAPH 2024 paper "X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention"

• Host: GitHub
• URL: https://github.com/bytedance/X-Portrait
• Owner: bytedance
• License: apache-2.0
• Created: 2024-07-18T22:21:40.000Z (12 months ago)
• Default Branch: main
• Last Pushed: 2024-08-01T23:30:57.000Z (11 months ago)
• Last Synced: 2024-12-24T20:05:37.255Z (6 months ago)
• Topics: research
• Language: Python
• Homepage:
• Size: 9.1 MB
• Stars: 428
• Watchers: 15
• Forks: 35
• Open Issues: 11
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.txt

Awesome Lists containing this project

README

        




  
X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention

  


                You Xie,

                Hongyi Xu,

                Guoxian Song,

                Chao Wang,

                Yichun Shi,

                Linjie Luo

    


       ByteDance Inc. 

    


    


        

        

        

        

    


  

  

  

    

    

      

    

    

  


This repository contains the video generation code of SIGGRAPH 2024 paper [X-Portrait](https://arxiv.org/pdf/2403.15931). 

## Installation

Note: Python 3.9 and Cuda 11.8 are required.

```shell

bash env_install.sh

```

## Model

Please download pre-trained model from [here](https://drive.google.com/drive/folders/1Bq0n-w1VT5l99CoaVg02hFpqE5eGLo9O?usp=sharing), and save it under "checkpoint/"

## Testing

```shell

bash scripts/test_xportrait.sh

```

parameters:  

**model_config**: config file of the corresponding model  

**output_dir**: output path for generated video  

**source_image**: path of source image  

**driving_video**: path of driving video  

**best_frame**: specify the frame index in the driving video where the head pose best matches the source image (note: precision of best_frame index might affect the final quality)  

**out_frames**: number of generation frames  

**num_mix**: number of overlapping frames when applying prompt travelling during inference  

**ddim_steps**: number of inference steps (e.g., 30 steps for ddim)     

## Performance Boost

**efficiency**: Our model is compatible with LCM LoRA (https://huggingface.co/latent-consistency/lcm-lora-sdv1-5), which helps reduce the number of inference steps.  

**expressiveness**: Expressiveness of the results could be boosted if results of other face reenactment approaches, e.g., face vid2vid, could be provided via parameter "--initial_facevid2vid_results".  

## 🎓 Citation

If you find this codebase useful for your research, please use the following entry.

```BibTeX

@inproceedings{xie2024x,

  title={X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention},

  author={Xie, You and Xu, Hongyi and Song, Guoxian and Wang, Chao and Shi, Yichun and Luo, Linjie},

  journal={arXiv preprint arXiv:2403.15931},

  year={2024}

}

```