Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/yoyo-nb/thin-plate-spline-motion-model
[CVPR 2022] Thin-Plate Spline Motion Model for Image Animation.
https://github.com/yoyo-nb/thin-plate-spline-motion-model
deep-learning deepfake face-animation face-reenactment image-animation motion-transfer pose-transfer talking-head
Last synced: about 2 hours ago
JSON representation
[CVPR 2022] Thin-Plate Spline Motion Model for Image Animation.
- Host: GitHub
- URL: https://github.com/yoyo-nb/thin-plate-spline-motion-model
- Owner: yoyo-nb
- License: mit
- Created: 2021-11-24T16:10:46.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-02-10T23:29:11.000Z (12 months ago)
- Last Synced: 2025-01-17T07:01:58.974Z (7 days ago)
- Topics: deep-learning, deepfake, face-animation, face-reenactment, image-animation, motion-transfer, pose-transfer, talking-head
- Language: Jupyter Notebook
- Homepage:
- Size: 31.6 MB
- Stars: 3,510
- Watchers: 63
- Forks: 557
- Open Issues: 85
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# [CVPR2022] Thin-Plate Spline Motion Model for Image Animation
[](LICENSE)
Source code of the CVPR'2022 paper "Thin-Plate Spline Motion Model for Image Animation"
[**Paper**](https://arxiv.org/abs/2203.14367) **|** [**Supp**](https://cloud.tsinghua.edu.cn/f/f7b8573bb5b04583949f/?dl=1)
### Example animation
**PS**: The paper trains the model for 100 epochs for a fair comparison. You can use more data and train for more epochs to get better performance.
### Web demo for animation
- Integrated into [Huggingface Spaces 🤗](https://huggingface.co/spaces) using [Gradio](https://github.com/gradio-app/gradio). Try out the Web Demo: [](https://huggingface.co/spaces/CVPR/Image-Animation-using-Thin-Plate-Spline-Motion-Model)
- Try the web demo for animation here: [](https://replicate.com/yoyo-nb/thin-plate-spline-motion-model)
- Google Colab: [](https://colab.research.google.com/drive/1DREfdpnaBhqISg0fuQlAAIwyGVn1loH_?usp=sharing)
### Pre-trained models
- ~~[Tsinghua Cloud](https://cloud.tsinghua.edu.cn/d/30ab8765da364fefa101/)~~
- [Yandex](https://disk.yandex.com/d/bWopgbGj1ZUV1w)
- [Google Drive](https://drive.google.com/drive/folders/1pNDo1ODQIb5HVObRtCmubqJikmR7VVLT?usp=sharing)
- [Baidu Yun](https://pan.baidu.com/s/1hnXmDpIbRC6WqE3tF9c5QA?pwd=1234)
### Installation
We support ```python3```.(Recommended version is Python 3.9).
To install the dependencies run:
```bash
pip install -r requirements.txt
```
### YAML configs
There are several configuration files one for each `dataset` in the `config` folder named as ```config/dataset_name.yaml```.
See description of the parameters in the ```config/taichi-256.yaml```.
### Datasets
1) **MGif**. Follow [Monkey-Net](https://github.com/AliaksandrSiarohin/monkey-net).
2) **TaiChiHD** and **VoxCeleb**. Follow instructions from [video-preprocessing](https://github.com/AliaksandrSiarohin/video-preprocessing).
3) **TED-talks**. Follow instructions from [MRAA](https://github.com/snap-research/articulated-animation).
Here are **VoxCeleb**, **TaiChiHD** and **TED-talks** pre-processed datasets used in the paper. [Baidu Yun](https://pan.baidu.com/s/1HKJOtXBIiP_tlLiFbzn3oA?pwd=x7xv)
Download all files under the folder, then merge the files and decompress, for example:
```bash
cat vox.tar.* > vox.tar
tar xvf vox.tar
```
### Training
To train a model on specific dataset run:
```
CUDA_VISIBLE_DEVICES=0,1 python run.py --config config/dataset_name.yaml --device_ids 0,1
```
A log folder named after the timestamp will be created. Checkpoints, loss values, reconstruction results will be saved to this folder.
#### Training AVD network
To train a model on specific dataset run:
```
CUDA_VISIBLE_DEVICES=0 python run.py --mode train_avd --checkpoint '{checkpoint_folder}/checkpoint.pth.tar' --config config/dataset_name.yaml
```
Checkpoints, loss values, reconstruction results will be saved to `{checkpoint_folder}`.
### Evaluation on video reconstruction
To evaluate the reconstruction performance run:
```
CUDA_VISIBLE_DEVICES=0 python run.py --mode reconstruction --config config/dataset_name.yaml --checkpoint '{checkpoint_folder}/checkpoint.pth.tar'
```
The `reconstruction` subfolder will be created in `{checkpoint_folder}`.
The generated video will be stored to this folder, also generated videos will be stored in ```png``` subfolder in loss-less '.png' format for evaluation.
To compute metrics, follow instructions from [pose-evaluation](https://github.com/AliaksandrSiarohin/pose-evaluation).
### Image animation demo
- notebook: `demo.ipynb`, edit the config cell and run for image animation.
- python:
```bash
CUDA_VISIBLE_DEVICES=0 python demo.py --config config/vox-256.yaml --checkpoint checkpoints/vox.pth.tar --source_image ./source.jpg --driving_video ./driving.mp4
```
# Acknowledgments
The main code is based upon [FOMM](https://github.com/AliaksandrSiarohin/first-order-model) and [MRAA](https://github.com/snap-research/articulated-animation)
Thanks for the excellent works!
And Thanks to:
- [@chenxwh](https://github.com/chenxwh): Add Web Demo & Docker environment [](https://replicate.com/yoyo-nb/thin-plate-spline-motion-model)
- [@TalkUHulk](https://github.com/TalkUHulk): The C++/Python demo is provided in [Image-Animation-Turbo-Boost](https://github.com/TalkUHulk/Image-Animation-Turbo-Boost)
- [@AK391](https://github.com/AK391): Add huggingface web demo [](https://huggingface.co/spaces/CVPR/Image-Animation-using-Thin-Plate-Spline-Motion-Model)