{"id":15033884,"url":"https://github.com/yoyo-nb/thin-plate-spline-motion-model","last_synced_at":"2025-05-14T22:09:01.318Z","repository":{"id":37288202,"uuid":"431549854","full_name":"yoyo-nb/Thin-Plate-Spline-Motion-Model","owner":"yoyo-nb","description":"[CVPR 2022] Thin-Plate Spline Motion Model for Image Animation.","archived":false,"fork":false,"pushed_at":"2024-02-10T23:29:11.000Z","size":33138,"stargazers_count":3568,"open_issues_count":85,"forks_count":560,"subscribers_count":63,"default_branch":"main","last_synced_at":"2025-05-14T22:08:54.686Z","etag":null,"topics":["deep-learning","deepfake","face-animation","face-reenactment","image-animation","motion-transfer","pose-transfer","talking-head"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yoyo-nb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-24T16:10:46.000Z","updated_at":"2025-05-14T06:41:31.000Z","dependencies_parsed_at":"2024-01-06T20:18:03.954Z","dependency_job_id":"f670869a-fd73-4b2b-9528-68f2b8e732ac","html_url":"https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yoyo-nb%2FThin-Plate-Spline-Motion-Model","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yoyo-nb%2FThin-Plate-Spline-Motion-Model/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yoyo-nb%2FThin-Plate-Spline-Motion-Model/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yoyo-nb%2FThin-Plate-Spline-Motion-Model/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yoyo-nb","download_url":"https://codeload.github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254235700,"owners_count":22036964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","deepfake","face-animation","face-reenactment","image-animation","motion-transfer","pose-transfer","talking-head"],"created_at":"2024-09-24T20:23:06.292Z","updated_at":"2025-05-14T22:08:56.297Z","avatar_url":"https://github.com/yoyo-nb.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [CVPR2022] Thin-Plate Spline Motion Model for Image Animation\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n![stars](https://img.shields.io/github/stars/yoyo-nb/Thin-Plate-Spline-Motion-Model.svg?style=flat)\n![GitHub repo size](https://img.shields.io/github/repo-size/yoyo-nb/Thin-Plate-Spline-Motion-Model.svg)\n\nSource code of the CVPR'2022 paper \"Thin-Plate Spline Motion Model for Image Animation\"\n\n[**Paper**](https://arxiv.org/abs/2203.14367) **|** [**Supp**](https://cloud.tsinghua.edu.cn/f/f7b8573bb5b04583949f/?dl=1)\n\n### Example animation\n\n![vox](assets/vox.gif)\n![ted](assets/ted.gif)\n\n**PS**: The paper trains the model for 100 epochs for a fair comparison. You can use more data and train for more epochs to get better performance.\n\n\n### Web demo for animation\n- Integrated into [Huggingface Spaces 🤗](https://huggingface.co/spaces) using [Gradio](https://github.com/gradio-app/gradio). Try out the Web Demo: [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/CVPR/Image-Animation-using-Thin-Plate-Spline-Motion-Model)\n- Try the web demo for animation here: [![Replicate](https://replicate.com/yoyo-nb/thin-plate-spline-motion-model/badge)](https://replicate.com/yoyo-nb/thin-plate-spline-motion-model)\n- Google Colab: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1DREfdpnaBhqISg0fuQlAAIwyGVn1loH_?usp=sharing)\n\n### Pre-trained models\n- ~~[Tsinghua Cloud](https://cloud.tsinghua.edu.cn/d/30ab8765da364fefa101/)~~\n- [Yandex](https://disk.yandex.com/d/bWopgbGj1ZUV1w)\n- [Google Drive](https://drive.google.com/drive/folders/1pNDo1ODQIb5HVObRtCmubqJikmR7VVLT?usp=sharing)\n- [Baidu Yun](https://pan.baidu.com/s/1hnXmDpIbRC6WqE3tF9c5QA?pwd=1234)\n\n### Installation\n\nWe support ```python3```.(Recommended version is Python 3.9).\nTo install the dependencies run:\n```bash\npip install -r requirements.txt\n```\n\n\n### YAML configs\n \nThere are several configuration files one for each `dataset` in the `config` folder named as ```config/dataset_name.yaml```. \n\nSee description of the parameters in the ```config/taichi-256.yaml```.\n\n### Datasets\n\n1) **MGif**. Follow [Monkey-Net](https://github.com/AliaksandrSiarohin/monkey-net).\n\n2) **TaiChiHD** and **VoxCeleb**. Follow instructions from [video-preprocessing](https://github.com/AliaksandrSiarohin/video-preprocessing). \n\n3) **TED-talks**. Follow instructions from [MRAA](https://github.com/snap-research/articulated-animation).\n\nHere are **VoxCeleb**, **TaiChiHD** and **TED-talks**  pre-processed datasets used in the paper. [Baidu Yun](https://pan.baidu.com/s/1HKJOtXBIiP_tlLiFbzn3oA?pwd=x7xv)\nDownload all files under the folder, then merge the files and decompress, for example:\n```bash\ncat vox.tar.* \u003e vox.tar\ntar xvf vox.tar\n```\n\n\n### Training\nTo train a model on specific dataset run:\n```\nCUDA_VISIBLE_DEVICES=0,1 python run.py --config config/dataset_name.yaml --device_ids 0,1\n```\nA log folder named after the timestamp will be created. Checkpoints, loss values, reconstruction results will be saved to this folder.\n\n\n#### Training AVD network\nTo train a model on specific dataset run:\n```\nCUDA_VISIBLE_DEVICES=0 python run.py --mode train_avd --checkpoint '{checkpoint_folder}/checkpoint.pth.tar' --config config/dataset_name.yaml\n```\nCheckpoints, loss values, reconstruction results will be saved to `{checkpoint_folder}`.\n\n\n\n### Evaluation on video reconstruction\n\nTo evaluate the reconstruction performance run:\n```\nCUDA_VISIBLE_DEVICES=0 python run.py --mode reconstruction --config config/dataset_name.yaml --checkpoint '{checkpoint_folder}/checkpoint.pth.tar'\n```\nThe `reconstruction` subfolder will be created in `{checkpoint_folder}`.\nThe generated video will be stored to this folder, also generated videos will be stored in ```png``` subfolder in loss-less '.png' format for evaluation.\nTo compute metrics, follow instructions from [pose-evaluation](https://github.com/AliaksandrSiarohin/pose-evaluation).\n\n\n### Image animation demo\n- notebook: `demo.ipynb`, edit the config cell and run for image animation.\n- python:\n```bash\nCUDA_VISIBLE_DEVICES=0 python demo.py --config config/vox-256.yaml --checkpoint checkpoints/vox.pth.tar --source_image ./source.jpg --driving_video ./driving.mp4\n```\n\n# Acknowledgments\nThe main code is based upon [FOMM](https://github.com/AliaksandrSiarohin/first-order-model) and [MRAA](https://github.com/snap-research/articulated-animation)\n\nThanks for the excellent works!\n\nAnd Thanks to:\n\n- [@chenxwh](https://github.com/chenxwh): Add Web Demo \u0026 Docker environment [![Replicate](https://replicate.com/yoyo-nb/thin-plate-spline-motion-model/badge)](https://replicate.com/yoyo-nb/thin-plate-spline-motion-model) \n\n- [@TalkUHulk](https://github.com/TalkUHulk): The C++/Python demo is provided in [Image-Animation-Turbo-Boost](https://github.com/TalkUHulk/Image-Animation-Turbo-Boost)\n\n- [@AK391](https://github.com/AK391): Add huggingface web demo [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/CVPR/Image-Animation-using-Thin-Plate-Spline-Motion-Model)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyoyo-nb%2Fthin-plate-spline-motion-model","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyoyo-nb%2Fthin-plate-spline-motion-model","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyoyo-nb%2Fthin-plate-spline-motion-model/lists"}