{"id":15030527,"url":"https://github.com/snap-research/articulated-animation","last_synced_at":"2025-05-15T23:07:03.070Z","repository":{"id":37738673,"uuid":"350778704","full_name":"snap-research/articulated-animation","owner":"snap-research","description":"Code for Motion Representations for Articulated Animation paper","archived":false,"fork":false,"pushed_at":"2024-03-01T23:35:14.000Z","size":48044,"stargazers_count":1254,"open_issues_count":55,"forks_count":356,"subscribers_count":40,"default_branch":"main","last_synced_at":"2025-04-08T10:24:53.080Z","etag":null,"topics":["deep-learning","first-order-motion-model","image-animation","video-generation"],"latest_commit_sha":null,"homepage":"https://snap-research.github.io/articulated-animation/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snap-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-23T16:17:44.000Z","updated_at":"2025-04-03T11:18:13.000Z","dependencies_parsed_at":"2024-03-02T00:39:32.149Z","dependency_job_id":null,"html_url":"https://github.com/snap-research/articulated-animation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2Farticulated-animation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2Farticulated-animation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2Farticulated-animation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2Farticulated-animation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snap-research","download_url":"https://codeload.github.com/snap-research/articulated-animation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254436948,"owners_count":22070947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","first-order-motion-model","image-animation","video-generation"],"created_at":"2024-09-24T20:13:36.488Z","updated_at":"2025-05-15T23:06:57.815Z","avatar_url":"https://github.com/snap-research.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Motion Representations for Articulated Animation\n\nThis repository contains the source code for the CVPR'2021 paper [Motion Representations for Articulated Animation](https://arxiv.org/abs/2104.11280) by [Aliaksandr Siarohin](https://aliaksandrsiarohin.github.io/aliaksandr-siarohin-website/), [Oliver  Woodford](https://ojwoodford.github.io/), [Jian Ren](https://alanspike.github.io/), [Menglei Chai](https://mlchai.com/) and [Sergey Tulyakov](http://www.stulyakov.com/). \n\nFor more qualitiative examples visit our [project page](https://snap-research.github.io/articulated-animation/).\n\n## Example animation\n\nHere is an example of several images produced by our method. In the first column the driving video is shown. For the remaining columns the top image is animated by using motions extracted from the driving. \n\n![Screenshot](sup-mat/teaser.gif)\n\n### Installation\n\nWe support ```python3```. To install the dependencies run:\n```bash\npip install -r requirements.txt\n```\n\n### YAML configs\n\nThere are several configuration files one for each `dataset` in the `config` folder named as ```config/dataset_name.yaml```. See ```config/dataset.yaml``` to get the description of each parameter.\n\nSee description of the parameters in the ```config/vox256.yaml```. We adjust the the configuration to run on 1 V100 GPU, training on 256x256 dataset takes approximatly 2 days.\n\n### Pre-trained checkpoints\nCheckpoints can be found in https://drive.google.com/drive/folders/1jCeFPqfU_wKNYwof0ONICwsj3xHlr_tb?usp=sharing.\n\n### Animation Demo\nTo run a demo, download a checkpoint and run the following command:\n```bash\npython demo.py  --config config/dataset_name.yaml --driving_video path/to/driving --source_image path/to/source --checkpoint path/to/checkpoint\n```\nThe result will be stored in ```result.mp4```. To use Animation via Disentaglemet add ```--mode avd```, for standard animation add  ```--mode standard``` instead.\n\n### Colab Demo \nWe prepared a demo runnable in google-colab, see: ```demo.ipynb```.\n\n\n### Training\n\nTo train a model run:\n```bash\nCUDA_VISIBLE_DEVICES=0 python run.py --config config/dataset_name.yaml --device_ids 0\n```\nThe code will create a folder in the log directory (each run will create a time-stamped new folder). Checkpoints will be saved to this folder.\nTo check the loss values during training see ```log.txt```.\nYou can also check training data reconstructions in the ```train-vis``` subfolder.\nThen to train **Animation via disentaglement (AVD)** use:\n\n```bash\nCUDA_VISIBLE_DEVICES=0 python run.py --checkpoint log/{folder}/cpk.pth --config config/dataset_name.yaml --device_ids 0 --mode train_avd\n```\nWhere ```{folder}``` is the name of the folder created in the previous step. (Note: use backslash '\\' before space.)\nThis will use the same folder where checkpoint was previously stored.\nIt will create a new checkpoint containing all the previous models and the trained avd_network.\nYou can monitor performance in log file and visualizations in train-vis folder.\n\n### Evaluation on video reconstruction\n\nTo evaluate the reconstruction performance run:\n```bash\nCUDA_VISIBLE_DEVICES=0 python run.py --config config/dataset_name.yaml --mode reconstruction --checkpoint log/{folder}/cpk.pth\n```\nWhere ```{folder}``` is the name of the folder created in the previous step. (Note: use backslash '\\' before space.)\nThe ```reconstruction``` subfolder will be created in the checkpoint folder.\nThe generated video will be stored to this folder, also generated videos will be stored in ```png``` subfolder in loss-less '.png' format for evaluation.\nInstructions for computing metrics from the paper can be found [here](https://github.com/AliaksandrSiarohin/pose-evaluation).\n\n### TED dataset\nFor obtaining TED dataset run the following commands:\n```bash\ngit clone https://github.com/AliaksandrSiarohin/video-preprocessing\ncd video-preprocessing\npython load_videos.py --metadata ../data/ted384-metadata.csv --format .mp4 --out_folder ../data/TED384-v2 --workers 8 --image_shape 384,384\n```\n\n### Training on your own dataset\n1) Resize all the videos to the same size, e.g 256x256, the videos can be in '.gif', '.mp4' or folder with images.\nWe recommend the latter, for each video make a separate folder with all the frames in '.png' format. This format is loss-less, and it has better i/o performance.\n\n2) Create a folder ```data/dataset_name``` with 2 subfolders ```train``` and ```test```, put training videos in the ```train``` and testing in the ```test```.\n\n3) Create a config file ```config/dataset_name.yaml```. See description of the parameters in the ```config/vox256.yaml```.  Specify the dataset root in dataset_params specify by setting  ```root_dir:  data/dataset_name```.  Adjust other parameters as desired, such as the number of epochs for example. Specify ```id_sampling: False``` if you do not want to use id_sampling.\n\n\n#### Additional notes\n\nCitation: \n```\n@inproceedings{siarohin2021motion,\n        author={Siarohin, Aliaksandr and Woodford, Oliver and Ren, Jian and Chai, Menglei and Tulyakov, Sergey},\n        title={Motion Representations for Articulated Animation},\n        booktitle = {CVPR},\n        year = {2021}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnap-research%2Farticulated-animation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnap-research%2Farticulated-animation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnap-research%2Farticulated-animation/lists"}