{"id":18614454,"url":"https://github.com/zju3dv/animatable_nerf","last_synced_at":"2025-04-05T03:12:33.257Z","repository":{"id":38680393,"uuid":"364952759","full_name":"zju3dv/animatable_nerf","owner":"zju3dv","description":"Code for \"Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos\" TPAMI 2024, ICCV 2021","archived":false,"fork":false,"pushed_at":"2024-01-21T08:23:39.000Z","size":223,"stargazers_count":517,"open_issues_count":6,"forks_count":52,"subscribers_count":32,"default_branch":"master","last_synced_at":"2025-03-29T02:08:41.178Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zju3dv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-06T15:15:56.000Z","updated_at":"2025-03-26T07:39:30.000Z","dependencies_parsed_at":"2024-01-18T18:10:04.372Z","dependency_job_id":"28a03526-1f42-4ceb-a298-401d0439fc0d","html_url":"https://github.com/zju3dv/animatable_nerf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2Fanimatable_nerf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2Fanimatable_nerf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2Fanimatable_nerf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2Fanimatable_nerf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zju3dv","download_url":"https://codeload.github.com/zju3dv/animatable_nerf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247280272,"owners_count":20912967,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T03:25:56.953Z","updated_at":"2025-04-05T03:12:33.237Z","avatar_url":"https://github.com/zju3dv.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF Related Tasks"],"readme":"**News**\n\n* `01/21/2024` We release the [Mobile-Stage dataset](https://docs.google.com/forms/d/e/1FAIpQLSeEbjuTV7w0lfryl-9FPX1VteuPGkbqjDvxXebY02Tm6BMejQ/viewform?usp=sf_link) and [SyntheticHuman++ dataset](https://docs.google.com/forms/d/e/1FAIpQLSctrS3NZ5pThqQzakvitykNgJxpGifT9AOKEtWYZ_JqFQIb3w/viewform?usp=sf_link).\n* `01/12/2024` [Animatable Neural Fields](https://arxiv.org/abs/2203.08133) gets accepted to TPAMI.\n* `07/09/2022` This repository includes the implementation of Animatable SDF (now dubbed [Animatable Neural Fields](https://arxiv.org/abs/2203.08133)).\n* `07/09/2022` We release the [extended version](https://arxiv.org/abs/2203.08133) of Animatable NeRF. We evaluated three different versions of Animatable Neural Fields, including vanilla Animatable NeRF, a version where the neural blend weight field is replaced with displacement field and a version where the canonical NeRF model is replaced with a neural surface field (output is SDF instead of volume density, also using displacement field). We also provide evaluation framework for reconstruction quality comparison.\n* `10/28/2021` To make the comparison with Animatable NeRF easier on the Human3.6M dataset, we save the quantitative results at [here](https://zjueducn-my.sharepoint.com/:f:/g/personal/pengsida_zju_edu_cn/EpW0AHZh1OtDoa-vTaaCAYgBddyACEICg-941VYgyASk7g?e=W4KvSK), which also contains the results of other methods, including Neural Body, D-NeRF, Multi-view Neural Human Rendering, and Deferred Neural Human Rendering.\n\n# Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos\n\n### [Project Page](https://zju3dv.github.io/animatable_sdf) | [Video](https://www.youtube.com/watch?v=eWOSWbmfJo4) | [Paper](https://arxiv.org/abs/2203.08133) | [Data](https://github.com/zju3dv/animatable_nerf/blob/master/INSTALL.md#zju-mocap-dataset) | [ICCV21](https://zju3dv.github.io/animatable_nerf)\n\n![teaser](https://zju3dv.github.io/animatable_nerf/images/github_teaser.gif)\n\n\u003e [Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos](https://arxiv.org/abs/2203.081332)  \n\u003e Sida Peng, Zhen Xu, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Hujun Bao, Xiaowei Zhou  \n\u003e TPAMI 2024\n\n\u003e [Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies](https://arxiv.org/abs/2105.02872)  \n\u003e Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, Hujun Bao  \n\u003e ICCV 2021\n\nAny questions or discussions are welcomed!\n\n## Installation\n\nPlease see [INSTALL.md](INSTALL.md) for manual installation.\n\n## Run the code on Human3.6M\n\nSince the license of Human3.6M dataset does not allow us to distribute its data, we cannot release the processed Human3.6M dataset publicly. If someone is interested at the processed data, please email me.\n\nWe provide the pretrained models at [here](https://drive.google.com/drive/folders/1XH5zUMkguUW64GKulWTo8oOWZra6Dnzy?usp=sharing).\n\n### Test on Human3.6M\n\nThe command lines for test are recorded in [test.sh](test.sh).\n\nTake the test on `S9` as an example.\n\n1. Download the corresponding pretrained models, and put it to `$ROOT/data/trained_model/deform/aninerf_s9p/latest.pth` and `$ROOT/data/trained_model/deform/aninerf_s9p_full/latest.pth`.\n2. Test on training human poses:\n\n    ```shell\n    python run.py --type evaluate --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p resume True\n    ```\n\n3. Test on unseen human poses:\n\n    ```shell\n    python run.py --type evaluate --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p_full resume True aninerf_animation True init_aninerf aninerf_s9p test_novel_pose True\n    ```\n\n### Visualization on Human3.6M\n\nTake the visualization on `S9` as an example.\n\n1. Download the corresponding pretrained models, and put it to `$ROOT/data/trained_model/deform/aninerf_s9p/latest.pth` and `$ROOT/data/trained_model/deform/aninerf_s9p_full/latest.pth`.\n2. Visualization:\n    * Visualize novel views of the 0-th frame\n\n    ```shell\n    python run.py --type visualize --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p resume True vis_novel_view True begin_ith_frame 0\n    ```\n\n    * Visualize views of dynamic humans with 3-th camera\n\n    ```shell\n    python run.py --type visualize --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p resume True vis_pose_sequence True test_view \"3,\"\n    ```\n\n    * Visualize mesh\n\n    ```shell\n    # generate meshes\n    python run.py --type visualize --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p vis_posed_mesh True\n    ```\n\n3. The results of visualization are located at `$ROOT/data/novel_view/aninerf_s9p` and `$ROOT/data/novel_pose/aninerf_s9p`.\n\n### Training on Human3.6M\n\nTake the training on `S9` as an example. The command lines for training are recorded in [train.sh](train.sh).\n\n1. Train:\n\n    ```shell\n    # training\n    python train_net.py --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p resume False\n\n    # training the blend weight fields of unseen human poses\n    python train_net.py --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p_full resume False aninerf_animation True init_aninerf aninerf_s9p\n    ```\n\n2. Tensorboard:\n\n    ```shell\n    tensorboard --logdir data/record/deform\n    ```\n\n## Run the code on ZJU-MoCap\n\nIf someone wants to download the ZJU-Mocap dataset, please fill in the [agreement](https://pengsida.net/project_page_assets/files/ZJU-MoCap_Agreement.pdf), and email me (pengsida@zju.edu.cn) and cc Xiaowei Zhou (xwzhou@zju.edu.cn) to request the download link.\n\nWe provide the pretrained models at [here](https://drive.google.com/drive/folders/1XH5zUMkguUW64GKulWTo8oOWZra6Dnzy?usp=sharing).\n\n### Test on ZJU-MoCap\n\nThe command lines for test are recorded in [test.sh](test.sh).\n\nTake the test on `313` as an example.\n\n1. Download the corresponding pretrained models, and put it to `$ROOT/data/trained_model/deform/aninerf_313/latest.pth` and `$ROOT/data/trained_model/deform/aninerf_313_full/latest.pth`.\n2. Test on training human poses:\n\n    ```shell\n    python run.py --type evaluate --cfg_file configs/aninerf_313.yaml exp_name aninerf_313 resume True\n    ```\n\n3. Test on unseen human poses:\n\n    ```shell\n    python run.py --type evaluate --cfg_file configs/aninerf_313.yaml exp_name aninerf_313_full resume True aninerf_animation True init_aninerf aninerf_313 test_novel_pose True\n    ```\n\n### Visualization on ZJU-MoCap\n\nTake the visualization on `313` as an example.\n\n1. Download the corresponding pretrained models, and put it to `$ROOT/data/trained_model/deform/aninerf_313/latest.pth` and `$ROOT/data/trained_model/deform/aninerf_313_full/latest.pth`.\n2. Visualization:\n    * Visualize novel views of the 0-th frame\n\n    ```shell\n    python run.py --type visualize --cfg_file configs/aninerf_313.yaml exp_name aninerf_313 resume True vis_novel_view True begin_ith_frame 0\n    ```\n\n    * Visualize views of dynamic humans with 0-th camera\n\n    ```shell\n    python run.py --type visualize --cfg_file configs/aninerf_313.yaml exp_name aninerf_313 resume True vis_pose_sequence True test_view \"0,\"\n    ```\n\n    * Visualize mesh\n\n    ```shell\n    # generate meshes\n    python run.py --type visualize --cfg_file configs/aninerf_313.yaml exp_name aninerf_313 vis_posed_mesh True\n    ```\n\n3. The results of visualization are located at `$ROOT/data/novel_view/aninerf_313` and `$ROOT/data/novel_pose/aninerf_313`.\n\n### Training on ZJU-MoCap\n\nTake the training on `313` as an example. The command lines for training are recorded in [train.sh](train.sh).\n\n1. Train:\n\n    ```shell\n    # training\n    python train_net.py --cfg_file configs/aninerf_313.yaml exp_name aninerf_313 resume False\n\n    # training the blend weight fields of unseen human poses\n    python train_net.py --cfg_file configs/aninerf_313.yaml exp_name aninerf_313_full resume False aninerf_animation True init_aninerf aninerf_313\n    ```\n\n2. Tensorboard:\n\n    ```shell\n    tensorboard --logdir data/record/deform\n    ```\n\n## Extended Version\n\nAddtional training and test commandlines are recorded in [train.sh](train.sh) and [test.sh](test.sh).\n\nMoreover, we compiled a list of all possible commands to run in [extension.sh](extension.sh) using on the S9 sequence of the Human3.6M dataset.\n\nThis include training, evaluating and visualizing the original Animatable NeRF implementation and all three extented versions.\n\nHere we list the portion of the commands for the SDF-PDF configuration:\n\n```shell\n# extension: anisdf_pdf\n\n# evaluating on training poses for anisdf_pdf\npython run.py --type evaluate --cfg_file configs/sdf_pdf/anisdf_pdf_s9p.yaml exp_name anisdf_pdf_s9p resume True\n\n# evaluating on novel poses for anisdf_pdf\npython run.py --type evaluate --cfg_file configs/sdf_pdf/anisdf_pdf_s9p.yaml exp_name anisdf_pdf_s9p resume True test_novel_pose True\n\n# visualizing novel view of 0th frame for anisdf_pdf\npython run.py --type visualize --cfg_file configs/sdf_pdf/anisdf_pdf_s9p.yaml exp_name anisdf_pdf_s9p resume True vis_novel_view True begin_ith_frame 0\n\n# visualizing animation of 3rd camera for anisdf_pdf\npython run.py --type visualize --cfg_file configs/sdf_pdf/anisdf_pdf_s9p.yaml exp_name anisdf_pdf_s9p resume True vis_pose_sequence True test_view \"3,\"\n\n# generating posed mesh for anisdf_pdf\npython run.py --type visualize --cfg_file configs/sdf_pdf/anisdf_pdf_s9p.yaml exp_name anisdf_pdf_s9p vis_posed_mesh True\n\n# training base model for anisdf_pdf\npython train_net.py --cfg_file configs/sdf_pdf/anisdf_pdf_s9p.yaml exp_name anisdf_pdf_s9p resume False\n```\n\nTo run Animatable NeRF on other officially supported datasets, simply change the `--cfg_file` and `exp_name` parameters.\n\nNote that for Animatable NeRF with pose-dependent displacement field (NeRF-PDF) and Animatable Neural Surface with pose-dependent displacement field (SDF-PDF), there's no need for training the blend weight fields of unseen human poses.\n\n### MonoCap dataset\n\nMonoCap is a dataset composed by authors of [animatable sdf](https://zju3dv.github.io/animatable_sdf/) from [DeepCap](https://people.mpi-inf.mpg.de/~mhaberma/projects/2020-cvpr-deepcap/) and [DynaCap](https://people.mpi-inf.mpg.de/~mhaberma/projects/2021-ddc/).\n\nSince the license of [DeepCap](https://people.mpi-inf.mpg.de/~mhaberma/projects/2020-cvpr-deepcap/) and [DynaCap](https://people.mpi-inf.mpg.de/~mhaberma/projects/2021-ddc/) dataset does not allow us to distribute its data, we cannot release the processed MonoCap dataset publicly. If you are interested in the processed data, please download the raw data from [here](https://gvv-assets.mpi-inf.mpg.de/) and email me for instructions on how to process the data.\n\n### SyntheticHuman Dataset\n\nSyntheticHuman is a dataset created by authors of [animatable sdf](https://zju3dv.github.io/animatable_sdf/). It contains multi-view videos of 3D human rendered from characters in the [RenderPeople](https://renderpeople.com/) dataset along with the groud truth 3D model.\n\nSince the license of the [RenderPeople](https://renderpeople.com/) dataset does not allow distribution of the 3D model, we cannot realease the processed SyntheticHuman dataset publicly. If you are interested in this dataset, please email me for instructions on how to generate the data.\n\n## Citation\n\nIf you find this code useful for your research, please use the following BibTeX entry.\n\n```bibtex\n@article{peng2024animatable,\n    title={Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos},\n    author={Peng, Sida and Xu, Zhen and Dong, Junting and Wang, Qianqian and Zhang, Shangzhan and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},\n    journal={TPAMI},\n    year={2024},\n    publisher={IEEE}\n}\n\n@inproceedings{peng2021animatable,\n  title={Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies},\n  author={Peng, Sida and Dong, Junting and Wang, Qianqian and Zhang, Shangzhan and Shuai, Qing and Zhou, Xiaowei and Bao, Hujun},\n  booktitle={ICCV},\n  year={2021}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzju3dv%2Fanimatable_nerf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzju3dv%2Fanimatable_nerf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzju3dv%2Fanimatable_nerf/lists"}