{"id":13445091,"url":"https://zju3dv.github.io/neuralbody/","last_synced_at":"2025-03-20T19:31:31.354Z","repository":{"id":37634514,"uuid":"325798871","full_name":"zju3dv/neuralbody","owner":"zju3dv","description":"Code for \"Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans\" CVPR 2021 best paper candidate","archived":false,"fork":false,"pushed_at":"2024-01-21T04:11:43.000Z","size":256,"stargazers_count":910,"open_issues_count":3,"forks_count":130,"subscribers_count":42,"default_branch":"master","last_synced_at":"2024-08-01T04:02:40.810Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zju3dv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-12-31T12:46:36.000Z","updated_at":"2024-08-01T04:02:40.824Z","dependencies_parsed_at":"2023-02-09T03:16:14.722Z","dependency_job_id":"0effb6b8-6e7f-44a1-a70b-6d1a780eafd0","html_url":"https://github.com/zju3dv/neuralbody","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2Fneuralbody","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2Fneuralbody/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2Fneuralbody/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2Fneuralbody/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zju3dv","download_url":"https://codeload.github.com/zju3dv/neuralbody/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221800051,"owners_count":16882469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T04:01:49.629Z","updated_at":"2024-10-28T07:30:23.815Z","avatar_url":"https://github.com/zju3dv.png","language":"Python","funding_links":[],"categories":["Papers","Tools, Pipeline \u0026 Utilities"],"sub_categories":["Gaussian Splatting \u0026 NeRF"],"readme":"**News**\n\n* `01/21/2024` We release the [Mobile-Stage dataset](https://docs.google.com/forms/d/e/1FAIpQLSeEbjuTV7w0lfryl-9FPX1VteuPGkbqjDvxXebY02Tm6BMejQ/viewform?usp=sf_link) and [SyntheticHuman++ dataset](https://docs.google.com/forms/d/e/1FAIpQLSctrS3NZ5pThqQzakvitykNgJxpGifT9AOKEtWYZ_JqFQIb3w/viewform?usp=sf_link).\n* `11/04/2023` The enhanced version of the paper has been accepted to T-PAMI. We update information about the journal version of the paper.\n* `05/17/2021` To make the comparison on ZJU-MoCap easier, we save quantitative and qualitative results of other methods at [here](https://github.com/zju3dv/neuralbody/blob/master/supplementary_material.md#results-of-other-methods-on-zju-mocap), including Neural Volumes, Multi-view Neural Human Rendering, and Deferred Neural Human Rendering.\n* `05/13/2021` To make the following works easier compare with our model, we save our rendering results of ZJU-MoCap at [here](https://zjueducn-my.sharepoint.com/:u:/g/personal/pengsida_zju_edu_cn/Ea3VOUy204VAiVJ-V-OGd9YBxdhbtfpS-U6icD_rDq0mUQ?e=cAcylK) and write a [document](supplementary_material.md) that describes the training and test protocols.\n* `05/12/2021` The code supports the test and visualization on unseen human poses.\n* `05/12/2021` We update the ZJU-MoCap dataset with better fitted SMPL using [EasyMocap](https://github.com/zju3dv/EasyMocap). We also release a [website](https://zju3dv.github.io/zju_mocap/) for visualization. Please see [here](https://github.com/zju3dv/neuralbody#potential-problems-of-provided-smpl-parameters) for the usage of provided smpl parameters.\n\n# Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans\n### [Project Page](https://zju3dv.github.io/neuralbody) | [Video](https://www.youtube.com/watch?v=BPCAMeBCE-8) | [Journal Paper](https://ieeexplore.ieee.org/document/10045794) | [Conference Paper](https://arxiv.org/pdf/2012.15838.pdf) | [Data](https://github.com/zju3dv/neuralbody/blob/master/INSTALL.md#zju-mocap-dataset)\n\n![monocular](https://zju3dv.github.io/neuralbody/images/monocular.gif)\n\n\u003e [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://arxiv.org/pdf/2012.15838.pdf)  \n\u003e Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, Xiaowei Zhou  \n\u003e CVPR 2021\n\n\u003e [Implicit Neural Representations with Structured Latent Codes for Human Body Modeling](https://ieeexplore.ieee.org/document/10045794)  \n\u003e Sida Peng, Chen Geng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, Xiaowei Zhou  \n\u003e TPAMI 2023\n\nAny questions or discussions are welcomed!\n\n## Installation\n\nPlease see [INSTALL.md](INSTALL.md) for manual installation.\n\n### Installation using docker\n\nPlease see [docker/README.md](docker/README.md).\n\nThanks to [Zhaoyi Wan](https://github.com/wanzysky) for providing the docker implementation.\n\n## Run the code on the custom dataset\n\nPlease see [CUSTOM](tools/custom).\n\n## Run the code on People-Snapshot\n\nPlease see [INSTALL.md](INSTALL.md) to download the dataset.\n\nWe provide the pretrained models at [here](https://drive.google.com/drive/folders/1yR2KauFaM7kvQgsdlS_qsj9u9Y9qu9C-?usp=sharing).\n\n### Process People-Snapshot\n\nWe already provide some processed data. If you want to process more videos of People-Snapshot, you could use [tools/process_snapshot.py](tools/process_snapshot.py).\n\nYou can also visualize smpl parameters of People-Snapshot with [tools/vis_snapshot.py](tools/vis_snapshot.py).\n\n### Visualization on People-Snapshot\n\nTake the visualization on `female-3-casual` as an example. The command lines for visualization are recorded in [visualize.sh](visualize.sh).\n\n1. Download the corresponding pretrained model and put it to `$ROOT/data/trained_model/if_nerf/female3c/latest.pth`.\n2. Visualization:\n    * Visualize novel views of single frame\n    ```\n    python run.py --type visualize --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c vis_novel_view True num_render_views 144\n    ```\n\n    ![monocular](https://zju3dv.github.io/neuralbody/images/monocular_render.gif)\n\n    * Visualize views of dynamic humans with fixed camera\n    ```\n    python run.py --type visualize --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c vis_novel_pose True\n    ```\n\n    ![monocular](https://zju3dv.github.io/neuralbody/images/monocular_perform.gif)\n\n    * Visualize mesh\n    ```\n    # generate meshes\n    python run.py --type visualize --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c vis_mesh True train.num_workers 0\n    # visualize a specific mesh\n    python tools/render_mesh.py --exp_name female3c --dataset people_snapshot --mesh_ind 226\n    ```\n\n    ![monocular](https://zju3dv.github.io/neuralbody/images/monocular_mesh.gif)\n\n3. The results of visualization are located at `$ROOT/data/render/female3c` and `$ROOT/data/perform/female3c`.\n\n### Training on People-Snapshot\n\nTake the training on `female-3-casual` as an example. The command lines for training are recorded in [train.sh](train.sh).\n\n1. Train:\n    ```\n    # training\n    python train_net.py --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c resume False\n    # distributed training\n    python -m torch.distributed.launch --nproc_per_node=4 train_net.py --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c resume False gpus \"0, 1, 2, 3\" distributed True\n    ```\n2. Train with white background:\n    ```\n    # training\n    python train_net.py --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c resume False white_bkgd True\n    ```\n3. Tensorboard:\n    ```\n    tensorboard --logdir data/record/if_nerf\n    ```\n\n## Run the code on ZJU-MoCap\n\nPlease see [INSTALL.md](INSTALL.md) to download the dataset.\n\nWe provide the pretrained models at [here](https://drive.google.com/drive/folders/1yR2KauFaM7kvQgsdlS_qsj9u9Y9qu9C-?usp=sharing).\n\n### Potential problems of provided smpl parameters\n\n1. The newly fitted parameters locate in `new_params`. Currently, the released pretrained models are trained on previously fitted parameters, which locate in `params`.\n2. The smpl parameters of ZJU-MoCap have different definition from the one of MPI's smplx.\n    * If you want to extract vertices from the provided smpl parameters, please use `zju_smpl/extract_vertices.py`.\n    * The reason that we use the current definition is described at [here](https://github.com/zju3dv/EasyMocap/blob/master/doc/02_output.md#attention-for-smplsmpl-x-users).\n\nIt is okay to train Neural Body with smpl parameters fitted by smplx.\n\n### Test on ZJU-MoCap\n\nThe command lines for test are recorded in [test.sh](test.sh).\n\nTake the test on `sequence 313` as an example.\n\n1. Download the corresponding pretrained model and put it to `$ROOT/data/trained_model/if_nerf/xyzc_313/latest.pth`.\n2. Test on training human poses:\n    ```\n    python run.py --type evaluate --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313\n    ```\n3. Test on unseen human poses:\n    ```\n    python run.py --type evaluate --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 test_novel_pose True\n    ```\n\n### Visualization on ZJU-MoCap\n\nTake the visualization on `sequence 313` as an example. The command lines for visualization are recorded in [visualize.sh](visualize.sh).\n\n1. Download the corresponding pretrained model and put it to `$ROOT/data/trained_model/if_nerf/xyzc_313/latest.pth`.\n2. Visualization:\n    * Visualize novel views of single frame\n    ```\n    python run.py --type visualize --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 vis_novel_view True\n    ```\n    ![zju_mocap](https://zju3dv.github.io/neuralbody/images/zju_mocap_render_313.gif)\n\n    * Visualize novel views of single frame by rotating the SMPL model\n    ```\n    python run.py --type visualize --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 vis_novel_view True num_render_views 100\n    ```\n    ![zju_mocap](https://zju3dv.github.io/neuralbody/images/rotate_smpl.gif)\n\n    * Visualize views of dynamic humans with fixed camera\n    ```\n    python run.py --type visualize --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 vis_novel_pose True num_render_frame 1000 num_render_views 1\n    ```\n    ![zju_mocap](https://zju3dv.github.io/neuralbody/images/zju_mocap_perform_fixed_313.gif) \n\n    * Visualize views of dynamic humans with rotated camera\n    ```\n    python run.py --type visualize --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 vis_novel_pose True num_render_frame 1000\n    ```\n    ![zju_mocap](https://zju3dv.github.io/neuralbody/images/zju_mocap_perform_313.gif)\n\n    * Visualize mesh\n    ```\n    # generate meshes\n    python run.py --type visualize --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 vis_mesh True train.num_workers 0\n    # visualize a specific mesh\n    python tools/render_mesh.py --exp_name xyzc_313 --dataset zju_mocap --mesh_ind 0\n    ```\n    ![zju_mocap](https://zju3dv.github.io/neuralbody/images/zju_mocap_mesh.gif)\n\n4. The results of visualization are located at `$ROOT/data/render/xyzc_313` and `$ROOT/data/perform/xyzc_313`.\n\n### Training on ZJU-MoCap\n\nTake the training on `sequence 313` as an example. The command lines for training are recorded in [train.sh](train.sh).\n\n1. Train:\n    ```\n    # training\n    python train_net.py --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 resume False\n    # distributed training\n    python -m torch.distributed.launch --nproc_per_node=4 train_net.py --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 resume False gpus \"0, 1, 2, 3\" distributed True\n    ```\n2. Train with white background:\n    ```\n    # training\n    python train_net.py --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 resume False white_bkgd True\n    ```\n3. Tensorboard:\n    ```\n    tensorboard --logdir data/record/if_nerf\n    ```\n\n## Citation\n\nIf you find this code useful for your research, please use the following BibTeX entry.\n\n```\n@article{peng2023implicit,\n  title={Implicit Neural Representations with Structured Latent Codes for Human Body Modeling},\n  author={Peng, Sida and Geng, Chen and Zhang, Yuanqing and Xu, Yinghao and Wang, Qianqian and Shuai, Qing and Zhou, Xiaowei and Bao, Hujun},\n  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},\n  year={2023},\n  publisher={IEEE}\n}\n\n@inproceedings{peng2021neural,\n  title={Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans},\n  author={Peng, Sida and Zhang, Yuanqing and Xu, Yinghao and Wang, Qianqian and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},\n  booktitle={CVPR},\n  year={2021}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/zju3dv.github.io%2Fneuralbody%2F","html_url":"https://awesome.ecosyste.ms/projects/zju3dv.github.io%2Fneuralbody%2F","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/zju3dv.github.io%2Fneuralbody%2F/lists"}