{"id":15037000,"url":"https://github.com/apple/ml-neuman","last_synced_at":"2025-05-16T10:08:46.754Z","repository":{"id":52678637,"uuid":"516054113","full_name":"apple/ml-neuman","owner":"apple","description":"Official repository of NeuMan: Neural Human Radiance Field from a Single Video (ECCV 2022)","archived":false,"fork":false,"pushed_at":"2023-05-23T14:27:10.000Z","size":7182,"stargazers_count":1283,"open_issues_count":55,"forks_count":147,"subscribers_count":33,"default_branch":"main","last_synced_at":"2025-05-03T20:02:44.073Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apple.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-20T16:21:42.000Z","updated_at":"2025-04-02T13:43:43.000Z","dependencies_parsed_at":"2024-12-15T17:05:49.167Z","dependency_job_id":"9d263f18-854e-419e-bc9e-623e0ebbc645","html_url":"https://github.com/apple/ml-neuman","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-neuman","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-neuman/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-neuman/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-neuman/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apple","download_url":"https://codeload.github.com/apple/ml-neuman/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254509478,"owners_count":22082892,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T20:32:59.681Z","updated_at":"2025-05-16T10:08:41.746Z","avatar_url":"https://github.com/apple.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF Related Tasks"],"readme":"## NeuMan: Neural Human Radiance Field from a Single Video\n\nThis repository is a reference implementation for NeuMan. NeuMan reconstructs both the background scene and an animatable human from a single video using neural radiance fields.\n\n[[Paper]](https://arxiv.org/abs/2203.12575) \n\n \u003cp align=\"center\"\u003e\n  \u003cimg src=\"./resources/teaser.gif\" height=\"260\"\u003e\n\u003c/p\u003e\n\n### Video demos\n\nNovel view and novel pose synthesis\n\n[[Bike]](https://docs-assets.developer.apple.com/ml-research/datasets/neuman/bike.mp4)\n[[Citron]](https://docs-assets.developer.apple.com/ml-research/datasets/neuman/citron.mp4)\n[[Parking lot]](https://docs-assets.developer.apple.com/ml-research/datasets/neuman/demo3.mp4)\n[[Jogging]](https://docs-assets.developer.apple.com/ml-research/datasets/neuman/jogging.mp4)\n[[Lab]](https://docs-assets.developer.apple.com/ml-research/datasets/neuman/lab.mp4)\n[[Seattle]](https://docs-assets.developer.apple.com/ml-research/datasets/neuman/seattle.mp4)\n\nCompositional Synthesis\n\n[[Handshake]](https://docs-assets.developer.apple.com/ml-research/datasets/neuman/handshake.mp4)\n[[Dance]](https://docs-assets.developer.apple.com/ml-research/datasets/neuman/dance.mp4)\n\n### Environment\n\nTo create the environment using Conda:\n\n```sh\nconda env create -f environment.yml\n```\n\nAlternately, you can create the environment by executing:\n\n```sh\nconda create -n neuman_env python=3.7 -y;\nconda activate neuman_env;\nconda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=10.2 -c pytorch;\n# For RTX 30 series GPU with CUDA version 11.x, please use:\n# conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge\nconda install -c fvcore -c iopath -c conda-forge fvcore iopath;\nconda install -c bottler nvidiacub;\nconda install pytorch3d -c pytorch3d;\nconda install -c conda-forge igl;\npip install opencv-python joblib open3d imageio tensorboardX chumpy lpips scikit-image ipython matplotlib;\n```\n\nNotice that `pytorch3d` requires a specific version of pytorch, in our case `pytorch=1.8.0`.\n\nActivate the environment:\n\n```sh \nconda activate neuman_env\n```\n\n### Demo\n\n- Download SMPL weights:\n  - Registration is required to download the UV map(Download UV map in OBJ format) from [SMPL](https://smpl.is.tue.mpg.de/download.php).\n  - Download neutral SMPL weights(SMPLIFY_CODE_V2.ZIP) from [SMPLify](https://smplify.is.tue.mpg.de/download.php), extract `basicModel_neutral_lbs_10_207_0_v1.0.0.pkl` and rename it to `SMPL_NEUTRAL.pkl`.\n  - Put the all the downloaded files into `./data/smplx` folder with following structure:\n   ```bash\n    .\n    └── data\n        └── smplx\n            ├── smpl\n            │   └── SMPL_NEUTRAL.pkl\n            └── smpl_uv.obj\n   ```\n\n- Download NeuMan dataset and pretrained models:\n  - Data ([download](https://docs-assets.developer.apple.com/ml-research/datasets/neuman/dataset.zip))\n  - Pretrained models ([download](https://docs-assets.developer.apple.com/ml-research/datasets/neuman/pretrained.zip))\n\n  Alternately, run the following script to set up data and pretrained models.\n\n  ```sh\n  bash setup_data_and_models.sh\n  ```\n- (*Optional*) Download AMASS dataset for reposing:\n  - AMASS dataset is used for rendering novel poses, specifically `render_reposing.py` and `render_gathering.py`.\n  - We used SFU mocap(SMPL+H G) subset, please download from [AMASS](https://amass.is.tue.mpg.de/download.php).\n  - Put the downloaded mocap data in to `./data/SFU` folder.\n   ```bash\n    .\n    └── data\n        └── SFU\n            ├── 0005\n            ├── 0007\n            │   ...\n            └── 0018\n   ```\n\n- Render using pretrained model\n\n  Render 360 views of a canonical human:\n      \n  ```sh\n  python render_360.py --scene_dir ./data/bike --weights_path ./out/bike_human/checkpoint.pth.tar --mode canonical_360\n  ```\n     \n  Render 360 views of a posed human:\n      \n  ```sh\n  python render_360.py --scene_dir ./data/bike --weights_path ./out/bike_human/checkpoint.pth.tar --mode posed_360\n  ```\n\n  Render test views of a sequence, and evaluate the metrics:\n     \n  ```sh\n  python render_test_views.py --scene_dir ./data/bike --weights_path ./out/bike_human/checkpoint.pth.tar\n  ```\n      \n  Render novel poses with the background:\n      \n  ```sh\n  python render_reposing.py --scene_dir ./data/bike --weights_path ./out/bike_human/checkpoint.pth.tar --motion_name=jumpandroll\n  ```\n      \n  Render telegathering:\n      \n  ```sh\n  python render_gathering.py --actors parkinglot seattle citron --scene_dir ./data/seattle --weights_path ./out/seattle_human/checkpoint.pth.tar\n  ```\n\n\n### Training\n\n- Download NeuMan dataset\n\n- Train scene NeRF\n```sh\npython train.py --scene_dir ./data/bike/ --name=bike_background --train_mode=bkg\n```\n\n- Train human NeRF\n```sh\npython train.py --scene_dir ./data/bike  --name=bike_human --load_background=bike_background --train_mode=smpl_and_offset\n```\n\n### Use your own video\n\n- Preprocess: Check [preprocess](./preprocess/README.md)\n\n### Citation\n\n```\n@inproceedings{jiang2022neuman,\n  title={NeuMan: Neural Human Radiance Field from a Single Video},\n  author={Jiang, Wei and Yi, Kwang Moo and Samei, Golnoosh and Tuzel, Oncel and Ranjan, Anurag},\n  booktitle={Proceedings of the European conference on computer vision (ECCV)},\n  year={2022}\n}\n```\n\n### License\n\nThe code is released under the [LICENSE](./LICENSE) terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-neuman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapple%2Fml-neuman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-neuman/lists"}