{"id":13445081,"url":"https://zju3dv.github.io/enerf/","last_synced_at":"2025-03-20T19:31:24.893Z","repository":{"id":63885076,"uuid":"570214325","full_name":"zju3dv/ENeRF","owner":"zju3dv","description":"SIGGRAPH Asia 2022: Code for \"Efficient Neural Radiance Fields for Interactive Free-viewpoint Video\"","archived":false,"fork":false,"pushed_at":"2024-03-24T01:10:56.000Z","size":3694,"stargazers_count":431,"open_issues_count":13,"forks_count":29,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-03-14T13:09:54.186Z","etag":null,"topics":["4d-reconstruction","dynamic-view-synthesis","novel-view-synthesis","siggraph-asia-2022"],"latest_commit_sha":null,"homepage":"https://zju3dv.github.io/enerf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zju3dv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-24T15:48:17.000Z","updated_at":"2025-02-06T06:54:28.000Z","dependencies_parsed_at":"2023-02-08T17:00:29.139Z","dependency_job_id":"13104ab0-3bba-4505-98ba-66944791e0bd","html_url":"https://github.com/zju3dv/ENeRF","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2FENeRF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2FENeRF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2FENeRF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2FENeRF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zju3dv","download_url":"https://codeload.github.com/zju3dv/ENeRF/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244676455,"owners_count":20491829,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["4d-reconstruction","dynamic-view-synthesis","novel-view-synthesis","siggraph-asia-2022"],"created_at":"2024-07-31T04:01:48.807Z","updated_at":"2025-03-20T19:31:24.863Z","avatar_url":"https://github.com/zju3dv.png","language":"Python","funding_links":[],"categories":["Papers","Tools, Pipeline \u0026 Utilities"],"sub_categories":["Photogrammetry \u0026 3D Scanning"],"readme":"**News**\n\n* `02/12/2023` We release ENeRF object-compositional representation code including training and visualization for ENeRF-Outdoor dataset.\n* `01/10/2023` We release [ENeRF-Outdoor](https://github.com/zju3dv/ENeRF/blob/master/docs/enerf_outdoor.md) dataset.\n\n# ENeRF: Efficient Neural Radiance Fields for Interactive Free-viewpoint Video\n\n\u003e [Efficient Neural Radiance Fields for Interactive Free-viewpoint Video](https://arxiv.org/abs/2112.01517)  \n\u003e Haotong Lin*, Sida Peng*, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao and Xiaowei Zhou \\\n\u003e SIGGRAPH Asia 2022 conference track  \n\u003e [Project Page](https://zju3dv.github.io/enerf)\n\n## Installation\n\n### Set up the python environment\n\n```\nconda create -n enerf python=3.8\nconda activate enerf\npip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html # Important!\npip install -r requirements.txt\n```\n\n### Set up datasets\n\n#### 0. Set up workspace\nThe workspace is the disk directory that stores datasets, training logs, checkpoints and results. Please ensure it has enough space. \n```\nexport workspace=$PATH_TO_YOUR_WORKSPACE\n```\n   \n#### 1. Pre-trained model\n\nDownload the pretrained model from [dtu_pretrain](https://drive.google.com/drive/folders/10vGC0_DuwLJwfy9OwUHhK7pRPoNP5rux?usp=share_link) (Pretrained on DTU dataset.)\n\nPut it into `$workspace/trained_model/enerf/dtu_pretrain/latest.pth`.\n\n#### 2. DTU\nDownload the preprocessed [DTU training data](https://drive.google.com/file/d/1eDjh-_bxKKnEuz5h-HXS7EDJn59clx6V/view)\nand [Depth_raw](https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/cascade-stereo/CasMVSNet/dtu_data/dtu_train_hr/Depths_raw.zip) from original [MVSNet repo](https://github.com/YoYo000/MVSNet)\nand unzip. [MVSNeRF](https://github.com/apchenstu/mvsnerf) provide a [DTU example](https://1drv.ms/u/s!AjyDwSVHuwr8zhAAXh7x5We9czKj?e=oStQ48), please follow with the example's folder structure.\n\n```\nmv dtu_example.zip $workspace\ncd $workspace\nunzip dtu_example.zip\n```\nThis script only shows the example directory structure. You should download all the scenes in the DTU dataset and organize the data according to the example directory structure. Otherwise you can only do evaluation and fine-tuning on the example data.\n\n#### 2. NeRF Synthetic and Real Forward-facing\nDownload the NeRF Synthetic and Real Forward-facing datasets from [NeRF](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) and unzip them to $workspace. \nYou should have the following directory.\n```\n$workspace/nerf_llff_data\n$workspace/nerf_synthetic\n```\n#### 3. ZJU-MoCap\n\nDownload the ZJU-MoCap dataset from [NeuralBody](https://github.com/zju3dv/neuralbody/blob/master/INSTALL.md#zju-mocap-dataset).\nPut it into $workspace/zju_mocap/CoreView_313.\n\n#### 4. ENeRF-Outdoor\n\nDownload the ENeRF-Outdoor dataset from this [link](https://github.com/zju3dv/ENeRF/blob/master/docs/enerf_outdoor.md).\nPut it into $workspace/enerf_outdoor/actor1.\n\u003c!-- #### 5. DynamicCap --\u003e\n\u003c!-- #### 6. Custom Data --\u003e\n\n## Training and fine-tuning\n\n### Training\nUse the following command to train a generalizable model on DTU.\n```\npython train_net.py --cfg_file configs/enerf/dtu_pretrain.yaml \n```\n\nOur code also supports multi-gpu training. The published pretrained model was trained for 138000 iterations with 4 GPUs.\n```\npython -m torch.distributed.launch --nproc_per_node=4 train_net.py --cfg_file configs/enerf/dtu_pretrain.yaml distributed True gpus 0,1,2,3\n```\n\n\n### Fine-tuning\n\n```\ncd $workspace/trained_model/enerf\nmkdir dtu_ft_scan114\ncp dtu_pretrain/138.pth dtu_ft_scan114\ncd $codespace # codespace is the directory of the ENeRF code\npython train_net.py --cfg_file configs/enerf/dtu/scan114.yaml\n```\n\nFine-tuning for 3000 and 11000 iterations takes about 11 minutes and 40 minutes, respectively, on our test machine ( i9-12900K CPU, RTX 3090 GPU).\n\n### Fine-tuning on the ZJU-MoCap dataset\n\n```\npython train_net.py --cfg_file configs/enerf/zjumocap/zjumocap_train.yaml\n```\n\n### Training on the ENeRF-Outdoor dataset (from scratch)\n\n```\npython train_net.py --cfg_file configs/enerf/enerf_outdoor/actor1.yaml\n```\n\n## Evaluation\n\n### Evaluate the pretrained model on DTU\n\nUse the following command to evaluate the pretrained model on DTU.\n```\npython run.py --type evaluate --cfg_file configs/enerf/dtu_pretrain.yaml enerf.cas_config.render_if False,True enerf.cas_config.volume_planes 48,8 enerf.eval_depth True\n```\n\n\n```\n{'psnr': 27.60513418439332, 'ssim': 0.9570619, 'lpips': 0.08897018397692591}\n{'abs': 4.2624497, 'acc_2': 0.8003020328362158, 'acc_10': 0.9279663826227568}\n{'mvs_abs': 4.4139433, 'mvs_acc_2': 0.7711405202036934, 'mvs_acc_10': 0.9262374398033109}\nFPS:  21.778975517304048\n```\n\n21.8 FPS@512x640 is tested on a desktop with an Intel i9-12900K CPU and an RTX 3090 GPU. **Add the \"save_result True\" parameter at the end of the command to save the rendering result.**\n\n### Evaluate the pretrained model on LLFF and NeRF datasets\n\n```\npython run.py --type evaluate --cfg_file configs/enerf/nerf_eval.yaml\n```\n\n```\npython run.py --type evaluate --cfg_file configs/enerf/llff_eval.yaml\n```\n\n### Evaluate the pretrained model on ZJU-MoCap dataset.\n\n```\npython run.py --type evaluate --cfg_file configs/enerf/zjumocap_eval.yaml\n```\n\n```\n==============================\nCoreView_313_level1 psnr: 31.48 ssim: 0.971 lpips:0.042\n{'psnr': 31.477305846323087, 'ssim': 0.9714806, 'lpips': 0.04184799361974001}\n==============================\nFPS:  49.24468263992353\n```\n\n### Visualization for ENeRF-Outdoor dataset.\n\n```\npython run.py --type visualize --cfg_file configs/enerf/enerf_outdoor/actor1_path.yaml\n```\n\n## Interactive Rendering\n\nWe release the interactive rendering GUI for ZJU-MoCap dataset.\n\n```\npython gui_human.py --cfg_file configs/enerf/interactive/zjumocap.yaml\n```\n\n```\nUsage:\n\nMouse wheel:          Zoom in/out\nMouse left button:    Move\nMouse right button:   Rotate\nKeyboard a:           Align #  Hold down a and then use the mouse right button to rotate the object for a good rendering trajectory\nKeyboard s:           Snap\n```\n\n## Citation\n\nIf you find this code useful for your research, please use the following BibTeX entry.\n\n```\n@inproceedings{lin2022enerf,\n  title={Efficient Neural Radiance Fields for Interactive Free-viewpoint Video},\n  author={Lin, Haotong and Peng, Sida and Xu, Zhen and Yan, Yunzhi and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},\n  booktitle={SIGGRAPH Asia Conference Proceedings},\n  year={2022}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/zju3dv.github.io%2Fenerf%2F","html_url":"https://awesome.ecosyste.ms/projects/zju3dv.github.io%2Fenerf%2F","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/zju3dv.github.io%2Fenerf%2F/lists"}