{"id":13445096,"url":"https://liuyuan-pal.github.io/NeuRay/","last_synced_at":"2025-03-20T19:31:36.639Z","repository":{"id":39345677,"uuid":"386864397","full_name":"liuyuan-pal/NeuRay","owner":"liuyuan-pal","description":"[CVPR2022] Neural Rays for Occlusion-aware Image-based Rendering","archived":false,"fork":false,"pushed_at":"2023-08-21T02:41:01.000Z","size":113672,"stargazers_count":403,"open_issues_count":18,"forks_count":31,"subscribers_count":34,"default_branch":"main","last_synced_at":"2024-08-01T04:02:48.711Z","etag":null,"topics":["nerf","neural-rendering","novel-view-synthesis","radiance-field"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/liuyuan-pal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-07-17T07:05:16.000Z","updated_at":"2024-07-09T08:34:09.000Z","dependencies_parsed_at":"2024-01-18T18:25:09.079Z","dependency_job_id":null,"html_url":"https://github.com/liuyuan-pal/NeuRay","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuyuan-pal%2FNeuRay","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuyuan-pal%2FNeuRay/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuyuan-pal%2FNeuRay/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuyuan-pal%2FNeuRay/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/liuyuan-pal","download_url":"https://codeload.github.com/liuyuan-pal/NeuRay/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244676455,"owners_count":20491829,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nerf","neural-rendering","novel-view-synthesis","radiance-field"],"created_at":"2024-07-31T04:01:49.903Z","updated_at":"2025-03-20T19:31:35.539Z","avatar_url":"https://github.com/liuyuan-pal.png","language":"Python","funding_links":[],"categories":["Papers","Tools, Pipeline \u0026 Utilities"],"sub_categories":["Gaussian Splatting \u0026 NeRF"],"readme":"# NeuRay\n\nRendered video without training on the scene.\n\n![](assets/desktop.gif)\n\n## [Project page](https://liuyuan-pal.github.io/NeuRay/) | [Paper](https://arxiv.org/abs/2107.13421)\n\n## Todo List\n\n- [x] Generalization models and rendering codes.\n- [x] Training of generalization models.\n- [x] Finetuning codes and finetuned models.\n\n## Usage\n### Setup\n```shell\ngit clone git@github.com:liuyuan-pal/NeuRay.git\ncd NeuRay\npip install -r requirements.txt\n```\n\u003cdetails\u003e\n  \u003csummary\u003e Dependencies \u003c/summary\u003e\n\n  - torch==1.7.1\n  - opencv_python==4.4.0\n  - tensorflow==2.4.1\n  - numpy==1.19.2\n  - scipy==1.5.2\n\n\u003c/details\u003e\n\n### Download datasets and pretrained models\n1. Download processed datasets: [DTU-Test](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/yuanly_connect_hku_hk/ESZ5vNtkX6dJlJKt_xoJXkMBwLHmPvnXF0UQhaJQIw858w?e=u2DqHd) / [LLFF](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/yuanly_connect_hku_hk/EbI1OMqOjOdEtS3NqNguPXsBXOfEnG0MWMmD0If-7OR4dg?e=bf6Pvu) / [NeRF Synthetic](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/yuanly_connect_hku_hk/Ec7yNxwmVbBDmccPar34yOgBwGDyztVfpV-XRIhyKLEg2Q?e=gYKSTm).\n2. Download pretrained model [NeuRay-Depth](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/yuanly_connect_hku_hk/EZAC_ae_zExMu-A393Hl3U0B7tYhSHvmyK8MkDX7Q2sNfw?e=vQ0DUl) and [NeuRay-CostVolume](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/yuanly_connect_hku_hk/ERzZxknfEP5ErHBnFNP2AAYBRs99KmFkZUXj8rTu23Fv4g?e=uo6Euu).\n3. Organize datasets and models as follows\n```\nNeuRay\n|-- data\n    |--model\n        |-- neuray_gen_cost_volume\n        |-- neuray_gen_depth\n    |-- dtu_test\n    |-- llff_colmap\n    |-- nerf_synthetic\n```\n\n### Render\n```shell\n# render on lego of the NeRF synthetic dataset\npython render.py --cfg configs/gen/neuray_gen_depth.yaml \\  \n                 --database nerf_synthetic/lego/black_800 \\ # nerf_synthetic/lego/black_400\n                 --pose_type eval                 \n\n# render on snowman of the DTU dataset\npython render.py --cfg configs/gen/neuray_gen_depth.yaml \\  \n                 --database dtu_test/snowman/black_800 \\ # dtu_test/snowman/black_400\n                 --pose_type eval \n                 \n# render on fern of the LLFF dataset\npython render.py --cfg configs/gen/neuray_gen_depth.yaml \\\n                 --database llff_colmap/fern/high \\ # llff_colmap/fern/low\n                 --pose_type eval\n```\nThe rendered images locate in `data/render/\u003cdatabase_name\u003e/\u003crenderer_name\u003e-pretrain-eval/`.\nIf the `pose_type` is `eval`, we also generate ground-truth images in `data/render/\u003cdatabase_name\u003e/gt`. \n\n#### Explanation on parameters of `render.py`.\n\n- `cfg` is the path to the renderer config file, which can also be `configs/gen/neuray_gen_cost_volume.yaml`\n- `database` is a database name consisting of `\u003cdataset_name\u003e/\u003cscene_name\u003e/\u003cscene_setting\u003e`.\n  -  `nerf_synthetic/lego/black_800` means the scene \"lego\" from the \"nerf_synthetic\" dataset using \"black\" background and the resolution \"800X800\".\n  - `dtu_test/snowman/black_800` means the scene \"snowman\" from the \"dtu_test\" dataset using \"black\" background and the resolution \"800X600\".\n  - `llff_colmap/fern/high` means the scene \"fern\" from the \"llff_colmap\" dataset using \"high\" resolution (1008X756).\n  - We may also use `llff_colmlap/fern/low` which renders with \"low\" resolution (504X378)\n\n### Evaluation\n\n```shell\n# psnr/ssim/lpips will be printed on screen\npython eval.py --dir_pr data/render/\u003cdatabase_name\u003e/\u003crenderer_name\u003e-pretrain-eval \\\n               --dir_gt data/render/\u003cdatabase_name\u003e/gt\n\n# example of evaluation on \"fern\".\n# note we should already render images in the \"dir_pr\".\npython eval.py --dir_pr data/render/llff_colmap/fern/high/neuray_gen_depth-pretrain-eval \\\n               --dir_gt data/render/llff_colmap/fern/high/gt\n```\n\n### Render on custom scenes\n\nTo render on custom scenes, please refer to [this](custom_rendering.md)\n\n## Generalization model training\n\n### Download training sets\n\n1. Download [Google Scanned Objects](https://github.com/googleinterns/IBRNet#e-google-scanned-objects), [RealEstate10K](https://github.com/googleinterns/IBRNet#d-realestate10k)\n[Space Dataset](https://github.com/googleinterns/IBRNet#c-spaces-dataset) and [LLFF released Scenes](https://github.com/googleinterns/IBRNet#b-llff-released-scenes) from [IBRNet](https://github.com/googleinterns/IBRNet).\n2. ~~Download colmap depth for forward-facing scenes at [here](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/yuanly_connect_hku_hk/EX0M0c_DyUFDiz1c-ebSO_oBTEeWk8jRYNwCHMgbFH0Pww?e=bO9stn).~~\n3. Download [DTU](https://roboimagedata.compute.dtu.dk/?page_id=36) training images at [here](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/yuanly_connect_hku_hk/EXcPUeyIqAdHrS2LUCmrRJwB8UN0QItiPBm90YuldNm0Ig?e=2POyCI).\n4. ~~Download colmap depth for DTU training images at [here](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/yuanly_connect_hku_hk/EfkjOG2b1epNl322dE3EOeQBAm_Ncver5EmPN4mOZE0ZnA?e=R975nx).~~\n5. The COLMAP depth maps are not available anymore. You may need to run the COLMAP by yourself.\n\nRename directories and organize datasets like\n```shell\nNeuRay\n|-- data\n    |-- google_scanned_objects\n    |-- real_estate_dataset # RealEstate10k-subset  \n    |-- real_iconic_noface\n    |-- spaces_dataset\n    |-- colmap_forward_cache\n    |-- dtu_train\n    |-- colmap_dtu_cache\n```\n\n### Train generalization model\n\nTrain the model with NeuRay initialized from the estimated depth of COLMAP. \n```shell\npython run_training.py --cfg configs/train/gen/neuray_gen_depth_train.yaml\n```\n\nTrain the model with NeuRay initialized from constructed cost volumes.\n```shell\npython run_training.py --cfg configs/train/gen/neuray_gen_cost_volume_train.yaml\n```\n\nModels will be saved at `data/model`. On every 10k steps, we will validate the model and images will be saved at `data/vis_val/\u003cmodel_name\u003e-\u003cval_set_name\u003e`\n\n### Render with trained models\n```shell\npython render.py --cfg configs/gen/neuray_gen_depth_train.yaml \\\n                 --database llff_colmap/fern/high \\\n                 --pose_type eval\n```\n\n## Scene-specific finetuning\n\n### Finetuning\n```shell\n# finetune on lego from the NeRF synthetic dataset\npython run_training.py --cfg configs/train/ft/neuray_ft_depth_lego.yaml\n\n# finetune on fern from the LLFF dataset\npython run_training.py --cfg configs/train/ft/neuray_ft_depth_fern.yaml\n\n# finetune on birds from the DTU dataset\npython run_training.py --cfg configs/train/ft/neuray_ft_depth_birds.yaml\n\n# finetune the model initialized from cost volume\npython run_training.py --cfg configs/train/ft/neuray_ft_cv_lego.yaml\n```\nThe finetuned models will be saved at `data/model`.\n\n### Finetuned models\nWe provide the finetuned models on the NeRF synthetic datasets at [here](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/yuanly_connect_hku_hk/EZsnVJ4qkTRLtcaF5WdnLhcBfU3BOjTTbVbWV6Id2bEPNg?e=lderS0).\n\nDownload the models and organize files like\n```shell\nNeuRay\n|-- data\n    |-- model\n        |-- neuray_ft_lego_pretrain\n        |-- neuray_ft_chair_pretrain\n        ...\n```\n\n### Render with finetuned models\n```shell\n# render on lego of the NeRF synthetic dataset\npython render.py --cfg configs/ft/neuray_ft_lego_pretrain.yaml \\  \n                 --database nerf_synthetic/lego/black_800 \\\n                 --pose_type eval \\\n                 --render_type ft\n```\n\n## Code explanation\n\nWe have provided explanation on variable naming convention in [here](codes_explanations.md) to make our codes more readable.\n\n## Acknowledgements\nIn this repository, we have used codes or datasets from the following repositories. \nWe thank all the authors for sharing great codes or datasets.\n\n- [IBRNet](https://github.com/googleinterns/IBRNet)\n- [MVSNet-official](https://github.com/YoYo000/MVSNet) and [MVSNet-kwea123](https://github.com/kwea123/CasMVSNet_pl)\n- [BlendedMVS](https://github.com/YoYo000/BlendedMVS)\n- [NeRF-official](https://github.com/bmild/nerf) and [NeRF-torch](https://github.com/yenchenlin/nerf-pytorch)\n- [MVSNeRF](https://github.com/apchenstu/mvsnerf)\n- [PixelNeRF](https://github.com/sxyu/pixel-nerf)\n- [COLMAP](https://github.com/colmap/colmap)\n- [IDR](https://lioryariv.github.io/idr/)\n- [RealEstate10K](https://google.github.io/realestate10k/)\n- [DeepView](https://augmentedperception.github.io/deepview/)\n- [Google Scanned Objects](https://app.ignitionrobotics.org/GoogleResearch/fuel/collections/Google%20Scanned%20Objects)\n- [LLFF](https://github.com/Fyusion/LLFF)\n- [DTU](https://roboimagedata.compute.dtu.dk/?page_id=36)\n\n## Citation\n```\n@inproceedings{liu2022neuray,\n  title={Neural Rays for Occlusion-aware Image-based Rendering},\n  author={Liu, Yuan and Peng, Sida and Liu, Lingjie and Wang, Qianqian and Wang, Peng and Theobalt, Christian and Zhou, Xiaowei and Wang, Wenping},\n  booktitle={CVPR},\n  year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/liuyuan-pal.github.io%2FNeuRay%2F","html_url":"https://awesome.ecosyste.ms/projects/liuyuan-pal.github.io%2FNeuRay%2F","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/liuyuan-pal.github.io%2FNeuRay%2F/lists"}