{"id":18614333,"url":"https://github.com/apchenstu/mvsnerf","last_synced_at":"2025-04-04T22:08:16.465Z","repository":{"id":41107990,"uuid":"351798635","full_name":"apchenstu/mvsnerf","owner":"apchenstu","description":"[ICCV 2021] Our work presents a novel neural rendering approach that can efficiently reconstruct geometric and neural radiance fields for view synthesis.","archived":false,"fork":false,"pushed_at":"2022-09-08T09:33:11.000Z","size":1972,"stargazers_count":686,"open_issues_count":65,"forks_count":86,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-03-28T21:07:44.767Z","etag":null,"topics":["3d-reconstruction","nerf"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apchenstu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-26T13:53:54.000Z","updated_at":"2025-03-20T08:04:19.000Z","dependencies_parsed_at":"2022-07-12T18:17:24.234Z","dependency_job_id":null,"html_url":"https://github.com/apchenstu/mvsnerf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apchenstu%2Fmvsnerf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apchenstu%2Fmvsnerf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apchenstu%2Fmvsnerf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apchenstu%2Fmvsnerf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apchenstu","download_url":"https://codeload.github.com/apchenstu/mvsnerf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247256114,"owners_count":20909240,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","nerf"],"created_at":"2024-11-07T03:25:54.738Z","updated_at":"2025-04-04T22:08:16.450Z","avatar_url":"https://github.com/apchenstu.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF"],"readme":"# MVSNeRF\n## [Project page](https://apchenstu.github.io/mvsnerf/) |  [Paper](https://arxiv.org/abs/2103.15595)\nThis repository contains a pytorch lightning implementation for the ICCV 2021 paper: [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://arxiv.org/abs/2103.15595). Our work present a novel neural rendering approach that can efficiently reconstruct\ngeometric and neural radiance fields for view synthesis, Moreover, if dense images are captured, our estimated radiance field representation can be easily fine-tuned; this leads to fast per-scene reconstruction.\u003cbr\u003e\u003cbr\u003e\n\n![Pipeline](configs/pipeline.png)\n\n## Installation\n\n#### Tested on Ubuntu 20.04 + Pytorch 1.10.1 + Pytorch Lignting 1.3.5\n\nInstall environment:\n```\nconda create -n mvsnerf python=3.8\nconda activate mvsnerf\npip install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio==0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html\npip install pytorch-lightning==1.3.5 imageio pillow scikit-image opencv-python configargparse lpips kornia warmup_scheduler matplotlib test-tube imageio-ffmpeg\n```\n\n\n## Training\nPlease see each subsection for training on different datasets. Available training datasets:\n\n* [DTU](#dtu)\n* [Blender](#blender) (Realistic Synthetic)\n* [LLFF](#llff) (Real Forward-Facing)\n* [Your own data](#your-own-data) (images/intrinsic/extrinsic/nearfar boundles)\n\n### DTU dataset\n\n#### Data download\n\nDownload the preprocessed [DTU training data](https://drive.google.com/file/d/1eDjh-_bxKKnEuz5h-HXS7EDJn59clx6V/view)\nand [Depth_raw](https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/cascade-stereo/CasMVSNet/dtu_data/dtu_train_hr/Depths_raw.zip) from original [MVSNet repo](https://github.com/YoYo000/MVSNet)\nand unzip. We provide a [DTU example](https://1drv.ms/u/s!AjyDwSVHuwr8zhAAXh7x5We9czKj?e=oStQ48), please\nfollow with the example's folder structure.\n\n#### Training model\n\nRun\n```\nCUDA_VISIBLE_DEVICES=$cuda  python train_mvs_nerf_pl.py \\\n   --expname $exp_name\n   --num_epochs 6\n   --use_viewdirs \\\n   --dataset_name dtu \\\n   --datadir $DTU_DIR\n```\nMore options refer to the `opt.py`, training command example:\n```\nCUDA_VISIBLE_DEVICES=0  python train_mvs_nerf_pl.py\n    --with_depth  --imgScale_test 1.0 \\\n    --expname mvs-nerf-is-all-your-need \\\n    --num_epochs 6 --N_samples 128 --use_viewdirs --batch_size 1024 \\\n    --dataset_name dtu \\\n    --datadir path/to/dtu/data \\\n    --N_vis 6\n```\n\nYou may need to add `--with_depth` if you want to quantity depth during training. `--N_vis` denotes the validation frequency.\n`--imgScale_test` is the downsample ratio during validation, like 0.5. The training process takes about 30h on single RTX 2080Ti\nfor 6 epochs. \n\n*Important*: please always set batch_size to 1 when you are trining a genelize model, you can enlarge it when fine-tuning.\n\n*Checkpoint*: a pre-trained checkpint is included in `ckpts/mvsnerf-v0.tar`. \n\n*Evaluation*: We also provide a rendering and quantity scipt  in `renderer.ipynb`, \nand you can also use the run_batch.py if you want to testing or finetuning on different dataset. More results can be found from\n[Here](https://drive.google.com/drive/folders/1ko8OW38iDtj4fHvX0e3Wom9YvtJNTSXu?usp=sharing),\nplease check your configuration if your rendering result looks absnormal.\n\nRendering from the trained model should have result like this:\n\n![no-finetuned](https://user-images.githubusercontent.com/16453770/124207949-210b8300-db19-11eb-9ab9-610eff35395e.gif)\n\n## Finetuning\n### Blender\n\u003cdetails\u003e\n  \u003csummary\u003eSteps\u003c/summary\u003e\n\n#### Data download\n\nDownload `nerf_synthetic.zip` from [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1)\n\n```\nCUDA_VISIBLE_DEVICES=0  python train_mvs_nerf_finetuning_pl.py  \\\n    --dataset_name blender --datadir /path/to/nerf_synthetic/lego \\\n    --expname lego-ft  --with_rgb_loss  --batch_size 1024  \\\n    --num_epochs 1 --imgScale_test 1.0 --white_bkgd  --pad 0 \\\n    --ckpt ./ckpts/mvsnerf-v0.tar --N_vis 1\n```\n\n\u003c/details\u003e\n\n### LLFF\n\u003cdetails\u003e\n  \u003csummary\u003eSteps\u003c/summary\u003e\n\n\n#### Data download\n\nDownload `nerf_llff_data.zip` from [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1)\n\n```\nCUDA_VISIBLE_DEVICES=0  python train_mvs_nerf_finetuning_pl.py  \\\n    --dataset_name llff --datadir /path/to/nerf_llff_data/{scene_name} \\\n    --expname horns-ft  --with_rgb_loss  --batch_size 1024  \\\n    --num_epochs 1 --imgScale_test 1.0  --pad 24 \\\n    --ckpt ./ckpts/mvsnerf-v0.tar --N_vis 1\n```\n\n\u003c/details\u003e\n\n### DTU\n\u003cdetails\u003e\n  \u003csummary\u003eSteps\u003c/summary\u003e\n\n```\nCUDA_VISIBLE_DEVICES=0  python train_mvs_nerf_finetuning_pl.py  \\\n    --dataset_name dtu_ft --datadir /path/to/DTU/mvs_training/dtu/scan1 \\\n    --expname scan1-ft  --with_rgb_loss  --batch_size 1024  \\\n    --num_epochs 1 --imgScale_test 1.0   --pad 24 \\\n    --ckpt ./ckpts/mvsnerf-v0.tar --N_vis 1\n```\n\n\u003c/details\u003e\n\n## Rendering\nAfter training or finetuning, you can render free-viewpoint videos\nwith the `renderer-video.ipynb`. if you want to use your own data,\nplease using the right hand coordinate system (intrinsic, nearfar and extrinsic either with\ncamera to world or world to camera in opencv format) and modify the rendering scipts.\n\nAfter 10k iterations (~ 15min), you should have videos like this:\n\n![finetuned](https://user-images.githubusercontent.com/16453770/124207013-15b75800-db17-11eb-8d96-e8dbe4181c98.gif)\n\n\n## Citation\nIf you find our code or paper helps, please consider citing:\n```\n@article{chen2021mvsnerf,\n  title={MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo},\n  author={Chen, Anpei and Xu, Zexiang and Zhao, Fuqiang and Zhang, Xiaoshuai and Xiang, Fanbo and Yu, Jingyi and Su, Hao},\n  journal={arXiv preprint arXiv:2103.15595},\n  year={2021}\n}\n```\n\nBig thanks to [**CasMVSNet_pl**](https://github.com/kwea123/CasMVSNet_pl), our code is partially\nborrowing from them.\n\n## Relevant Works\n[**MVSNet: Depth Inference for Unstructured Multi-view Stereo (ECCV 2018)**](https://arxiv.org/abs/1804.02505)\u003cbr\u003e\nYao Yao, Zixin Luo, Shiwei Li, Tian Fang, Long Quan\n\n[**Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (CVPR 2020)**](https://arxiv.org/abs/1912.06378)\u003cbr\u003e\nXiaodong Gu, Zhiwen Fan, Zuozhuo Dai, Siyu Zhu, Feitong Tan, Ping Tan\n\n[**NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020)**](http://www.matthewtancik.com/nerf)\u003cbr\u003e\nBen Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng\n\n[**IBRNet: Learning Multi-View Image-Based Rendering (CVPR 2021)**](https://ibrnet.github.io/)\u003cbr\u003e\nQianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, Thomas Funkhouser\n\n[**PixelNeRF: Neural Radiance Fields from One or Few Images (CVPR 2021)**](https://alexyu.net/pixelnerf/)\u003cbr\u003e\nAlex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapchenstu%2Fmvsnerf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapchenstu%2Fmvsnerf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapchenstu%2Fmvsnerf/lists"}