{"id":19186801,"url":"https://github.com/donydchen/matchnerf","last_synced_at":"2025-03-17T12:10:50.716Z","repository":{"id":154444007,"uuid":"632070416","full_name":"donydchen/matchnerf","owner":"donydchen","description":"🖨️ [arXiv'23] Official PyTorch Implementation of MatchNeRF","archived":false,"fork":false,"pushed_at":"2024-08-29T12:14:57.000Z","size":958,"stargazers_count":172,"open_issues_count":0,"forks_count":5,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-03-09T06:26:50.454Z","etag":null,"topics":["3d-reconstruction","generalizable-nerf","nerf"],"latest_commit_sha":null,"homepage":"https://donydchen.github.io/matchnerf/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/donydchen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-24T16:41:39.000Z","updated_at":"2025-02-17T07:02:11.000Z","dependencies_parsed_at":"2024-10-30T05:51:28.305Z","dependency_job_id":null,"html_url":"https://github.com/donydchen/matchnerf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donydchen%2Fmatchnerf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donydchen%2Fmatchnerf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donydchen%2Fmatchnerf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donydchen%2Fmatchnerf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/donydchen","download_url":"https://codeload.github.com/donydchen/matchnerf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244031033,"owners_count":20386534,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","generalizable-nerf","nerf"],"created_at":"2024-11-09T11:16:47.109Z","updated_at":"2025-03-17T12:10:50.691Z","avatar_url":"https://github.com/donydchen.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv\u003e\n\u003c!-- \u003cimg src='https://i.imgur.com/tFP6Q3p.gif' align=\"right\" height=\"120px\" width=\"180px\" alt='house'\u003e --\u003e\n\u003cimg src='https://i.imgur.com/Tq07diD.gif' align=\"right\" height=\"120px\" width=\"66px\" alt='sculpture'\u003e \n\u003cimg src='https://i.imgur.com/3boKX8u.gif' align=\"right\" height=\"120px\" width=\"180px\" alt='printer'\u003e \n\u003c/div\u003e\n\n\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\n\n# MatchNeRF\n\n\n\n\nOfficial PyTorch implementation for MatchNeRF, a new generalizable NeRF approach that employs **explicit correspondence matching** as the geometry prior and can perform novel view synthesis on unseen scenarios with as few as two source views as input, **without requiring any retraining and fine-tuning**. \u003cbr\u003e\n\n\n\u003e**[Explicit Correspondence Matching for Generalizable Neural Radiance Fields](http://arxiv.org/abs/2304.12294)**  \n\u003e[Yuedong Chen](https://donydchen.github.io/)\u003csup\u003e1\u003c/sup\u003e,\n[Haofei Xu](https://haofeixu.github.io/)\u003csup\u003e2\u003c/sup\u003e,\n[Qianyi Wu](https://qianyiwu.github.io/)\u003csup\u003e1\u003c/sup\u003e,\n[Chuanxia Zheng](https://www.chuanxiaz.com/)\u003csup\u003e3\u003c/sup\u003e,\n[Tat-Jen Cham](https://personal.ntu.edu.sg/astjcham/)\u003csup\u003e4\u003c/sup\u003e,\n[Jianfei Cai](https://jianfei-cai.github.io/)\u003csup\u003e1\u003c/sup\u003e  \n\u003e\u003csup\u003e1\u003c/sup\u003eMonash University, \u003csup\u003e2\u003c/sup\u003eETH Zurich, \u003csup\u003e3\u003c/sup\u003eUniversity of Oxford, \u003csup\u003e4\u003c/sup\u003eNanyang Technological University  \narXiv 2023\n### [Paper](http://arxiv.org/abs/2304.12294) | [Project Page](https://donydchen.github.io/matchnerf) | [Code](https://github.com/donydchen/matchnerf)\n\n\u003cimg src=\"docs/matchnerf.png\"\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eRecent Updates\u003c/summary\u003e\n\n* `25-Apr-2023`: released MatchNeRF codes and models.\n\n\u003c/details\u003e\n\n\u003cbr\u003e\n\n\n----\n\n\n### Table of Contents\n\n* [Setup Environment](#setup-environment)\n* [Download Datasets](#download-datasets)\n  * [DTU (for both training and testing)](#dtu-for-both-training-and-testing)\n  * [Blender (for testing only)](#blender-for-testing-only)\n  * [Real Forward Facing (for testing only)](#real-forward-facing-for-testing-only)\n* [Testing](#testing)\n* [Training](#training)\n* [Rendering Video](#rendering-video)\n* [Use Your Own Data](#use-your-own-data)\n* [Miscellaneous](#miscellaneous)\n\n\n## Setup Environment\n\nThis project is developed and tested on a **CUDA11** device. For other CUDA version, manually update the `requirements.txt` file to match the settings before preceding.\n\n```bash\ngit clone --recursive https://github.com/donydchen/matchnerf.git\ncd matchnerf\nconda create --name matchnerf python=3.8\nconda activate matchnerf\npip install -r requirements.txt\n```\n\nFor rendering video output, it requires `ffmpeg` to be installed on the system, you can double check by running `ffmpeg -version`. If `ffmpeg` does not exist, consider installing it by running `conda install ffmpeg`.\n\n## Download Datasets\n\n### DTU (for both training and testing)\n\n* Download the preprocessed DTU training data [dtu_training.rar](https://drive.google.com/file/d/1eDjh-_bxKKnEuz5h-HXS7EDJn59clx6V/view) and [Depth_raw.zip](https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/cascade-stereo/CasMVSNet/dtu_data/dtu_train_hr/Depths_raw.zip) from original [MVSNet repo](https://github.com/YoYo000/MVSNet).\n\n* Extract 'Cameras/' and 'Rectified/' from the above downloaded 'dtu_training.rar', and extract 'Depths' from the 'Depth_raw.zip'. Link all three folders to `data/DTU`, which should then have the following structure\n\n```bash\ndata/DTU/\n    |__ Cameras/\n    |__ Depths/\n    |__ Rectified/\n```\n\n### Blender (for testing only)\n\n* Download [nerf_synthetic.zip](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) and extract to `data/nerf_synthetic`.\n\n### Real Forward Facing (for testing only)\n\n* Download [nerf_llff_data.zip](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) and extract to `data/nerf_llff_data`.\n\n## Testing\n\n### MVSNeRF Setting (3 Nearest Views)\n\nDownload the pretrained model [matchnerf_3v.pth](https://drive.google.com/file/d/1Powy38EOtsrMYN7nh5rx5ySMJ7LUgGSq) and save to `configs/pretrained_models/matchnerf_3v.pth`, then run\n\n```bash\npython test.py --yaml=test --name=matchnerf_3v\n```\n\nIf encounters CUDA out-of-memory, please reduce the ray sampling number, e.g., append `--nerf.rand_rays_test==4096` to the command.\n\nPerformance should be exactly the same as below,\n\n| Dataset    | PSNR  | SSIM  | LPIPS |\n| ------- | ------| ----- | ------|\n| DTU                  | 26.91 | 0.934 | 0.159 |\n| Real Forward Facing  | 22.43 | 0.805 | 0.244 |\n| Blender | 23.20 | 0.897 | 0.164 |\n\n## Training\n\nDownload the GMFlow pretrained weight ([gmflow_sintel-0c07dcb3.pth](https://drive.google.com/file/d/1d5C5cgHIxWGsFR1vYs5XrQbbUiZl9TX2/view)) from  the original [GMFlow repo](https://github.com/haofeixu/gmflow), and save it to `configs/pretrained_models/gmflow_sintel-0c07dcb3.pth`, then run\n\n```bash\npython train.py --yaml=train\n```\n\n## Rendering Video\n\n```bash\npython test.py --yaml=test_video --name=matchnerf_3v_video\n```\n\nResults (without any per-scene fine-tuning) should be similar as below,\n\n\u003cdetails\u003e\n  \u003csummary\u003eVisual Results\u003c/summary\u003e\n\n![dtu_scan38_view24](https://i.imgur.com/r2vtiaL.gif)\u003cbr\u003e\n*DTU: scan38_view24*\n\n![blender_materials_view36](https://i.imgur.com/eMZjC1K.gif)\u003cbr\u003e\n*Blender: materials_view36*\n\n![llff_leaves_view13](https://i.imgur.com/oLaKtMX.gif)\u003cbr\u003e\n*Real Forward Facing: leaves_view13*\n\n\u003c/details\u003e\n\n\n## Use Your Own Data\n\n* Download the model ([matchnerf_3v_ibr.pth](https://drive.google.com/file/d/1eGY_pkPxxWiSbGFn-Ype8JvW9GqYVfiq)) pretrained with IBRNet data (follow 'GPNR Setting 1'), and save it to `configs/pretrained_models/matchnerf_3v_ibr.pth`.\n* Following the instructions detailed in the [LLFF repo](https://github.com/Fyusion/LLFF#1-recover-camera-poses), use [img2poses.py](https://github.com/Fyusion/LLFF/blob/master/imgs2poses.py) to recover camera poses.\n* Update the colmap data loader at `datasets/colmap.py` accordingly.\n\nWe provide the following 3 input views demo for your reference.\n\n```bash\n# lower resolution but fast\npython test.py --yaml=demo_own\n# full version\npython test.py --yaml=test_video_own\n```\n\nThe generated video will look like,\n\n![colmap_printer](https://i.imgur.com/3boKX8u.gif)\u003cbr\u003e\n*Demo: own data, printer*\n\n\n## Miscellaneous\n\n### Citation\n\nIf you use this project for your research, please cite our paper.\n\n```bibtex\n@article{chen2023matchnerf,\n    title={Explicit Correspondence Matching for Generalizable Neural Radiance Fields},\n    author={Chen, Yuedong and Xu, Haofei and Wu, Qianyi and Zheng, Chuanxia and Cham, Tat-Jen and Cai, Jianfei},\n    journal={arXiv preprint arXiv:2304.12294},\n    year={2023}\n}\n```\n\n### Pull Request\n\nYou are more than welcome to contribute to this project by sending a [pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests).\n\n### Acknowledgments\n\nThis implementation borrowed many code snippets from [GMFlow](https://github.com/haofeixu/gmflow), [MVSNeRF](https://github.com/apchenstu/mvsnerf), [BARF](https://github.com/chenhsuanlin/bundle-adjusting-NeRF) and [GIRAFFE](https://github.com/autonomousvision/giraffe). Many thanks for all the above mentioned projects.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdonydchen%2Fmatchnerf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdonydchen%2Fmatchnerf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdonydchen%2Fmatchnerf/lists"}