{"id":18614361,"url":"https://github.com/idiap/GeoNeRF","last_synced_at":"2025-04-11T00:30:34.077Z","repository":{"id":39605892,"uuid":"497975711","full_name":"idiap/GeoNeRF","owner":"idiap","description":"Generalizing NeRF with Geometry Priors","archived":false,"fork":false,"pushed_at":"2022-09-02T11:31:33.000Z","size":50,"stargazers_count":117,"open_issues_count":1,"forks_count":7,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-04-07T21:41:27.963Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/idiap.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-30T14:23:52.000Z","updated_at":"2025-02-25T03:59:02.000Z","dependencies_parsed_at":"2022-08-01T08:09:45.552Z","dependency_job_id":null,"html_url":"https://github.com/idiap/GeoNeRF","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2FGeoNeRF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2FGeoNeRF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2FGeoNeRF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2FGeoNeRF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/idiap","download_url":"https://codeload.github.com/idiap/GeoNeRF/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322208,"owners_count":21084333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T03:25:55.234Z","updated_at":"2025-04-11T00:30:34.054Z","avatar_url":"https://github.com/idiap.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF"],"readme":"\u003e # [CVPR 2022] GeoNeRF: Generalizing NeRF with Geometry Priors \u003cbr\u003e\r\n\u003e Mohammad Mahdi Johari, Yann Lepoittevin, François Fleuret \u003cbr\u003e\r\n\u003e [Project Page](https://www.idiap.ch/paper/geonerf/) | [Paper](https://arxiv.org/abs/2111.13539)\r\n\r\nThis repository contains a PyTorch Lightning implementation of our paper, GeoNeRF: Generalizing NeRF with Geometry Priors.\r\n\r\n## Installation\r\n\r\n#### Tested on NVIDIA Tesla V100 and GeForce RTX 3090 GPUs with PyTorch 1.9 and PyTorch Lightning 1.3.7\r\n\r\nTo install the dependencies, in addition to PyTorch, run:\r\n\r\n```\r\npip install -r requirements.txt\r\n```\r\n\r\n## Evaluation and Training\r\nTo reproduce our results, download pretrained weights from [here](https://drive.google.com/drive/folders/1ZtAc7VYvltcdodT_BrUrQ_4IAhz_L-Rf?usp=sharing) and put them in [pretrained_weights](./pretrained_weights) folder. Then, follow the instructions for each of the [LLFF (Real Forward-Facing)](#llff-real-forward-facing-dataset), [NeRF (Realistic Synthetic)](#nerf-realistic-synthetic-dataset), and [DTU](#dtu-dataset) datasets.\r\n\r\n## LLFF (Real Forward-Facing) Dataset\r\nDownload `nerf_llff_data.zip` from [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) and set its path as `llff_path` in the [config_llff.txt](./configs/config_llff.txt) file.\r\n\r\nFor evaluating our generalizable model (`pretrained.ckpt` model in the [pretrained_weights](./pretrained_weights) folder), set the `scene` properly (e.g. fern) and set the number of source views to 9 (nb_views = 9) in the [config_llff.txt](./configs/config_llff.txt) file and run the following command:\r\n\r\n```\r\npython run_geo_nerf.py --config configs/config_llff.txt --eval\r\n```\r\n\r\nFor fine-tuning on a specific scene, set nb_views = 7 and run the following command:\r\n\r\n```\r\npython run_geo_nerf.py --config configs/config_llff.txt\r\n```\r\n\r\nOnce fine-tuning is finished, run the evaluation command with nb_views = 9 to get the final rendered results.\r\n\r\n## NeRF (Realistic Synthetic) Dataset\r\nDownload `nerf_synthetic.zip` from [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) and set its path as `nerf_path` in the [config_nerf.txt](configs/config_nerf.txt) file.\r\n\r\nFor evaluating our generalizable model (`pretrained.ckpt` model in the [pretrained_weights](./pretrained_weights) folder), set the `scene` properly (e.g. lego) and set the number of source views to 9 (nb_views = 9) in the [config_nerf.txt](configs/config_nerf.txt) file and run the following command:\r\n\r\n```\r\npython run_geo_nerf.py --config configs/config_nerf.txt --eval\r\n```\r\n\r\nFor fine-tuning on a specific scene, set nb_views = 7 and run the following command:\r\n\r\n```\r\npython run_geo_nerf.py --config configs/config_nerf.txt\r\n```\r\n\r\nOnce fine-tuning is finished, run the evaluation command with nb_views = 9 to get the final rendered results.\r\n\r\n## DTU Dataset\r\n Download the preprocessed [DTU training data](https://drive.google.com/file/d/1eDjh-_bxKKnEuz5h-HXS7EDJn59clx6V/view) \r\nand replace its `Depths` directory with [Depth_raw](https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/cascade-stereo/CasMVSNet/dtu_data/dtu_train_hr/Depths_raw.zip) from original [MVSNet repository](https://github.com/YoYo000/MVSNet), and set `dtu_pre_path` referring to this dataset in the [config_dtu.txt](configs/config_dtu.txt) file.\r\n\r\nThen, download the original `Rectified` images from [DTU Website](https://roboimagedata.compute.dtu.dk/?page_id=36), and set `dtu_path` in the [config_dtu.txt](configs/config_dtu.txt) file accordingly.\r\n\r\nFor evaluating our generalizable model (`pretrained.ckpt` model in the [pretrained_weights](./pretrained_weights) folder), set the `scene` properly (e.g. scan21) and set the number of source views to 9 (nb_views = 9) in the [config_dtu.txt](./configs/config_dtu.txt) file and run the following command:\r\n\r\n```\r\npython run_geo_nerf.py --config configs/config_dtu.txt --eval\r\n```\r\n\r\nFor fine-tuning on a specific scene, use the same nb_views = 9 and run the following command:\r\n\r\n```\r\npython run_geo_nerf.py --config configs/config_dtu.txt\r\n```\r\n\r\nOnce fine-tuning is finished, run the evaluation command with nb_views = 9 to get the final rendered results.\r\n\r\n### RGBD Compatible model\r\nBy adding `--use_depth` argument to the aforementioned commands, you can use our RGB compatible model on the DTU dataset and exploit the ground truth, low-resolution depths to help the rendering process. The pretrained weights for this model is `pretrained_w_depth.ckpt`.\r\n\r\n## Training From Scratch\r\nFor training our model from scratch, first, prepare the following datasets:\r\n\r\n* The original `Rectified` images from [DTU](https://roboimagedata.compute.dtu.dk/?page_id=36). Set the corresponding path as  `dtu_path` in the [config_general.txt](configs/config_general.txt) file.\r\n\r\n* The preprocessed [DTU training data](https://drive.google.com/file/d/1eDjh-_bxKKnEuz5h-HXS7EDJn59clx6V/view) \r\nwith the replacement of its `Depths` directory with [Depth_raw](https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/cascade-stereo/CasMVSNet/dtu_data/dtu_train_hr/Depths_raw.zip). Set the corresponding path as  `dtu_pre_path` in the [config_general.txt](configs/config_general.txt) file.\r\n\r\n * LLFF released scenes. Download [real_iconic_noface.zip](https://drive.google.com/drive/folders/1M-_Fdn4ajDa0CS8-iqejv0fQQeuonpKF) and remove the test scenes with the following command:\r\n    ```\r\n    unzip real_iconic_noface.zip\r\n    cd real_iconic_noface/\r\n    rm -rf data2_fernvlsb data2_hugetrike data2_trexsanta data3_orchid data5_leafscene data5_lotr data5_redflower\r\n    ```\r\n    Then, set the corresponding path as  `llff_path` in the [config_general.txt](configs/config_general.txt) file.\r\n\r\n* Collected scenes from [IBRNet](https://github.com/googleinterns/IBRNet) ([Subset1](https://drive.google.com/file/d/1rkzl3ecL3H0Xxf5WTyc2Swv30RIyr1R_/view?usp=sharing) and [Subset2](https://drive.google.com/file/d/1Uxw0neyiIn3Ve8mpRsO6A06KfbqNrWuq/view?usp=sharing)). Set the corresponding paths as  `ibrnet1_path` and `ibrnet2_path` in the [config_general.txt](configs/config_general.txt) file.\r\n\r\nAlso, download `nerf_llff_data.zip` and `nerf_synthetic.zip` from [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) for validation and testing and set their corresponding paths as  `llff_test_path` and `nerf_path` in the [config_general.txt](configs/config_general.txt) file.\r\n\r\nOnce all the datasets are available, train the network from scratch with the following command:\r\n```\r\npython run_geo_nerf.py --config configs/config_general.txt\r\n```\r\n### Contact\r\nYou can contact the author through email: mohammad.johari At idiap.ch.\r\n\r\n## Citing\r\nIf you find our work useful, please consider citing:\r\n```BibTeX\r\n@inproceedings{johari-et-al-2022,\r\n  author = {Johari, M. and Lepoittevin, Y. and Fleuret, F.},\r\n  title = {GeoNeRF: Generalizing NeRF with Geometry Priors},\r\n  booktitle = {Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)},\r\n  year = {2022}\r\n}\r\n```\r\n\r\n### Acknowledgement\r\nThis work was supported by ams OSRAM.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidiap%2FGeoNeRF","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidiap%2FGeoNeRF","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidiap%2FGeoNeRF/lists"}