{"id":19401179,"url":"https://github.com/google-research/sparf","last_synced_at":"2025-05-16T18:03:26.770Z","repository":{"id":156428894,"uuid":"632945930","full_name":"google-research/sparf","owner":"google-research","description":"This is the official code release for SPARF: Neural Radiance Fields from Sparse and Noisy Poses [CVPR 2023-Highlight]","archived":false,"fork":false,"pushed_at":"2025-03-21T22:18:34.000Z","size":1275,"stargazers_count":295,"open_issues_count":5,"forks_count":15,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-04-06T11:07:24.509Z","etag":null,"topics":["nerf","novel-view-synthesis","pose-refinement"],"latest_commit_sha":null,"homepage":"https://prunetruong.com/sparf.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-26T13:03:51.000Z","updated_at":"2025-04-02T12:38:25.000Z","dependencies_parsed_at":"2024-03-09T13:25:55.763Z","dependency_job_id":"cb01809c-2cec-45ca-b71c-6590be93fa9f","html_url":"https://github.com/google-research/sparf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fsparf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fsparf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fsparf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fsparf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/sparf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254582901,"owners_count":22095518,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nerf","novel-view-synthesis","pose-refinement"],"created_at":"2024-11-10T11:17:27.278Z","updated_at":"2025-05-16T18:03:26.714Z","avatar_url":"https://github.com/google-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SPARF: Neural Radiance Fields from Sparse and Noisy Poses\n\n*This is not an officially supported Google product.*\n\nThis repository contains the code for the paper:\n[SPARF: Neural Radiance Fields from Sparse and Noisy Poses](https://arxiv.org/abs/2211.11738). In CVPR, 2023 (Highlight). \n\nAuthors: [Prune Truong](https://prunetruong.com/),\n[Marie-Julie Rakotosaona](http://www.lix.polytechnique.fr/Labo/Marie-Julie.RAKOTOSAONA/),\n[Fabian Manhardt](https://campar.in.tum.de/Main/FabianManhardt),\nand [Federico Tombari](https://federicotombari.github.io/)   \n\n\n\\[[arXiv preprint](https://arxiv.org/abs/2211.11738)\\]\n\\[[Website](https://prunetruong.com/sparf.github.io/)\\]\n\\[[Youtube teaser](https://www.youtube.com/watch?v=ARMKrcJlULE)\\]\n\n\nOur approach SPARF produces realistic novel-view rendering given as few as 2 or 3 input images, with noisy camera poses. We add two novel constraints into the pose-NeRF optimization: the multi-view correspondence loss and the depth-consistency loss. \n\n![](docs/sparf_method.jpg)\n\n\n\nPlease contact Prune Truong (prune.truong@vision.ee.ethz.ch) if you have any questions!\n\n\nWe provide PyTorch code for all experiments: BARF/SPARF for joint pose-NeRF training, NeRF/SPARF when considering fixed ground-truth poses as input.\n\n\n--------------------------------------\n## Citation\n\nIf you find our code useful for your research, please cite\n```\n@inproceedings{sparf2023,\n  title={SPARF: Neural Radiance Fields from Sparse and Noisy Poses},\n  author = {Truong, Prune and Rakotosaona, Marie-Julie and Manhardt, Fabian and Tombari, Federico},\n  publisher = {{IEEE/CVF} Conference on Computer Vision and Pattern Recognition, {CVPR}},\n  year = {2023}\n}\n```\n\n--------------------------------------\n\n## Installation\n\nThis code is developed with Python3 (`python3`) with Cuda 11.3. All models were trained on single A100 (or V100) GPU with 20GB of memory. This code does NOT support multi-GPU training. \nPlease use the following command for installation. \n\n* It is recommended to use [Anaconda](https://www.anaconda.com/products/individual) to set up the environment. Install the dependencies and activate the environment `sparf-env` with\n```bash\nconda create -n sparf-env python=3\nconda activate sparf-env\npip install -r requirements.txt\n```\nFor the specific versions of the packages used to develop this model, run the following instead. \n```bash\npip install -r requirements_w_version.txt\n```\n\n* Make sure to the dependencies of the submodules are downloaded. Everything should be included in the provided requirement file, except for cupy:\nWe use [PDC-Net](https://arxiv.org/abs/2101.01710) to extract correspondences between the views, for which CuPy is a required dependency. It can be installed using pip install cupy or alternatively using one of the provided binary packages as outlined in the CuPy repository. For another CUDA version, change accordingly.\n```bash\npip install cupy-cuda113 --no-cache-dir \n```\n\n* Initialize the external submodule dependencies with\n```bash\ngit submodule update --init --recursive\ngit submodule update --recursive --remote\n```\n\n* Optional: The submodule contains lots of unnecessary files. You can suppress them by running\n```bash\nbash third_party/remove_unused_files.sh\n```\n\n* Create source/admin/local.py by running the following command and update the paths to the datasets and workdirs. \nWe provide an example admin/local_example.py where all datasets are stored in data/. \n```bash\npython -c \"from source.admin.environment import create_default_local_file; create_default_local_file()\"\n```\n\n* Download the pre-trained model of [PDC-Net](https://arxiv.org/abs/2101.01710) [here](https://drive.google.com/file/d/1nOpC0MFWNV8N6ue0csed4I2K_ffX64BL/view). You will need to change the path to the pre-trained model in [train_settings/default_config.py](https://github.com/google-research/sparf/blob/main/train_settings/default_config.py#L190) `cfg.flow_ckpt_path` (L.190). \n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003eMake sure the correspondence network is installed and running correctly\u003c/b\u003e\u003c/summary\u003e\n\nRun the following command\n```bash\npython third_party/test_pdcnet_installation.py\n```\nYou should obtain the following image as third_party/test_pdcnet.png\n\n![](docs/test_pdcnet.png)\n\n\n\u003c/details\u003e\n\n--------------------------------------\n\n## Datasets\n\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003eDTU\u003c/b\u003e\u003c/summary\u003e\n\n* Images: We use the DTU dataset, processed by PixelNeRF, where the images are processed and resized to 300 x 400.\nDownload the data [here](https://drive.google.com/drive/folders/1PsT3uKwqHHD2bEEHkIXB99AlIjtmrEiR?usp=sharing). \n\n* Mask Data: For evaluation, we report also masked metrics. For this, we use the object masks provided by DVR, IDR and RegNeRF. [RegNeRF](https://github.com/google-research/google-research/tree/master/regnerf) provides the full mask data (for the test splits), which you can download [here](https://drive.google.com/file/d/1Yt5T3LJ9DZDiHbtd9PDFNHqJAd7wt-_E/view?usp=sharing). \n\n* Ground-truth depth maps: For evaluation, we report the depth error. For this, we download the [depth maps](https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/cascade-stereo/CasMVSNet/dtu_data/dtu_train_hr/Depths_raw.zip). They are extracted from [MVSNeRF](https://github.com/apchenstu/mvsnerf#:~:text=training%20data%20and-,Depth_raw,-from%20original%20MVSNet).  \n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003eLLFF\u003c/b\u003e\u003c/summary\u003e\n\nThe LLFF real-world data can be found in the [NeRF Google Drive](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1).\nFor convenience, you can download them with the following script: (under this repo)\n```bash\ngdown --id 16VnMcF1KJYxN9QId6TClMsZRahHNMW5g # download nerf_llff_data.zip\nunzip nerf_llff_data.zip\nrm -f nerf_llff_data.zip\nmv nerf_llff_data data/llff\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003eReplica\u003c/b\u003e\u003c/summary\u003e\n\nYou can download the replica dataset with the following script:\n```bash\n# you can also download the Replica.zip manually through\n# link: https://caiyun.139.com/m/i?1A5Ch5C3abNiL password: v3fY (the zip is split into smaller zips because of the size limitation of caiyun)\nwget https://cvg-data.inf.ethz.ch/nice-slam/data/Replica.zip\nunzip Replica.zip\n  ```\n\u003c/details\u003e\n\n--------------------------------------\n\n\n## I. Running the code\n\n### Training: Quick start\n\nThe installation should have generated a local configuration file \"admin/local.py\". \nIn case the file was not generated, run \n```python -c \"from source.admin.environment import create_default_local_file; create_default_local_file()\"```to generate it. \n* Next, set the paths to the training workspace `self.workspace_dir`, i.e. the directory where the model weights and checkpoints will be saved. \n* Set the path to `self.tensorboard_dir`, where the tensorboard files will be stored \n* Set the path to `self.log_dir`, where the logs will be stored\n* Set the path to `self.eval_dir`, where the test results will be stored. \n* Also set the paths to the datasets you want to use (and which should be downloaded beforehand, see above). \n\n* Set the path to the [PDC-Net](https://arxiv.org/abs/2101.01710) pre-trained model (previously downloaded) in train_settings/default_config.py `cfg.flow_ckpt_path`. \n\n        \nIf all the dependencies have been correctly installed, you can train/evaluate a network using the run_trainval.py script in the correct conda environment. All checkpointing, logging and saving metrics should be done automatically. \n\n```bash\nconda activate sparf-env\n# Selecting \u003ctrain_module\u003e \u003ctrain_name\u003e \u003cnbr_input_views\u003e \u003cscene_name\u003e\npython run_trainval.py \u003ctrain_module\u003e \u003ctrain_name\u003e --train_sub \u003cnbr_input_views\u003e --scene \u003cscene_name\u003e\n```\nHere, train_module is the sub-module inside train_settings and train_name is the name of the train setting file to be used.\n\nThe snapshots along with a video of the RGB and depth renderings will be saved in the directory `workspace_dir/\u003ctrain_module\u003e/subset_\u003cnbr_input_views\u003e/\u003cscene_name\u003e/\u003ctrain_name\u003e`.\n\nThe corresponding tensorboard file will be stored in `tensorboard_dir/\u003ctrain_module\u003e/subset_\u003cnbr_input_views\u003e/\u003cscene_name\u003e/\u003ctrain_name\u003e`.\n\nwhere `workspace_dir` and `tensorboard_dir` were set in the `source.admin.local.py` file. \n\n\nRunning this command directly computes results on the test set at the end of training, which will be stored in `eval_dir/dataset_name/\u003ctrain_module\u003e/subset_\u003cnbr_input_views\u003e/\u003cscene_name\u003e/\u003ctrain_name\u003e`.\nIf you wish to recompute them, or re-generate the video rendering, run: \n\n```bash\n# Selecting \u003ctrain_module\u003e \u003ctrain_name\u003e \u003cnbr_input_views\u003e \u003cscene_name\u003e\n# to regenerate the test metrics \npython run_trainval.py \u003ctrain_module\u003e \u003ctrain_name\u003e --train_sub \u003cnbr_input_views\u003e --scene \u003cscene_name\u003e --test_metrics_only True \n\n# to regenerate the video of the renderings\npython run_trainval.py \u003ctrain_module\u003e \u003ctrain_name\u003e --train_sub \u003cnbr_input_views\u003e --scene \u003cscene_name\u003e --render_video_only True \n\n```\n\n\u003cbr /\u003e\n\n### Training: Example \n\nThe configs are found in train_settings\n\nFor example, you can train using the included default SPARF settings for joint pose-nerf training on DTU, starting from initial noisy poses (3 views) by running:\n```bash\npython run_trainval.py joint_pose_nerf_training/dtu sparf --train_sub 3 --scene scan82\n```\nAll the snapshots will be stored in the directory `workspace_dir/joint_pose_nerf_training/dtu/subset_3/scan82/sparf`. All the tensorboard will be stored in `tensorboard_dir/joint_pose_nerf_training/dtu/subset_3/scan82/sparf`.\n\n\nRunning this command directly computes results on the test set at the end of training, which will be stored in `eval_dir/dtu/joint_pose_nerf_training/dtu/subset_3/scan82/sparf`.\n\n\n\u003cbr /\u003e\n\n### Visualizing the results\n  We have included code to visualize the training over TensorBoard.\n  The TensorBoard events include the following:\n  - **SCALARS**: the rendering losses and PSNR over the course of optimization. For joint pose-NeRF optimization, the rotational/translational errors with respect to the given poses are also computed. The validation metrics are also logged. \n  - **IMAGES**: visualization of the RGB images and the RGB/depth rendering. For joint pose-NeRF optimization, we also plot the optimized and ground-truth poses. \n  \n\n  At the end of the training, a video of the RGB and depth renderings is created, and saved with the snapshots. The poses used for the rendering are jitters of the optimized (or ground-truth) input poses. \n\n\n\u003cbr /\u003e\n\n### Evaluation of pre-trained models \n\nThe computation of the test metrics is included at the end of the training. But it must correspond\nto an existing train_settings file. To evaluate any given pre-trained models (associated with its\noptions.yaml file), run: \n```bash\n# Selecting \u003cckpt_dir\u003e \u003cout_dir\u003e \u003cexpname\u003e\npython eval.py --ckpt_dir \u003cckpt_dir\u003e --out_dir \u003cout_dir\u003e --expname \u003cexpname\u003e --plot True\n\n```\n\nThis will save the metrics file at `\u003cout_dir\u003e/\u003cexpname\u003e.json`. if `--plot` is True, figures of the renderings are saved in `\u003cout_dir\u003e/\u003cexpname\u003e/`. \n\u003cckpt_dir\u003e is the path to a **directory** containing a checkpoint (the latest will automatically be loaded) and an option file named `options.yaml`. \n\nWe provide pre-trained models for some of the experiments and some of the scenes [here](https://drive.google.com/drive/folders/1b3atnbgyR8macgZCvL4OneoLgrOMt6Mz?usp=share_link). \n\nTo make sure the code is running fine, for joint_pose_nerf_training/dtu/scan82, you should get similar metrics (with test-time optimization):\n  \n|              | Rot. error | Trans. error | PSNR (masked) | SSIM (masked) | LPIPS (masked) | Depth err.\n|--------------|--------|---------|---------|--------------|---------|--------------|\n| SPARF (Ours)  |  0.70 | 0.0097 | 17.58 (17.36) | 0.82 (0.91) | 0.19 (0.07) | 0.15\n\n\n\u003cbr /\u003e\n\n### Reproducing results from the publication \n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003eJoint pose-NeRF optimization on DTU from noisy poses (Tab. 4)\u003c/b\u003e\u003c/summary\u003e\n\nIn Tab. 4 of the main paper, we present results of the joint pose-NeRF optimization on DTU, starting from 3 images with noisy poses. To reproduce those results, run these commands for each of the 15 scenes. The results presented are the average over all scenes. \n```bash\n# \u003cSCENE\u003e is specific to datasets\n# DTU (\u003cSCENE\u003e={'scan8', 'scan21', 'scan30', 'scan31','scan34', 'scan38','scan40','scan41','scan45','scan55','scan63','scan82','scan103','scan110','scan114'})\n\n# SPARF\npython run_trainval.py joint_pose_nerf_training/dtu sparf --scene=\u003cSCENE\u003e --train_sub 3\n\n# SPARF without depth-consistency loss\n# Note that the depth-consistency loss adds training time and only leads to \n# a minor improvement. We therefore also include a version without the depth\n# consistency loss \npython run_trainval.py joint_pose_nerf_training/dtu sparf_wo_depth_cons_loss --scene=\u003cSCENE\u003e --train_sub 3\n\n# Baseline BARF\npython run_trainval.py joint_pose_nerf_training/dtu barf --scene=\u003cSCENE\u003e --train_sub 3\n```\n\nTo make sure the code is running fine, for scene='scan82', you should get similar metrics (with test-time optimization):\n  \n|              | Rot. error | Trans. error | PSNR (masked) | SSIM (masked) | LPIPS (masked) | Depth err.\n|--------------|--------|---------|---------|--------------|---------|--------------|\n| BARF | 13.28 | 0.41 | 7.79 (6.11) | 0.45 (0.77) | 0.67 (0.28) | 1.53\n| SPARF (Ours)  |  0.70 | 0.0097 | 17.58 (17.36) | 0.82 (0.91) | 0.19 (0.07) | 0.15\n\nScene 'scan30' is an example of failure cases, because no correspondences can be reliably extracted. \n\n\u003cbr /\u003e\u003cbr /\u003e\n\nWe provide pre-trained models of our SPARF with 3 input views for all 15 scenes [here](https://drive.google.com/drive/folders/1s0HcDEgEVFX7-rgnUgsKAolvElGNT1R8?usp=share_link). To evaluate them, run\n```bash\n# Selecting \u003cckpt_dir\u003e \u003cout_dir\u003e \u003cexpname\u003e\npython eval.py --ckpt_dir \u003cckpt_dir\u003e --out_dir \u003cout_dir\u003e --expname \u003cexpname\u003e --plot True\n```\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003eJoint pose-NeRF optimization on LLFF from identity poses (Tab. 5)\u003c/b\u003e\u003c/summary\u003e\n\nIn Tab. 5 of the main paper, we present results of the joint pose-NeRF optimization on LLFF, given 3 images with identity initial poses. To reproduce those results, run these commands for each of the 8 scenes. The results presented are the average over all scenes. \n```bash\n# \u003cSCENE\u003e is specific to datasets\n# LLFF (\u003cSCENE\u003e={'orchids', 'horns', 'trex', 'fern', 'flower', 'leaves', 'room', 'fortress'})\n\n# SPARF\npython run_trainval.py joint_pose_nerf_training/llff sparf --scene=\u003cSCENE\u003e --train_sub 3\n\n# SPARF without depth-consistency loss\npython run_trainval.py joint_pose_nerf_training/llff sparf_wo_depth_cons_loss --scene=\u003cSCENE\u003e --train_sub 3\n\n# Baseline BARF\npython run_trainval.py joint_pose_nerf_training/llff barf --scene=\u003cSCENE\u003e --train_sub 3\n```\n\nTo make sure the code is running fine, for scene='horns', you should get similar metrics (with test-time optimization):\n  \n|              | Rot. error | Trans. error | PSNR | SSIM | LPIPS |\n|--------------|--------|---------|---------|--------------|-----|\n| BARF | 5.53 | 0.326 | 14.34 | 0.34 | 0.54 | \n| SPARF (Ours)  |  0.027   |  0.002| 18.94 | 0.61 | 0.33\n\u003c/details\u003e\n\n\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003eJoint pose-NeRF optimization on Replica from COLMAP poses (Tab. 6)\u003c/b\u003e\u003c/summary\u003e\n\nIn Tab. 6 of the main paper, we present results of the joint pose-NeRF optimization on Replica, starting from 3 images with initial poses obtained by COLMAP with PDC-Net matches. To reproduce those results, run these commands for each of the 7 scenes. The results presented are the average over all scenes. \n```bash\n# \u003cSCENE\u003e is specific to datasets\n# Replica (\u003cSCENE\u003e={'room0', 'room1', 'room2',  'office0', 'office1', 'office2', 'office3'})\n\n# SPARF\npython run_trainval.py joint_pose_nerf_training/replica sparf --scene=\u003cSCENE\u003e --train_sub 3\n\n# SPARF without depth-consistency loss\npython run_trainval.py joint_pose_nerf_training/replica sparf_wo_depth_cons --scene=\u003cSCENE\u003e --train_sub 3\n\n# Baseline BARF\npython run_trainval.py joint_pose_nerf_training/replica barf --scene=\u003cSCENE\u003e --train_sub 3\n```\n\nTo make sure the code is running fine, for scene='office0', you should get similar metrics (with test-time optimization):\n  \n|              | Rot. error | Trans. error | PSNR | SSIM | LPIPS | Depth err. |\n|--------------|--------|---------|---------|--------------|---------|--------------|\n| BARF | 5.37 | 0.28 | 22.33 | 0.72 | 0.30 | 0.59 | \n| SPARF (Ours)  |    0.58 |0.010 | 28.38 | 0.90 | 0.13 | 0.36 |\n\n\u003c/details\u003e\n\n\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e NeRF optimization on DTU with fixed ground-truth poses (Tab. 7)\u003c/b\u003e\u003c/summary\u003e\n\nIn Tab. 7 of the main paper, we present results of NeRF-based approaches, trained with fixed ground-truth camera poses. To reproduce those results, run these commands for each of the 15 scenes. The results presented are the average over all scenes. \n```bash\n# \u003cSCENE\u003e is specific to datasets\n# DTU (\u003cSCENE\u003e={'scan8', 'scan21', 'scan30', 'scan31','scan34', 'scan38','scan40','scan41','scan45','scan55','scan63','scan82','scan103','scan110','scan114'})\n\n# SPARF\npython run_trainval.py nerf_training_w_gt_poses/dtu sparf --scene=\u003cSCENE\u003e --train_sub 3\n\n\n# Baseline NeRF\npython run_trainval.py nerf_training_w_gt_poses/dtu nerf --scene=\u003cSCENE\u003e --train_sub 3\n```\n\nTo make sure the code is running fine, for scene='scan82', you should get similar metrics:\n  \n|              | PSNR (masked) | SSIM (masked) | LPIPS (masked) | Depth err. |\n|--------------|--------|---------|---------|--------------|\n| NeRF | 4.57 (5.36) | 0.28 (0.77) | 0.28 (0.32) | 1.35 |\n| SPARF (Ours)  |  18.42 (21.71) | 0.87 (0.95) | 0.16 (0.04)  | 0.24  |\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e NeRF optimization on LLFF with fixed ground-truth poses (Tab. 7)\u003c/b\u003e\u003c/summary\u003e\n\nIn Tab. 7 of the main paper, we present results of NeRF-based approaches, trained with fixed ground-truth camera poses. To reproduce those results, run these commands for each of the 8 scenes. The results presented are the average over all scenes. Almost the same results are obtained using hierarchical sampling, or with only a coarse sampling. \n```bash\n# \u003cSCENE\u003e is specific to datasets\n# LLFF (\u003cSCENE\u003e={'orchids', 'horns', 'trex', 'fern', 'flower', 'leaves', 'room', 'fortress'})\n\n# SPARF\npython run_trainval.py nerf_training_w_gt_poses/llff sparf --scene=\u003cSCENE\u003e --train_sub 3\n\n\n# Baseline NeRF\npython run_trainval.py nerf_training_w_gt_poses/llff nerf --scene=\u003cSCENE\u003e --train_sub 3\n```\n\nTo make sure the code is running fine, for scene='horns', you should get similar metrics:\n  \n|              | PSNR (masked) | SSIM (masked) | LPIPS (masked) | \n|--------------|--------|---------|---------|\n| NeRF |  13.21 | 0.26 | 0.61 | \n| SPARF (Ours)  | 19.39 | 0.64 | 0.29 | \n\n\u003c/details\u003e\n\n--------------------------------------\n\n\n## II. Using a pre-trained model\n\nYou can find our pre-trained models (from running this repo), following the same structures as our train_settings files, [here](https://drive.google.com/drive/folders/1b3atnbgyR8macgZCvL4OneoLgrOMt6Mz?usp=share_link).\nAfter downloading the checkpoints, you can evaluate the model and render test images as indicated above.\n\n\n--------------------------------------\n\n\n## III. Downloading model predictions\n\nIf you don't want to run the code, you can also directly download the renderings of our models for the test set poses using the following link: https://drive.google.com/drive/folders/1lHryExsutZsbcKJlzO7QKM34YlUwKSSx?usp=share_link. The structure follows that of the train_settings files. They correspond to the renderings used in the paper, so they will be a bit different compared to the ones obtained when running the provided pre-trained models (for which the training was done with this codebase). \n\n--------------------------------------\n## IV. Codebase structure\n\nFor details on the dataloader and how to use your own, refer to [this doc](https://github.com/google-research/sparf/blob/main/docs/code_documentation.md). \n\nThe framework in `source/` consists of the following sub-modules.\n\n* training: \n    * core: the losses, sampling strategy and metric files\n    * engine: Classes for main training routine \n    * base.py: Base class for the NeRF/pose trainer. All the others inherit from it. \n    * **nerf_trainer.py**: The main engine for training a NeRF model, considering **fixed ground-truth poses**. Only the NeRF model is optimized. It inherits base.py. \n    * **joint_pose_nerf_trainer.py**: The main engine for training of the NeRF model jointly with refining initial noisy camera poses. This corresponds to SPARF's main use case. Optimized are the NeRF model and the camera pose estimates. It inherits nerf_trainer.py. \n    * **nerf_trainer_w_fixed_colmap_poses.py**: The main engine for training a NeRF model, considering fixed poses, obtained from COLMAP (or some other initialization). The poses are therefore most likely noisy. Only the NeRF model is optimized. This is to show comparison to the joint refinement.  It inherits nerf_trainer.py. \n* train_settings: Contains config files, specifying the training of a network. For each experiment, we create a new config file in train_settings. The logging and saving will adapt accordingly so only the config file needs to be created in train_settings. \n* admin: Includes functions for loading networks, tensorboard etc. and also contains environment settings.\n* datasets: Contains integration of a number of datasets. All datasets inherent from the base Dataset class in datasets/base.py\n* utils: Contains functions for logging, visualizations, geometric operations and colmap initalization. \n    * geometry: Utils for geometric transformations \n    * colmap_initialization: Utils to initializes poses with COLMAP, run with different matchers\n* models: Contains NeRF model definition, different pose parametrization and the correspondence network wrapper. \n\n\n\nSome tips on using and understanding the codebase:\n- Throughout the code base, the pose convention is [right, down, forward]. This is the standard form of camera extrinsic projection matrices in the notation of multi-view geometry (opencv, colmap). It transforms a world-frame 3D point x to the camera frame and projects it to pixel coordinates u with the standard equation u = K (R^{w2c} x + t^{w2c}). \n- We create a config file for each experiments (including changing each hyper-parameters). If you want to create a different experiment, create a new config file with a different name. \n- The configuration will be loaded as the variable `opt` and `self.settings` throughout the codebase.  \n- The data for the whole scene is loaded at the beginning of the training, after creating the dataset. Throughout the codebase, it is stored in `data_dict`. Important keys are 'image' (B, 3, H, W), 'pose' (w2c ground-truth poses in format (B, 3, 4)) and 'intr' (B, 3, 3). When the poses are optimized, the current estimate of the poses are stored in `data_dict.poses_w2c`. \n- The output of the NeRF model (computation graph for forward/backprop) is stored in `output_dict` throughout the codebase.\n- The loss factory can be found in [training/core/loss_factory.py](https://github.com/google-research/sparf/blob/main/source/training/core/loss_factory.py). In the config, the loss to use is defined as a string in `opt.loss_type`. The final loss can be a combination of multiple loss functions. The losses values are stored in a dict `loss_dict`. The weights of each are defined in `opt.loss_weight.\u003cname\u003e`. The final overall loss is automatically computed based on the defined `opt.loss_type` and `opt.loss_weight`. \nTo add a new loss function, create a new class with a main function `compute_loss()`. Add this new class to the loss factory. The output of the `compute_loss()` function should be `loss_dict`, `stats_dict` and `plotting_dict`. The actual loss value for backward is stored in `loss_dict.\u003cname\u003e`. You also need to add the new loss weight to `opt.loss_weight.\u003cname\u003e`. It will automatically be added to the overall loss and logged to Tensorboard.\n\n- If you are using a multi-GPU machine, you can add `--gpu=\u003cgpu_number\u003e` to specify which GPU to use. **Multi-GPU training/evaluation is currently not supported.**\n- The code was designed for a limited number of views. If loading many views instead and computing the correspondences exhaustively between the views, you might run into out-of-memory issues. \n- If restarting the training of an experiment, it will automatically load the last saved checkpoint. \n- (to be continued....)\n  \n\n\n\n--------------------------------------\n## V. License\nThis code is licensed under the Apache 2.0 License. See [LICENSE](https://github.com/google-research/sparf/blob/main/LICENSE) for more details.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fsparf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Fsparf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fsparf/lists"}