{"id":13961380,"url":"https://github.com/ActiveVisionLab/nerfmm","last_synced_at":"2025-07-21T06:30:38.669Z","repository":{"id":37713379,"uuid":"337025100","full_name":"ActiveVisionLab/nerfmm","owner":"ActiveVisionLab","description":"(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters","archived":false,"fork":false,"pushed_at":"2023-08-08T21:56:38.000Z","size":84788,"stargazers_count":552,"open_issues_count":2,"forks_count":55,"subscribers_count":20,"default_branch":"main","last_synced_at":"2024-08-09T17:29:35.016Z","etag":null,"topics":["nerf","neural-radiance-fields","novel-view-synthesis","pose-estimation"],"latest_commit_sha":null,"homepage":"https://nerfmm.active.vision","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ActiveVisionLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-02-08T09:37:47.000Z","updated_at":"2024-08-04T06:45:37.000Z","dependencies_parsed_at":"2024-01-15T03:59:25.069Z","dependency_job_id":"61f077fa-4cc0-496b-ab51-f4929695830a","html_url":"https://github.com/ActiveVisionLab/nerfmm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ActiveVisionLab%2Fnerfmm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ActiveVisionLab%2Fnerfmm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ActiveVisionLab%2Fnerfmm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ActiveVisionLab%2Fnerfmm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ActiveVisionLab","download_url":"https://codeload.github.com/ActiveVisionLab/nerfmm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226878782,"owners_count":17696718,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nerf","neural-radiance-fields","novel-view-synthesis","pose-estimation"],"created_at":"2024-08-08T17:01:05.755Z","updated_at":"2024-11-28T07:30:49.966Z","avatar_url":"https://github.com/ActiveVisionLab.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# NeRF--: Neural Radiance Fields Without Known Camera Parameters\n\n**[Project Page](https://nerfmm.active.vision/) |\n[Latest arXiv](https://arxiv.org/abs/2102.07064) |\n[Colab Notebook](https://colab.research.google.com/drive/1pRljG5lYj_dgNG_sMRyH2EKbpq3OezvK?usp=sharing) | \n[LLFF Released Data](https://www.robots.ox.ac.uk/~ryan/nerfmm2021/nerfmm_release_data.tar.gz) |\n[BLEFF Data](https://www.robots.ox.ac.uk/~ryan/nerfmm2021/BLEFF.tar.gz) |\n[ckpts](https://www.robots.ox.ac.uk/~ryan/nerfmm2021/nerfmm_ckpts.tar.gz)**\n\n[Zirui Wang¹](https://scholar.google.com/citations?user=zCBKqa8AAAAJ\u0026hl=en), \n[Shangzhe Wu²](http://elliottwu.com), \n[Weidi Xie²](https://weidixie.github.io/weidi-personal-webpage/), \n[Min Chen³](https://sites.google.com/site/drminchen/home), \n[Victor Adrian Prisacariu¹](http://www.robots.ox.ac.uk/~victor/). \n\n[¹Active Vision Lab (AVL)](https://www.robots.ox.ac.uk/~lav/) + \n[²Visual Geometry Group (VGG)](https://www.robots.ox.ac.uk/~vgg/) + \n[³e-Research Centre](https://www.oerc.ox.ac.uk/), \nUniversity of Oxford.\n\n\n## Update\n#### 8 Aug 2023\nSome BLEFF-related info discussed in issues:\n* [#34](/../../issues/34): training config.\n* [#37](/../../issues/37): coordinate system.\n* [#38](/../../issues/38): loading script.\n* [#41](/../../issues/41): non-normalised rotation matrix issue.\n\n#### 11 Aug 2022\nRelease our [checkpoints](https://www.robots.ox.ac.uk/~ryan/nerfmm2021/nerfmm_ckpts.tar.gz) on LLFF dataset.\n#### 7 Apr 2022\nUpdate our [arXiv paper](https://arxiv.org/abs/2102.07064) with\n  1. A breaking point analysis for the camera parameter estimation. In short, our method can tolerate ±20 degrees of rotation variance and ±20% of translation variance.\n  2. A customised dataset, which we named [Blender Forward Facing (BLEFF)](https://www.robots.ox.ac.uk/~ryan/nerfmm2021/BLEFF.tar.gz). We will provide the dataloader file to load this data soon.\n\n#### 20 Apr 2021\nInitial code release, corresponding to [arXiv paper v3](https://arxiv.org/abs/2102.07064v3).\n\n---\n## Overview\nWe provide 3 training targets in this repository, under the `tasks` directory:\n1. `tasks/nerfmm/train.py`: This is our main training script for the [NeRF-LLFF dataset](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1), which estimates camera poses, focal lenghts and a NeRF jointly and monitors the absolute trajectory error (ATE) between our estimation of camera parameters and COLMAP estimation during training. This target can also start training from a COLMAP initialisation and refine the COLMAP camera parameters.\n2. `tasks/refine_nerfmm/train.py`: This is the training script that refines a pretrained nerfmm system.\n3. `tasks/any_folder/train.py`: This is a training script that takes a folder that contains forward-facing images and trains with our nerfmm system without making any comparison with COLMAP. It is similar to what we offer in our [CoLab notebook](https://colab.research.google.com/drive/1pRljG5lYj_dgNG_sMRyH2EKbpq3OezvK?usp=sharing) and we treat this `any_folder` target as a **playgraound**, where users can try novel view synthesis by just providing an image folder and do not care how the camera parameter estimation compares with COLMAP. \n\nFor each target, we provide relevant utilities to evaluate our system. Specifically, \n- for the `nerfmm` target, we provide three utility files:\n    - `eval.py` to evaluate image rendering quality on validation splits with PSNR, SSIM and LPIPS, i.e, results in Table 1.\n    - `spiral.py` to render novel views using a spiral camera trajectory, i.e. results in Figure 1.\n    - `vis_learned_poses.py` to visualise our camera parameter estimation with COLMAP estimation in 3D. It also computes ATE between them, i.e. E1 in Table 2.\n- for the `refine_nerfmm` target, all utilities in `nerfmm` target above are compatible with `refine_nerfmm` target, since it just refines a pretrained nerfmm system.\n- for the `any_folder` target, it has its own `spiral.py` and `vis_learned_poses.py` utilities, as it does not compare with COLMAP. It does not have a `eval.py` file as this target is treated as a playground and does not split images to train/validation sets. It only provides novel view synthesis results via the `spiral.py` file.\n    \n---\n\n## Table of Content\n- [Environment](#Environment)\n- [Get Data](#Get-Data)\n- [Training](#Training)\n- [Evaluation](#Evaluation)\n- [Citation](#citation)\n\n## Environment\n\nWe provide a `environment.yml` file to set up a `conda` environment:\n\n```sh\ngit clone https://github.com/ActiveVisionLab/nerfmm.git\ncd nerfmm\nconda env create -f environment.yml\n```\n\nGenerally, our code should be able to run with any `pytorch \u003e= 1.1` .\n\n(Optional) Install `open3d` for visualisation. You might need a physical monitor to install this lib.\n```sh\npip install open3d\n```\n\n## Get Data\n\n#### LLFF Dataset\nWe use the [LLFF dataset](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) with two small structural changes: \n1. We remove their `image_4` and `image_8` folder and downsample images to any desirable resolution during data loading `dataloader/with_colmap.py`, by calling PyTorch's `interpolate` function.\n2. We explicitly generate two txt files for train/val image ids. i.e. take every 8th image as the validation set, as in the official NeRF train/val split. The only difference is that we store them as txt files while NeRF split them during data loading. The file produces these two txt files is `utils/split_dataset.py`.\n\nIn addition to the NeRF-LLFF dataset, we provide two demo scenes to demonstrate how to use the `any_folder` target.\n   \nWe pack the re-structured LLFF data and our data to a tar ball (~1.8G), to get it, run:\n```shell\nwget https://www.robots.ox.ac.uk/~ryan/nerfmm2021/nerfmm_release_data.tar.gz\ntar -xzvf path/to/the/tar.gz\n```\n\n#### BLEFF Dataset\nThere are mainly two reasons that motivate us to create BLEFF:\n1. We need to evaluate both the camera parameter estimation accuracy and image rendering quality at the same time.\n2. To facilitate the analysis of the robustness of our method, a dataset with progressively increasing pose perturbation levels is required.\n\nTo that end, we introduce a synthetic dataset BLEFF, containing 14 path-traced scenes, with each rendered in multiple\nlevels of rotation and translation perturbations. Those scenes are modified and rendered with \nopen-source blender files on [blendswap](https://www.blendswap.com/) and the license info can be found in our supplementary file.\n\nUsage of our data:\n```shell\nwget https://www.robots.ox.ac.uk/~ryan/nerfmm2021/BLEFF.tar.gz\ntar -xzvf path/to/the/tar.gz\n```\n\n## Training\nWe show how to:\n1. train a `nerfmm` from scratch, i.e. initialise camera poses with identity matrices and focal lengths with image resolution:\n    ```shell\n   python tasks/nerfmm/train.py \\\n   --base_dir='path/to/nerfmm_release/data' \\\n   --scene_name='LLFF/fern'\n    ```\n2. train a `nerfmm` from COLMAP initialisation:\n    ```shell\n    python tasks/nerfmm/train.py \\\n   --base_dir='path/to/nerfmm_release/data' \\\n   --scene_name='LLFF/fern' \\\n   --start_refine_pose_epoch=1000 \\\n   --start_refine_focal_epoch=1000\n    ```\n   This command initialises a `nerfmm` target with COLMAP parameters, trains with them for 1000 epochs, and starts refining those parameters after 1000 epochs. \n3. train a `nerfmm` from a pretrained `nerfmm`:\n    ```shell\n    python tasks/refine_nerfmm/train.py \\\n   --base_dir='path/to/nerfmm_release/data' \\\n   --scene_name='LLFF/fern' --start_refine_epoch=1000 \\\n   --ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'\n    ```\n   This command initialises a `refine_nerfmm` target with a set of pretrained `nerfmm` parameters, trains with them for 1000 epochs, and starts refining those parameters after 1000 epochs.\n4. train an `any_folder` from scratch given an image folder:\n    ```shell\n    python tasks/any_folder/train.py \\\n   --base_dir='path/to/nerfmm_release/data' \\\n   --scene_name='any_folder_demo/desk'\n    ```\n   This command trains an `any_folder` target using a provided demo scene `desk`. \n\n(Optional) set a symlink to the downloaded data:\n```shell\nmkdir data_dir  # do it in this nerfmm repo\ncd data_dir\nln -s /path/to/downloaded/data ./nerfmm_release_data\ncd ..\n```\nthis can simplify the above training commands, for example:\n```shell\npython tasks/nerfmm/train.py\n```\n\n## Evaluation\n### Compute image quality metrics\nCall `eval.py` in `nerfmm` target:\n```shell\npython tasks/nerfmm/eval.py \\\n--base_dir='path/to/nerfmm_release/data' \\\n--scene_name='LLFF/fern' \\\n--ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'\n```\nThis file can be used to evaluate a checkpoint trained with `nerfmm` or `refine_nerfmm` target. For some scenes, you might need to tweak with `--opt_eval_lr` option to get the best results. Common values for `opt_eval_lr` are 0.01 / 0.005 / 0.001 / 0.0005 / 0.0001. The default value is 0.001. Overall, it finds validation poses that can produce highest PSNR on validation set while freezing NeRF and focal lengths. We do this because the learned camera pose space is different from the COLMAP estimated camera pose space.\n\n### Render novel views\nCall `spiral.py` in each target. The `spiral.py` in `nerfmm` is compatible with `refine_nerfmm` target:\n```shell\npython spiral.py \\\n--base_dir='path/to/nerfmm_release/data' \\\n--scene_name='LLFF/fern' \\\n--ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'\n```\n\n### Visualise estimated poses in 3D\nCall `vis_learned_poses.py` in each target. The `vis_learned_poses.py` in `nerfmm` is compatible with `refine_nerfmm` target:\n```shell\npython vis_learned_poses.py \\\n--base_dir='path/to/nerfmm_release/data' \\\n--scene_name='LLFF/fern' \\\n--ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'\n```\n\n---\n\n## Acknowledgement\n[Shangzhe Wu](http://elliottwu.com) is supported by Facebook Research. [Weidi Xie](https://weidixie.github.io/weidi-personal-webpage/) is supported by Visual AI (EP/T028572/1).\n\nThe authors would like to thank\n[Tim Yuqing Tang](https://scholar.google.co.uk/citations?user=kQB_dOoAAAAJ\u0026hl=en) for insightful discussions and proofreading.\n \nDuring our NeRF implementation, we referenced several open sourced NeRF implementations, and we thank their contributions. Specifically, we referenced functions from [nerf](https://github.com/bmild/nerf) and [nerf-pytorch](https://github.com/yenchenlin/nerf-pytorch), and borrowed/modified code from [nerfplusplus](https://github.com/Kai-46/nerfplusplus) and [nerf_pl](https://github.com/kwea123/nerf_pl). We especially appreciate the detailed code comments and git issue answers in [nerf_pl](https://github.com/kwea123/nerf_pl).\n\n## Citation\n```\n@article{wang2021nerfmm,\n  title={Ne{RF}$--$: Neural Radiance Fields Without Known Camera Parameters},\n  author={Zirui Wang and Shangzhe Wu and Weidi Xie and Min Chen and Victor Adrian Prisacariu},\n  journal={arXiv preprint arXiv:2102.07064},\n  year={2021}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FActiveVisionLab%2Fnerfmm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FActiveVisionLab%2Fnerfmm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FActiveVisionLab%2Fnerfmm/lists"}