{"id":13474437,"url":"https://rover-xingyu.github.io/L2G-NeRF/","last_synced_at":"2025-03-26T21:31:34.904Z","repository":{"id":64605510,"uuid":"568838814","full_name":"rover-xingyu/L2G-NeRF","owner":"rover-xingyu","description":"[CVPR 2023] L2G-NeRF: Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields","archived":false,"fork":false,"pushed_at":"2024-03-06T12:38:06.000Z","size":1495,"stargazers_count":250,"open_issues_count":4,"forks_count":6,"subscribers_count":30,"default_branch":"main","last_synced_at":"2025-03-26T12:03:01.927Z","etag":null,"topics":["nerf"],"latest_commit_sha":null,"homepage":"https://rover-xingyu.github.io/L2G-NeRF/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rover-xingyu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-21T14:16:49.000Z","updated_at":"2025-02-10T13:48:49.000Z","dependencies_parsed_at":"2024-01-18T18:25:25.648Z","dependency_job_id":"8c24d65c-17b8-462b-a467-944d03b679be","html_url":"https://github.com/rover-xingyu/L2G-NeRF","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rover-xingyu%2FL2G-NeRF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rover-xingyu%2FL2G-NeRF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rover-xingyu%2FL2G-NeRF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rover-xingyu%2FL2G-NeRF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rover-xingyu","download_url":"https://codeload.github.com/rover-xingyu/L2G-NeRF/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245738667,"owners_count":20664325,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nerf"],"created_at":"2024-07-31T16:01:12.332Z","updated_at":"2025-03-26T21:31:34.872Z","avatar_url":"https://github.com/rover-xingyu.png","language":"Python","funding_links":[],"categories":["Uncategorized","Papers","Tools, Pipeline \u0026 Utilities"],"sub_categories":["Uncategorized","Photogrammetry \u0026 3D Scanning"],"readme":"# L2G-NeRF: Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields\n**[Project Page](https://rover-xingyu.github.io/L2G-NeRF/) |\n[Paper](https://arxiv.org/pdf/2211.11505.pdf) |\n[Video](https://www.youtube.com/watch?v=y8XP9Umt6Mw)**\n\n[Yue Chen¹](https://scholar.google.com/citations?user=M2hq1_UAAAAJ\u0026hl=en), \n[Xingyu Chen¹](https://scholar.google.com/citations?user=gDHPrWEAAAAJ\u0026hl=en), \n[Xuan Wang²](https://scholar.google.com/citations?user=h-3xd3EAAAAJ\u0026hl=en),\n[Qi Zhang³](https://scholar.google.com/citations?user=2vFjhHMAAAAJ\u0026hl=en), \n[Yu Guo¹](https://scholar.google.com/citations?user=OemeiSIAAAAJ\u0026hl=en), \n[Ying Shan³](https://scholar.google.com/citations?user=4oXBp9UAAAAJ\u0026hl=en), \n[Fei Wang¹](https://scholar.google.com/citations?user=uU2JTpUAAAAJ\u0026hl=en). \n\n[¹Xi'an Jiaotong University](http://en.xjtu.edu.cn/),\n[²Ant Group](https://www.antgroup.com/en),\n[³Tencent AI Lab](https://ai.tencent.com/ailab/en/index/). \n\nThis repository is an official implementation of [L2G-NeRF](https://rover-xingyu.github.io/L2G-NeRF/) using [pytorch](https://pytorch.org/). \n\n# :computer: Installation\n\n## Hardware\n\n* We implement all experiments on a single NVIDIA GeForce RTX 2080 Ti GPU. \n* L2G-NeRF takes about 4.5 and 8 hours for training in synthetic objects and real-world scenes, respectively, while training BARF takes about 8 and 10.5 hours.\n\n## Software\n\n* Clone this repo by `git clone https://github.com/rover-xingyu/L2G-NeRF`\n* This code is developed with Python3. PyTorch 1.9+ is required. It is recommended use [Anaconda](https://www.anaconda.com/products/individual) to set up the environment, use `conda env create --file requirements.yaml python=3` to install the dependencies and activate it by `conda activate L2G-NeRF`\n--------------------------------------\n\n# :key: Training and Evaluation\n\n## Data download\n\nBoth the Blender synthetic data and LLFF real-world data can be found in the [NeRF Google Drive](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1).\nFor convenience, you can download them with the following script:\n  ```bash\n  # Blender\n  gdown --id 18JxhpWD-4ZmuFKLzKlAw-w5PpzZxXOcG # download nerf_synthetic.zip\n  unzip nerf_synthetic.zip\n  rm -f nerf_synthetic.zip\n  mv nerf_synthetic data/blender\n  # LLFF\n  gdown --id 16VnMcF1KJYxN9QId6TClMsZRahHNMW5g # download nerf_llff_data.zip\n  unzip nerf_llff_data.zip\n  rm -f nerf_llff_data.zip\n  mv nerf_llff_data data/llff\n  ```\n--------------------------------------\n\n## Running L2G-NeRF\nTo train and evaluate L2G-NeRF:\n```bash\n# \u003cGROUP\u003e and \u003cNAME\u003e can be set to your likes, while \u003cSCENE\u003e is specific to datasets\n\n# NeRF (3D): Synthetic Objects\n# Blender (\u003cSCENE\u003e={chair,drums,ficus,hotdog,lego,materials,mic,ship})\npython3 train.py \\\n--model=l2g_nerf --yaml=l2g_nerf_blender \\\n--group=exp_synthetic --name=l2g_lego \\\n--data.scene=lego --gpu=3 \\\n--data.root=/the/data/path/of/nerf_synthetic/ \\\n--camera.noise_r=0.07 --camera.noise_t=0.5\n\npython3 evaluate.py \\\n--model=l2g_nerf --yaml=l2g_nerf_blender \\\n--group=exp_synthetic --name=l2g_lego \\\n--data.scene=lego --gpu=3 \\\n--data.root=/the/data/path/of/nerf_synthetic/ \\\n--data.val_sub= --resume\n\n# NeRF (3D): Real-World Scenes\n# LLFF (\u003cSCENE\u003e={fern,flower,fortress,horns,leaves,orchids,room,trex})\npython3 train.py \\\n--model=l2g_nerf --yaml=l2g_nerf_llff \\\n--group=exp_LLFF --name=l2g_fern \\\n--data.scene=fern --gpu=3 \\\n--data.root=/the/data/path/of/nerf_llff_data/ \\\n--loss_weight.global_alignment=2\n\npython3 evaluate.py \\\n--model=l2g_nerf --yaml=l2g_nerf_llff \\\n--group=exp_LLFF --name=l2g_fern \\\n--data.scene=fern --gpu=3 \\\n--data.root=/the/data/path/of/nerf_llff_data/ \\\n--loss_weight.global_alignment=2 \\\n--resume\n\n# Neural image Alignment (2D): Rigid\n# use the image of “Girl With a Pearl Earring” renovation ©Koorosh Orooj (CC BY-SA 4.0) for rigid image alignment\npython3 train.py \\\n--model=l2g_planar --yaml=l2g_planar \\\n--group=exp_planar --name=l2g_girl  \\\n--warp.type=rigid --warp.dof=3 \\\n--data.image_fname=data/girl.jpg \\\n--data.image_size=[595,512] \\\n--data.patch_crop=[260,260] \\\n--seed=1 --gpu=3\n\n# Neural image Alignment (2D): Homography\n# use the image of “cat” from ImageNet for homography image alignment\npython3 train.py \\\n--model=l2g_planar --yaml=l2g_planar \\\n--group=exp_planar --name=l2g_cat \\\n--warp.type=homography --warp.dof=8 \\\n--data.image_fname=data/cat.jpg \\\n--data.image_size=[360,480] \\\n--data.patch_crop=[180,180] \\\n--gpu=0\n```\n--------------------------------------\n\n## Running BARF\nIf you want to train and evaluate the BARF extension of the original NeRF model that jointly optimizes poses (coarse-to-fine positional encoding):\n```bash\n# \u003cGROUP\u003e and \u003cNAME\u003e can be set to your likes, while \u003cSCENE\u003e is specific to datasets\n\n# NeRF (3D): Synthetic Objects\n# Blender (\u003cSCENE\u003e={chair,drums,ficus,hotdog,lego,materials,mic,ship})\npython3 train.py \\\n--model=barf --yaml=barf_blender \\\n--group=exp_synthetic --name=barf_lego \\\n--data.scene=lego --gpu=1 \\\n--data.root=/the/data/path/of/nerf_synthetic/ \\\n--camera.noise_r=0.07 --camera.noise_t=0.5\n\npython3 evaluate.py \\\n--model=barf --yaml=barf_blender \\\n--group=exp_synthetic --name=barf_lego \\\n--data.scene=lego --gpu=1 \\\n--data.root=/the/data/path/of/nerf_synthetic/ \\\n--data.val_sub= --resume\n\n# NeRF (3D): Real-World Scenes\n# LLFF (\u003cSCENE\u003e={fern,flower,fortress,horns,leaves,orchids,room,trex})\npython3 train.py \\\n--model=barf --yaml=barf_llff \\\n--group=exp_LLFF --name=barf_fern \\\n--data.scene=fern --gpu=1 \\\n--data.root=/the/data/path/of/nerf_llff_data/\n\npython3 evaluate.py \\\n--model=barf --yaml=barf_llff \\\n--group=exp_LLFF --name=barf_fern \\\n--data.scene=fern --gpu=1 \\\n--data.root=/the/data/path/of/nerf_llff_data/ \\\n--resume\n\n# Neural image Alignment (2D): Rigid\n# use the image of “Girl With a Pearl Earring” renovation ©Koorosh Orooj (CC BY-SA 4.0) for rigid image alignment\npython3 train.py \\\n--model=planar --yaml=planar \\\n--group=exp_planar --name=barf_girl \\\n--warp.type=rigid --warp.dof=3 \\\n--data.image_fname=data/girl.jpg \\\n--data.image_size=[595,512] \\\n--data.patch_crop=[260,260] \\\n--seed=1 --gpu=1\n\n# Neural image Alignment (2D): Homography\n# use the image of “cat” from ImageNet for homography image alignment\npython3 train.py \\\n--model=planar --yaml=planar \\\n--group=exp_planar --name=barf_cat \\\n--warp.type=homography --warp.dof=8 \\\n--data.image_fname=data/cat.jpg \\\n--data.image_size=[360,480] \\\n--data.patch_crop=[180,180] \\\n--gpu=2\n```\n--------------------------------------\n\n## Running Naive\nIf you want to train and evaluate the Naive extension of the original NeRF model that jointly optimizes poses (full positional encoding):\n\n```bash\n# \u003cGROUP\u003e and \u003cNAME\u003e can be set to your likes, while \u003cSCENE\u003e is specific to datasets\n\n# NeRF (3D): Synthetic Objects\n# Blender (\u003cSCENE\u003e={chair,drums,ficus,hotdog,lego,materials,mic,ship})\npython3 train.py \\\n--model=barf --yaml=barf_blender \\\n--group=exp_synthetic --name=nerf_lego \\\n--data.scene=lego --gpu=2 \\\n--data.root=/home/cy/PNW/datasets/nerf_synthetic/ \\\n--barf_c2f=null \\\n--camera.noise_r=0.07 --camera.noise_t=0.5\n\npython3 evaluate.py \\\n--model=barf --yaml=barf_blender \\\n--group=exp_synthetic --name=nerf_lego \\\n--data.scene=lego --gpu=2 \\\n--data.root=/home/cy/PNW/datasets/nerf_synthetic/ \\\n--barf_c2f=null \\\n--data.val_sub= --resume\n\n# NeRF (3D): Real-World Scenes\n# LLFF (\u003cSCENE\u003e={fern,flower,fortress,horns,leaves,orchids,room,trex})\npython3 train.py \\\n--model=barf --yaml=barf_llff \\\n--group=exp_LLFF --name=nerf_fern \\\n--data.scene=fern --gpu=2 \\\n--data.root=/home/cy/PNW/datasets/nerf_llff_data/ \\\n--barf_c2f=null\n\npython3 evaluate.py \\\n--model=barf --yaml=barf_llff \\\n--group=exp_LLFF --name=nerf_fern \\\n--data.scene=fern --gpu=2 \\\n--data.root=/home/cy/PNW/datasets/nerf_llff_data/ \\\n--barf_c2f=null --resume \n\n# Neural image Alignment (2D): Rigid\n# use the image of “Girl With a Pearl Earring” renovation ©Koorosh Orooj (CC BY-SA 4.0) for rigid image alignment\npython3 train.py \\\n--model=planar --yaml=planar \\\n--group=exp_planar --name=naive_girl \\\n--warp.type=rigid --warp.dof=3 \\\n--data.image_fname=data/girl.jpg \\\n--data.image_size=[595,512] \\\n--data.patch_crop=[260,260] \\\n--seed=1 --gpu=2 --barf_c2f=null\n\n# Neural image Alignment (2D): Homography\n# use the image of “cat” from ImageNet for homography image alignment\npython3 train.py \\\n--model=planar --yaml=planar \\\n--group=exp_planar --name=naive_cat \\\n--warp.type=homography --warp.dof=8 \\\n--data.image_fname=data/cat.jpg \\\n--data.image_size=[360,480] \\\n--data.patch_crop=[180,180] \\\n--gpu=3 --barf_c2f=null\n```\n--------------------------------------\n\n## Running reference NeRF\nIf you want to train and evaluate the reference NeRF models (assuming known camera poses):\n```bash\n# \u003cGROUP\u003e and \u003cNAME\u003e can be set to your likes, while \u003cSCENE\u003e is specific to datasets\n\n# NeRF (3D): Synthetic Objects\n# Blender (\u003cSCENE\u003e={chair,drums,ficus,hotdog,lego,materials,mic,ship})\npython3 train.py \\\n--model=nerf --yaml=nerf_blender \\\n--group=exp_synthetic --name=ref_lego \\\n--data.scene=lego --gpu=0 \\\n--data.root=/home/cy/PNW/datasets/nerf_synthetic/ \n\npython3 evaluate.py \\\n--model=nerf --yaml=nerf_blender \\\n--group=exp_synthetic --name=ref_lego \\\n--data.scene=lego --gpu=0 \\\n--data.root=/home/cy/PNW/datasets/nerf_synthetic/ \\\n--data.val_sub= --resume\n\n# NeRF (3D): Real-World Scenes\n# LLFF (\u003cSCENE\u003e={fern,flower,fortress,horns,leaves,orchids,room,trex})\npython3 train.py \\\n--model=nerf --yaml=nerf_llff \\\n--group=exp_LLFF --name=ref_fern \\\n--data.scene=fern --gpu=0 \\\n--data.root=/home/cy/PNW/datasets/nerf_llff_data/\n\n\npython3 evaluate.py \\\n--model=nerf --yaml=nerf_llff \\\n--group=exp_LLFF --name=ref_fern \\\n--data.scene=fern --gpu=0 \\\n--data.root=/home/cy/PNW/datasets/nerf_llff_data/ \\\n--resume\n```\n--------------------------------------\n\n# :laughing: Visualization\n\n## Results and Videos\nAll the results will be stored in the directory `output/\u003cGROUP\u003e/\u003cNAME\u003e`.\nYou may want to organize your experiments by grouping different runs in the same group. Many videos will be created to visualize the pose optimization process and novel view synthesis.\n\n## TensorBoard\nThe TensorBoard events include the following:\n- **SCALARS**: the rendering losses and PSNR over the course of optimization. For L2G_NeRF/BARF/Naive, the rotational/translational errors with respect to the given poses are also computed.\n- **IMAGES**: visualization of the RGB images and the RGB/depth rendering.\n\n## Visdom\nThe visualization of 3D camera poses is provided in Visdom:\nRun `visdom -port 8600` to start the Visdom server.  The Visdom host server is default to `localhost`; this can be overridden with `--visdom.server` (see `options/base.yaml` for details). If you want to disable Visdom visualization, add `--visdom!`.\n\n## Mesh\nThe `extract_mesh.py` script provides a simple way to extract the underlying 3D geometry using marching cubes (supporte for the Blender dataset). Run as follows:\n```bash\npython3 extract_mesh.py \\\n--model=l2g_nerf --yaml=l2g_nerf_blender \\\n--group=exp_synthetic --name=l2g_lego \\\n--data.scene=lego --gpu=3 \\\n--data.root=/home/cy/PNW/datasets/nerf_synthetic/ \\\n--data.val_sub= --resume\n```\n--------------------------------------\n\n# :mag_right: Codebase structure\n\nThe main engine and network architecture in `model/l2g_nerf.py` inherit those from `model/nerf.py`.\n\nSome tips on using and understanding the codebase:\n- The computation graph for forward/backprop is stored in `var` throughout the codebase.\n- The losses are stored in `loss`. To add a new loss function, just implement it in `compute_loss()` and add its weight to `opt.loss_weight.\u003cname\u003e`. It will automatically be added to the overall loss and logged to Tensorboard.\n- If you are using a multi-GPU machine, you can set `--gpu=\u003cgpu_number\u003e` to specify which GPU to use. Multi-GPU training/evaluation is currently not supported.\n- To resume from a previous checkpoint, add `--resume=\u003cITER_NUMBER\u003e`, or just `--resume` to resume from the latest checkpoint.\n- To eliminate the global alignment objective, set `--loss_weight.global_alignment=null`, the ablation is equivalent to a local registration method.\n\n--------------------------------------\n# Citation\n\nIf you find this project useful for your research, please use the following BibTeX entry.\n\n```bibtex\n@inproceedings{chen2023local,\n  title={Local-to-global registration for bundle-adjusting neural radiance fields},\n  author={Chen, Yue and Chen, Xingyu and Wang, Xuan and Zhang, Qi and Guo, Yu and Shan, Ying and Wang, Fei},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={8264--8273},\n  year={2023}\n}\n```\n--------------------------------------\n\n# Acknowledge\nOur code is based on the awesome pytorch implementation of Bundle-Adjusting Neural Radiance Fields ([BARF](https://github.com/chenhsuanlin/bundle-adjusting-NeRF)). We appreciate all the contributors.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/rover-xingyu.github.io%2FL2G-NeRF%2F","html_url":"https://awesome.ecosyste.ms/projects/rover-xingyu.github.io%2FL2G-NeRF%2F","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/rover-xingyu.github.io%2FL2G-NeRF%2F/lists"}