{"id":19517466,"url":"https://github.com/cvg/nerf-on-the-go","last_synced_at":"2025-04-23T17:27:33.157Z","repository":{"id":241810934,"uuid":"807393372","full_name":"cvg/nerf-on-the-go","owner":"cvg","description":"[CVPR'24] NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild","archived":false,"fork":false,"pushed_at":"2024-08-23T21:56:30.000Z","size":16470,"stargazers_count":174,"open_issues_count":3,"forks_count":7,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-30T01:51:47.652Z","etag":null,"topics":["dinov2","dynamic-scenes","in-the-wild","nerf","neural-fields","neural-implicit-representations","neural-radiance-fields","uncertainty"],"latest_commit_sha":null,"homepage":"https://rwn17.github.io/nerf-on-the-go/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cvg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-29T02:45:57.000Z","updated_at":"2025-03-23T06:46:00.000Z","dependencies_parsed_at":"2024-08-23T22:49:45.454Z","dependency_job_id":"6c901dbb-8d4c-4a85-a800-0404f1a5df5d","html_url":"https://github.com/cvg/nerf-on-the-go","commit_stats":null,"previous_names":["cvg/nerf-on-the-go"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvg%2Fnerf-on-the-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvg%2Fnerf-on-the-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvg%2Fnerf-on-the-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvg%2Fnerf-on-the-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cvg","download_url":"https://codeload.github.com/cvg/nerf-on-the-go/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250479615,"owners_count":21437395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dinov2","dynamic-scenes","in-the-wild","nerf","neural-fields","neural-implicit-representations","neural-radiance-fields","uncertainty"],"created_at":"2024-11-11T00:03:45.548Z","updated_at":"2025-04-23T17:27:33.137Z","avatar_url":"https://github.com/cvg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# NeRF \u003cem\u003eOn-the-go\u003c/em\u003e: Exploiting Uncertainty for Distractor-free NeRFs in the Wild\n\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/rwn17\"\u003e\u003cstrong\u003eWeining Ren*\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://zzh2000.github.io\"\u003e\u003cstrong\u003eZihan Zhu*\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://inf.ethz.ch/people/people-atoz/person-detail.MjY0ODc2.TGlzdC8zMDQsLTIxNDE4MTU0NjA=.html\"\u003e\u003cstrong\u003eBoyang Sun\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://inf.ethz.ch/people/people-atoz/person-detail.Mjc4NTY0.TGlzdC8zMDQsLTIxNDE4MTU0NjA=.html\"\u003e\u003cstrong\u003eJulia Chen\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://people.inf.ethz.ch/pomarc/\"\u003e\u003cstrong\u003eMarc Pollefeys\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://pengsongyou.github.io\"\u003e\u003cstrong\u003eSongyou Peng\u003c/strong\u003e\u003c/a\u003e\n  \u003c/p\u003e\n  \u003cp align=\"center\"\u003e\u003cstrong\u003e(* Equal Contribution)\u003c/strong\u003e\u003c/p\u003e\n  \u003ch2 align=\"center\"\u003eCVPR 2024\u003c/h2\u003e\n  \u003ch3 align=\"center\"\u003e\u003ca href=\"https://arxiv.org/abs/2405.18715\"\u003ePaper\u003c/a\u003e | \u003ca href=\"https://youtu.be/mUQ_LOyonB0?si=uacxom-6ur7_oGRw\"\u003eVideo\u003c/a\u003e | \u003ca href=\"https://rwn17.github.io/nerf-on-the-go/\"\u003eProject Page\u003c/a\u003e\u003c/h3\u003e\n  \u003cp align=\"center\"\u003e\n  \u003ca href=\"\"\u003e\n    \u003cimg src=\"./media/teaser.gif\" alt=\"Logo\" width=\"80%\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n  \u003cdiv align=\"center\"\u003e\u003c/div\u003e\n\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails open=\"open\" style='padding: 10px; border-radius:5px 30px 30px 5px; border-style: solid; border-width: 1px;'\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#Description\"\u003eDescription\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#Setup\"\u003eSetup\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#Dataset Preparation\"\u003eDataset Preparation\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#Running\"\u003eRunning\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#Citation\"\u003eCitation\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#Contact\"\u003eContact\u003c/a\u003e\n    \u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n\n## Description\n\nThis repository hosts the official Jax implementation of the paper \"NeRF \u003cem\u003eon-the-go\u003c/em\u003e: Exploiting Uncertainty for Distractor-free NeRFs in the Wild\" (CVPR 2024). For more details, please visit our [project webpage](https://rwn17.github.io/nerf-on-the-go/).\n\nThis Repo is built upon [Multinerf](https://github.com/google-research/multinerf) codebase.\n\n## Setup\n\n```\n# Clone the repo.\ngit clone https://github.com/cvg/nerf-on-the-go\ncd nerf-on-the-go\n\n# Make a conda environment.\nconda create --name on-the-go python=3.9\nconda activate on-the-go\n\n# Prepare pip.\nconda install pip\npip install --upgrade pip\n\n\n# Install requirements.\npip install -r requirements.txt\n\n# Manually install rmbrualla's `pycolmap` (don't use pip's! It's different).\ngit clone https://github.com/rmbrualla/pycolmap.git ./internal/pycolmap\n\n# Confirm that all the unit tests pass.\n./scripts/run_all_unit_tests.sh\n```\nYou'll also need to update your [JAX](https://jax.readthedocs.io/en/latest/installation.html) installation to support GPUs or TPUs.\n\n```\npip install  jax==0.4.26 -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html\npip install  jaxlib==0.4.26+cuda12.cudnn89 -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html\n```\n\n### Instructions for ETH Euler\n\u003cdetails\u003e\n  \u003csummary\u003eClick to expand\u003c/summary\u003e\n\non ETH Euler, to support for GPU jax, you need to apply for a debug mode gpu and then upgrade the gcc and cuda\n```\nsrun -n 4 --mem-per-cpu=12000 --gpus=rtx_3090:1 --gres=gpumem:20g --time=4:00:00 --pty bash\nconda activate on-the-go\nmodule load eth_proxy gcc/8.2.0 cuda/12.1.1 cudnn/8.9.2.26\n```\n\nAfter loading the modules, verify their activation by executing ```module list```. Occasionally, modules may not load correctly, requiring you to load each one individually. Following this, proceed with the Jax installation:\n\n```\n# Installs the wheel compatible with CUDA 12 and cuDNN 8.9 or newer.\npip install  jax==0.4.26 -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html\npip install  jaxlib==0.4.26+cuda12.cudnn89 -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html\n```\n\nAfter successful installation, please rerun ```./scripts/run_all_unit_tests.sh```.\n\nThe installation process outlined above has been verified on the Euler system using an RTX 3090. You may get a warning \n```\nThe NVIDIA driver's CUDA version is 12.0 which is older than the ptxas CUDA version (12.4.131). Because the driver is older than the ptxas version, XLA is disabling parallel compilation, which may slow down compilation. You should update your NVIDIA driver or use the NVIDIA-provided CUDA forward compatibility packages.\n``` \n\nBut it's fine. The Euler supports up to CUDA 12.1, while JAX now requires a minimum of CUDA 12.3. As discussed in the JAX [Issue #18032](https://github.com/google/jax/issues/18032), this discrepancy primarily impacts compilation speed rather than overall functionality.\n\u003c/details\u003e\n\n\n## Dataset Preparation\n\n### Downloading the Dataset\nTo download the \"On-the-go\" dataset, execute the following command:\n```bash\nbash ./scripts/download_on-the-go.sh\n```\nThis script not only downloads the dataset but also downsamples the images as required.\n\n### Feature Extraction with DINOv2\nFor extracting features using the DINOv2, use the command below:\n```bash\nbash ./scripts/feature_extract.sh\n```\n\n\nAfter feature extraction, the dataset should be organized as \n```\non-the-go\n├── arcdetriomphe\n│   ├── images\n│   ├── images_{DOWNSAMPLE_RATE}\n│   ├── features_{DOWNSAMPLE_RATE}\n│   ├── split.json\n│   ├── transforms.json\n├── ....\n│\n└── tree\n    ├── images_{DOWNSAMPLE_RATE}\n    ├── ....\n    └── transforms.json\n```\n\n### Dataset Structure and Configuration Files\n- **split.json**: This file outlines the train and evaluation splits, following the naming conventions used in the RobustNeRF dataset, categorized as 'clutter' and 'clean'.\n- **transforms.json**: Contains pose and intrinsic information, formatted according to the Blender dataset format, derived from COLMAP files. Refer to the [Instant-NGP script](https://github.com/NVlabs/instant-ngp/blob/de507662d4b3398163e426fd426d48ff8f2895f6/scripts/colmap2nerf.py) for more details.\n\n### Future Updates\nWe plan to expand support to include custom datasets in future updates.\n\n\n## Running\n\nExample scripts for training, evaluating, and rendering can be found in\n`scripts/`. You'll need to change the paths to point to wherever the datasets\nare located. [Gin](https://github.com/google/gin-config) configuration files\nfor our model and some ablations can be found in `configs/`.\n\n1. Training on-the-go:\n```\nbash scripts/train_on-the-go.sh \n```\n\n2. Evaluating on-the-go:\n```\nbash scripts/eval_on-the-go.sh\n```\n\n3. Rendering on-the-go:\n```\nbash scirpts/render_on-the-go.sh\n```\n\nTensorboard is supported for logging.\n\n### Note\nSince we use a different recording device for ***arc de triomphe*** and ***patio*** scene, the image downsample rate(4 instead of 8) and feature downsample rate(2 instead of 4) is different. Please use a separate script to train them by \n\n```\nbash scripts/train_on-the-go_HD.sh\n```\n\n### OOM errors\n\nAbout **80G gpu memory** is needed to run current version.You may need to reduce the batch size (`Config.batch_size`) to avoid out of memory\nerrors. If you do this, but want to preserve quality, be sure to increase the number\nof training iterations and decrease the learning rate by whatever scale factor you\ndecrease batch size by. \n\n\n## Todo\n\n- [ ] Custom dataset tutorial\n- [ ] Support LPIPS calculation\n \n## Citation\n\nIf you use NeRF on-the-go, please cite \n\n```\n@InProceedings{Ren2024NeRF,\n    title={NeRF on-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild},\n    author={Ren, Weining and Zhu, Zihan and Sun, Boyang and Chen, Jiaqi and Pollefeys, Marc and Peng, Songyou},\n    booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    year={2024}\n}\n```\n\nAlso, this code is built upon multinerf, feel free to cite this entire codebase as:\n\n```\n@misc{multinerf2022,\n      title={{MultiNeRF}: {A} {Code} {Release} for {Mip-NeRF} 360, {Ref-NeRF}, and {RawNeRF}},\n      author={Ben Mildenhall and Dor Verbin and Pratul P. Srinivasan and Peter Hedman and Ricardo Martin-Brualla and Jonathan T. Barron},\n      year={2022},\n      url={https://github.com/google-research/multinerf},\n}\n```\n\n## Contact\nIf there is any problem, please contact Weining by ren.weining@connect.hku.hk\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcvg%2Fnerf-on-the-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcvg%2Fnerf-on-the-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcvg%2Fnerf-on-the-go/lists"}