{"id":13474455,"url":"https://nianticlabs.github.io/nerf-object-removal/","last_synced_at":"2025-03-26T21:31:43.327Z","repository":{"id":176358202,"uuid":"633783118","full_name":"nianticlabs/nerf-object-removal","owner":"nianticlabs","description":"[CVPR 2023] Removing Objects From Neural Radiance Fields","archived":false,"fork":false,"pushed_at":"2023-08-21T06:10:42.000Z","size":8682,"stargazers_count":110,"open_issues_count":11,"forks_count":9,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-03-11T02:24:02.229Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nianticlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-04-28T09:10:24.000Z","updated_at":"2025-02-06T21:45:36.000Z","dependencies_parsed_at":"2024-01-13T18:35:57.700Z","dependency_job_id":null,"html_url":"https://github.com/nianticlabs/nerf-object-removal","commit_stats":null,"previous_names":["nianticlabs/nerf-object-removal"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fnerf-object-removal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fnerf-object-removal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fnerf-object-removal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fnerf-object-removal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nianticlabs","download_url":"https://codeload.github.com/nianticlabs/nerf-object-removal/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245738700,"owners_count":20664330,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T16:01:12.474Z","updated_at":"2025-03-26T21:31:40.961Z","avatar_url":"https://github.com/nianticlabs.png","language":"Python","funding_links":[],"categories":["Uncategorized"],"sub_categories":["Uncategorized"],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003e Removing Objects from Neural Radiance Fields \u003c/h1\u003e\n\u003ca href=\"https://www.silvanweder.com\"\u003eSilvan Weder\u003c/a\u003e\u003csup\u003e1, 2\u003c/sup\u003e,\n\u003ca href=\"https://guiggh.github.io\"\u003eGuillermo Garcia-Hernando\u003c/a\u003e\u003csup\u003e2\u003c/sup\u003e,\n\u003ca href=\"http://aron.monszp.art\"\u003eÁron Monszpart\u003c/a\u003e\u003csup\u003e2\u003c/sup\u003e\n\u003ca href=\"https://people.inf.ethz.ch/marc.pollefeys\"\u003eMarc Pollefeys\u003c/a\u003e\u003csup\u003e2\u003c/sup\u003e,\n\u003ca href=\"http://www0.cs.ucl.ac.uk/staff/g.brostow\"\u003eGabriel Brostow\u003c/a\u003e\u003csup\u003e1, 3\u003c/sup\u003e, \u003cbr/\u003e\n\u003ca href=\"http://www.michaelfirman.co.uk\"\u003eMichael Firman\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e,\n\u003ca href=\"https://scholar.google.com/citations?user=7wWsNNcAAAAJ\"\u003eSara Vicente\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e \n  \n\u003csup\u003e1\u003c/sup\u003e\u003ca href=\"https://research.nianticlabs.com/\"\u003eNiantic\u003c/a\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n\u003csup\u003e2\u003c/sup\u003e\u003ca href=\"https://cvg.ethz.ch/index.php\"\u003eETH Zürich\u003c/a\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n\u003csup\u003e3\u003c/sup\u003e\u003ca href=\"http://www0.cs.ucl.ac.uk/staff/g.brostow/#Research\"\u003eUniversity College London\u003c/a\u003e\n  \n![teaser](./assets/teaser.png)\n\n[[Project Webpage](https://nianticlabs.github.io/nerf-object-removal/)]\n[[Paper](https://nianticlabs.github.io/nerf-object-removal/resources/RemovingObjectsFromNeRFs.pdf)]\n\n\u003c/div\u003e\n\u003cbr\u003e\u003cbr\u003e\n\nThis code is for non-commercial use; please see the [license file](LICENSE) for terms.\n\n## Installation\n\nAn Nvidia GPU is required to run this code. It's been tested on Ubuntu 20.04.4 LTS.\n\nFirst, you need to clone this repo:\n\n```shell\ngit clone --recursive git@github.com:nianticlabs/nerf-object-removal.git\n```\n\nIn order to install this repository, run the following commands. You might need to adjust the CUDA and cudNN versions in [install.sh](install.sh).\nThis installation is only tested on CUDA 11.1. With other versions it might not run due to JAX issues.\n\n```shell\nconda create -n object-removal python=3.8\nconda activate object-removal\n# make sure libcudart.so can be found:\nexport LD_LIBRARY_PATH=\"${CONDA_PREFIX}/lib:${CONDA_PREFIX}/lib/python3.8/site-packages/nvidia/cuda_runtime/lib/:${LD_LIBRARY_PATH}\"\nbash ./install.sh\n```\n\n## Download Dataset\n\nPlease download the data from here using the following command:\n\n```shell\nmkdir \"$(pwd)/data\"\nwget -P \"$(pwd)/data\" \"https://storage.googleapis.com/niantic-lon-static/research/nerf-object-removal/nerf-object-removal.zip\"\nunzip \"$(pwd)/data/nerf-object-removal.zip\" -d \"$(pwd)/data\"\n```\n\n## Download Big Lama Model from HF\n\nPlease download the pretrained Lama model from Hugging Face.\n\n```shell\ncd external/lama\nwget https://huggingface.co/smartywu/big-lama/resolve/main/big-lama.zip\nunzip big-lama.zip\ncd ../..\n```\n\n## Run End-to-end Optimization\n\nIn order to test our pipeline using the default configuration on a scene of the provided dataset, run the following command:\n\n```shell\nexport SCENE_NUMBER=\"001\"; \\\nROOT_DIR=\"$(pwd)/data/object-removal-custom-clean\" \\\nOUTPUT_DIR=\"$(pwd)/experiments/real\" \\\nbash ./run_real.sh model/configs/custom/default.gin \"${SCENE_NUMBER}\"\n```\n\nor on the synthetic dataset with\n\n```shell\nexport SCENE_NUMBER=\"001\"; \\\nROOT_DIR=\"$(pwd)/data/object-removal-custom-clean\" \\\nOUTPUT_DIR=\"$(pwd)/experiments/synthetic\" \\\nbash ./run_synthetic.sh model/configs/custom_synthetic/default.gin \"${SCENE_NUMBER}\"\n```\n\nYou have to adjust the ```$EXPERIMENT_PATH``` and the ```$DATA_PATH``` accordingly in the files above.\n\n\n## Test a trained model\n\nYou can test a trained model using the following command for a real scene\n\n```shell\nexport SCENE_NUMBER=\"001\"; \\\nROOT_DIR=\"$(pwd)/data/object-removal-custom-clean\" \\\nOUTPUT_DIR=\"$(pwd)/experiments/real\" \\\nbash ./test_real.sh model/configs/custom/default.gin \"${SCENE_NUMBER}\"\n```\n\nor the following command for a synthetic scene\n\n```shell\nexport SCENE_NUMBER=\"001\"; \\\nROOT_DIR=\"$(pwd)/data/object-removal-custom-clean\" \\\nOUTPUT_DIR=\"$(pwd)/experiments/synthetic\" \\\nbash ./test_synthetic.sh model/configs/custom_synthetic/default.gin \"${SCENE_NUMBER}\"\n```\n\n\n## Visualizing the results\n\nOnce the models have been optimized, you can visualize the results using [this notebook](notebooks/vis_experiment.ipynb).\n\nMake sure that you correctly set the ```$EXPERIMENT_DIR``` and the ```$GROUNDTRUTH_DIR``` and follow the notebook carefully to set all required options for your experiment. \n\n## Evaluating the results\n\nYou can evaluate the results using the [evaluation script](evaluation/eval.py).\n\n```shell\npython eval.py --experiment \"${EXPERIMENT_NAME}\" --experiment_root_dir \"${EXPERIMENT_DIR}\" --benchmark (synthetic/real)\n```\n\n## Visualizing the evaluation results\n\nYou van visualize the evaluation results using [this notebook](notebooks/vis_results.ipynb).\n\n## Running through Docker\n\nSee [docker/README.md](docker/README.md).\n\n## Citation\n\nIf you find this work helpful, please consider citing\n\n```\n@InProceedings{Weder_2023_CVPR,\n    author    = {Weder, Silvan and Garcia-Hernando, Guillermo and Monszpart, \\'Aron and Pollefeys, Marc and Brostow, Gabriel J. and Firman, Michael and Vicente, Sara},\n    title     = {Removing Objects From Neural Radiance Fields},\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month     = {June},\n    year      = {2023},\n    pages     = {16528-16538}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/nianticlabs.github.io%2Fnerf-object-removal%2F","html_url":"https://awesome.ecosyste.ms/projects/nianticlabs.github.io%2Fnerf-object-removal%2F","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/nianticlabs.github.io%2Fnerf-object-removal%2F/lists"}