{"id":18614464,"url":"https://github.com/rover-xingyu/Ha-NeRF","last_synced_at":"2025-04-11T00:30:50.654Z","repository":{"id":38087281,"uuid":"434066744","full_name":"rover-xingyu/Ha-NeRF","owner":"rover-xingyu","description":"[CVPR 2022] Ha-NeRF😆: Hallucinated Neural Radiance Fields in the Wild","archived":false,"fork":false,"pushed_at":"2024-03-06T12:34:53.000Z","size":34509,"stargazers_count":154,"open_issues_count":17,"forks_count":12,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-04-06T05:31:53.931Z","etag":null,"topics":["nerf"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rover-xingyu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-12-02T03:18:40.000Z","updated_at":"2025-03-29T08:59:00.000Z","dependencies_parsed_at":"2024-01-18T18:09:11.967Z","dependency_job_id":"9f50f704-3807-4740-9643-655bdefb96f4","html_url":"https://github.com/rover-xingyu/Ha-NeRF","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rover-xingyu%2FHa-NeRF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rover-xingyu%2FHa-NeRF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rover-xingyu%2FHa-NeRF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rover-xingyu%2FHa-NeRF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rover-xingyu","download_url":"https://codeload.github.com/rover-xingyu/Ha-NeRF/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322220,"owners_count":21084333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nerf"],"created_at":"2024-11-07T03:25:57.206Z","updated_at":"2025-04-11T00:30:45.635Z","avatar_url":"https://github.com/rover-xingyu.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF Related Tasks"],"readme":"# Ha-NeRF:laughing:: Hallucinated Neural Radiance Fields in the Wild \n**[Project Page](https://rover-xingyu.github.io/Ha-NeRF/) |\n[Paper](https://openaccess.thecvf.com/content/CVPR2022/papers/Chen_Hallucinated_Neural_Radiance_Fields_in_the_Wild_CVPR_2022_paper.pdf) |\n[Latest arXiv](https://arxiv.org/pdf/2111.15246.pdf) |\n[Supplementary](https://rover-xingyu.github.io/Ha-NeRF/files/Ha_NeRF_CVPR_2022_supp.pdf)**\n\n[Xingyu Chen¹](https://scholar.google.com/citations?user=gDHPrWEAAAAJ\u0026hl=en), \n[Qi Zhang²](https://scholar.google.com/citations?user=2vFjhHMAAAAJ\u0026hl=en), \n[Xiaoyu Li²](https://scholar.google.com/citations?user=Dt0PcAYAAAAJ\u0026hl=en), \n[Yue Chen¹](https://scholar.google.com/citations?user=M2hq1_UAAAAJ\u0026hl=en), \n[Ying Feng²](https://scholar.google.com/citations?user=PhkrqioAAAAJ\u0026hl=en),\n[Xuan Wang²](https://scholar.google.com/citations?user=h-3xd3EAAAAJ\u0026hl=en),\n[Jue Wang²](https://scholar.google.com/citations?user=Bt4uDWMAAAAJ\u0026hl=en). \n\n[¹Xi'an Jiaotong University](http://en.xjtu.edu.cn/),\n[²Tencent AI Lab](https://ai.tencent.com/ailab/en/index/).\n\n\nThis repository is an official implementation of [Ha-NeRF](https://rover-xingyu.github.io/Ha-NeRF/) (Hallucinated Neural Radiance Fields in the Wild) using pytorch ([pytorch-lightning](https://github.com/PyTorchLightning/pytorch-lightning)). \n\n# :computer: Installation\n\n## Hardware\n\n* OS: Ubuntu 18.04\n* NVIDIA GPU with **CUDA\u003e=10.2**\n* We optimize all implementations for 600 000 iterations with a batch size of 1024 on 4 A100s, where Ha-NeRF and NeRF-W take 20 and 18 hours, respectively. Both methods require around 20 GB of memory for training and 5 GB for inference.\n\n## Software\n\n* Clone this repo by `git clone https://github.com/rover-xingyu/Ha-NeRF`\n* Python\u003e=3.6 (installation via [anaconda](https://www.anaconda.com/distribution/) is recommended, use `conda create -n HaNeRF python=3.6` to create a conda environment and activate it by `conda activate HaNeRF`)\n* Python libraries\n    * Install core requirements by `pip install -r requirements.txt`\n    \n# :key: Training\n\n## Data download\n\nDownload the scenes you want from [here](https://www.cs.ubc.ca/~kmyi/imw2020/data.html) \n\nDownload the train/test split from [here](https://nerf-w.github.io/) and put under each scene's folder (the **same level** as the \"dense\" folder)\n\n(Optional but **highly** recommended) Run `python3.6 prepare_phototourism.py --root_dir $ROOT_DIR --img_downscale {an integer, e.g. 2 means half the image sizes}` to prepare the training data and save to disk first, if you want to run multiple experiments or run on multiple gpus. This will **largely** reduce the data preparation step before training.\n\nRun (example)\n\n```\npython3.6 prepare_phototourism.py --root_dir /path/to/the/datasets/brandenburg_gate/ --img_downscale 2\n```\n\n## Training model\nRun (example)\n```\npython3.6 train_mask_grid_sample.py \\\n  --root_dir /path/to/the/datasets/brandenburg_gate/ --dataset_name phototourism \\\n  --save_dir save \\\n  --img_downscale 2 --use_cache \\\n  --N_importance 64 --N_samples 64 \\\n  --num_epochs 20 --batch_size 1024 \\\n  --optimizer adam --lr 5e-4 --lr_scheduler cosine \\\n  --exp_name exp_HaNeRF_Brandenburg_Gate \\\n  --N_emb_xyz 15 --N_vocab 1500 \\\n  --use_mask --maskrs_max 5e-2 --maskrs_min 6e-3 --maskrs_k 1e-3 --maskrd 0 \\\n  --encode_a --N_a 48 --weightKL 1e-5 --encode_random --weightRecA 1e-3 --weightMS 1e-6 \\\n  --num_gpus 4\n```\n\nAdd `--encode_a` for using appearance hallucination module, `--use_mask` for using  anti-occlusion module. `--N_vocab` should be set to an integer larger than the number of images (dependent on different scenes). For example, \"brandenburg_gate\" has in total 1363 images (under `dense/images/`), so any number larger than 1363 works (no need to set to exactly the same number). **Attention!** If you forget to set this number, or it is set smaller than the number of images, the program will yield `RuntimeError: CUDA error: device-side assert triggered` (which comes from `torch.nn.Embedding`).\n\nSee [opt.py](opt.py) for all configurations.\n\nThe checkpoints and logs will be saved to `{save_dir}/ckpts/{scene_name} ` and `{save_dir}/logs/{scene_name}`, respectively.\n\nYou can monitor the training process by `tensorboard --logdir save/logs/exp_HaNeRF_Brandenburg_Gate --port=8600` and go to `localhost:8600` in your browser.\n\n# :mag_right: Evaluation\n\nUse [eval.py](eval.py) to inference on all test data. It will create folder `{save_dir}/results/{dataset_name}/{scene_name}` and save the rendered\nimages.\n\nRun (example)\n```\npython3.6 eval.py \\\n  --root_dir /path/to/the/datasets/brandenburg_gate/ \\\n  --save_dir save \\\n  --dataset_name phototourism --scene_name HaNeRF_Trevi_Fountain \\\n  --split test_test --img_downscale 2 \\\n  --N_samples 256 --N_importance 256 --N_emb_xyz 15 \\\n  --N_vocab 1500 --encode_a \\\n  --ckpt_path save/ckpts/HaNeRF_Brandenburg_Gate/epoch\\=19.ckpt \\\n  --chunk 16384 --img_wh 320 240\n```\n\nThen you can use [eval_metric.py](eval_metric.py) to get the quantitative report of different metrics based on the rendered images from [eval.py](eval.py). It will create a file `result.txt` in the folder `{save_dir}/results/{dataset_name}/{scene_name}` and save the metrics.\n\nRun (example)\n```\npython3.6 eval_metric.py \\\n  --root_dir /path/to/the/datasets/brandenburg_gate/ \\\n  --save_dir save \\\n  --dataset_name phototourism --scene_name HaNeRF_Trevi_Fountain \\\n  --split test_test --img_downscale 2 \\\n  --img_wh 320 240\n```\n\n# :laughing: Hallucination\n\nUse [hallucinate.py](hallucinate.py) to play with Ha-NeRF by hallucinating appearance from different scenes `{example_image}` in different views! It will create folder `{save_dir}/hallucination/{scene_name}` and render the hallucinations, finally create a gif out of them.\n\nRun (example)\n```\npython3.6 hallucinate.py \\\n    --save_dir save \\\n    --ckpt_path save/ckpts/HaNeRF_Trevi_Fountain/epoch\\=19.ckpt \\\n    --chunk 16384 \\\n    --example_image artworks \\\n    --scene_name artworks_2_fountain\n```\n\n# Cite\nIf you find our work useful, please consider citing:\n```bibtex\n@inproceedings{chen2022hallucinated,\n  title={Hallucinated neural radiance fields in the wild},\n  author={Chen, Xingyu and Zhang, Qi and Li, Xiaoyu and Chen, Yue and Feng, Ying and Wang, Xuan and Wang, Jue},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={12943--12952},\n  year={2022}\n}\n```\n\n# Acknowledge\nOur code is based on the awesome pytorch implementation of NeRF in the Wild ([NeRF-W](https://github.com/kwea123/nerf_pl/tree/nerfw/)). We appreciate all the contributors.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frover-xingyu%2FHa-NeRF","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frover-xingyu%2FHa-NeRF","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frover-xingyu%2FHa-NeRF/lists"}