{"id":18614309,"url":"https://github.com/VITA-Group/Aug-NeRF","last_synced_at":"2025-04-11T00:30:32.087Z","repository":{"id":37633802,"uuid":"485588321","full_name":"VITA-Group/Aug-NeRF","owner":"VITA-Group","description":"[CVPR 2022] \"Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations\" by Tianlong Chen*, Peihao Wang*, Zhiwen Fan, Zhangyang Wang ","archived":false,"fork":false,"pushed_at":"2022-06-13T02:25:36.000Z","size":210,"stargazers_count":124,"open_issues_count":4,"forks_count":8,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-03-29T09:41:56.629Z","etag":null,"topics":["adversarial-robustness","neural-radiance-fields","neural-rendering"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VITA-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-04-26T01:12:09.000Z","updated_at":"2024-08-06T08:54:59.000Z","dependencies_parsed_at":"2022-07-09T15:47:08.991Z","dependency_job_id":null,"html_url":"https://github.com/VITA-Group/Aug-NeRF","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FAug-NeRF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FAug-NeRF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FAug-NeRF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FAug-NeRF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VITA-Group","download_url":"https://codeload.github.com/VITA-Group/Aug-NeRF/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322208,"owners_count":21084333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-robustness","neural-radiance-fields","neural-rendering"],"created_at":"2024-11-07T03:25:54.402Z","updated_at":"2025-04-11T00:30:31.639Z","avatar_url":"https://github.com/VITA-Group.png","language":"Python","funding_links":[],"categories":["NeRF任务分类","Papers"],"sub_categories":["数据增强","NeRF"],"readme":"# Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\nOfficial implementation of CVPR 2022 paper [\"Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations\"](https://openaccess.thecvf.com/content/CVPR2022/papers/Chen_Aug-NeRF_Training_Stronger_Neural_Radiance_Fields_With_Triple-Level_Physically-Grounded_Augmentations_CVPR_2022_paper.pdf).\n\nTianlong Chen*, Peihao Wang*, Zhiwen Fan, Zhangyang (Atlas) Wang\n\n## Introduction\n\nNeural Radiance Field (NeRF) regresses a neural parameterized scene by differentially rendering multi-view images with ground-truth supervision. However, when interpolating novel views, NeRF often yields inconsistent and visually non-smooth geometric results, which we consider as a generalization gap between seen and unseen views. Recent advances in convolutional neural networks have demonstrated the promise of advanced robust data augmentations, either random or learned, in enhancing both in-distribution and out-of-distribution generalization. Inspired by that, we propose Augmented NeRF (Aug-NeRF), which for the first time brings the power of robust data augmentations into regularizing the NeRF training. Particularly, our proposal learns to seamlessly blend worst-case perturbations into three distinct levels of the NeRF pipeline with physical grounds, including (1) the input coordinates, to simulate imprecise camera parameters at image capture; (2) intermediate features, to smoothen the intrinsic feature manifold; and (3) pre-rendering output, to account for the potential degradation factors in the multi-view image supervision. Extensive results demonstrate that Aug-NeRF effectively boosts NeRF performance in both novel view synthesis (up to 1.5 dB PSNR gain) and underlying geometry reconstruction. Furthermore, thanks to the implicit smooth prior injected by the triple-level augmentations, Aug-NeRF can even recover scenes from heavily corrupted images, a highly challenging setting untackled before.\n\n![](figures/teaser.jpg)\n\n## Getting Started\n\n### Dependency\n\nWe recommend users to use `conda` to install the running environment. The following dependencies are required:\n```\npytorch=1.7.0\ntorchvision=0.8.0\ncudatoolkit=11.0\ntensorboard=2.7.0\nopencv\nimageio\nimageio-ffmpeg\nconfigargparse\nscipy\nmatplotlib\ntqdm\nmrc\nlpips\n```\n\n### Data Preparation\n\nTo run our code on NeRF dataset, users need first download data from official [cloud drive](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1). Then extract package files according to the following directory structure:\n```\n├── configs\n│   ├── ...\n│\n├── datasets\n│   ├── nerf_llff_data\n│   │   └── fern\n│   │   └── flower  # downloaded llff dataset\n│   │   └── horns   # downloaded llff dataset\n|   |   └── ...\n|   ├── nerf_synthetic\n|   |   └── lego\n|   |   └── ship    # downloaded synthetic dataset\n|   |   └── ...\n```\nThe last step is to generate and process data via our provided script:\n```\npython gen_dataset.py --config \u003cconfig_file\u003e\n```\nwhere `\u003cconfig_file\u003e` is the path to the configuration file of your experiment instance. Examples and pre-defined configuration files are provided in `configs` folder.\n\nOur synthetic noisy dataset will come soon ...\n\n## Usage\n\n### Training\n\nAfter generating datasets, users can train a vanilla NeRF by the following command:\n```\npython run_nerf.py --gpuid \u003cgpu_id\u003e --expname \u003coutput_folder\u003e --config \u003cdefault_config_file\u003e\n```\n\nTo apply adversarial augmentation, users can specify the option `--adv`:\n```\npython run_nerf.py --gpuid \u003cgpu_id\u003e --expname \u003coutput_folder\u003e --config \u003cdefault_config_file\u003e\n--adv \u003cadv types\u003e --pgd_alpha \u003cpgd alpha\u003e --pgd_iters \u003cpgd iters\u003e --pgd_eps \u003cpgd eps\u003e\n```\nwhere available options include 1) Coordinate perturbation: `pts_c`, `pts_f`, `zval_c`, `zval_f`, 2) Intermediate feature perturbation: `feat_c`, `feat_f` and 3) Pre-rendering output perturbation: `raw_c`, `raw_f`. Hyper-parameters `--pgd_alpha`, `--pgd_alpha`, `--pgd_iters`, `--pgd_eps` should be specified for each type of perturbations accordingly.\n\n### Visualization\n\nWhile training, users can view logging information through `tensorboard`:\n```\ntensorboard --logdir='./logs' --port \u003cyour_port\u003e --host 0.0.0.0\n```\n\nWhen training is done, users can evaluate and synthesize exhibition video by running:\n```\npython run_nerf.py --eval --eval_video --gpuid \u003cgpu_id\u003e --config \u003cdefault_config_file\u003e\n```\n\nTo visualize geometry, users can first generate density field with flag `--eval_vol`:\n```\npython run_nerf.py --eval --eval_vol --gpuid \u003cgpu_id\u003e --config \u003cdefault_config_file\u003e\n```\nThe exported volume will be saved into `\u003cexpname\u003e/eval/` directory (with both `.npy` and `.mrc` suffices). Then users can use Python library [PyMCubes](https://github.com/pmneila/PyMCubes) or [USCF Chimera](https://www.cgl.ucsf.edu/chimera/) to threshold and extract meshes from the density field.\n\n## Citation\n\nThis repository is build based on [NeRF-Pytorch](https://github.com/peihaowang/nerf-pytorch) repository.\n\nIf you find our code implementation helpful for your own resarch or work, please cite our paper.\n```\n@inproceedings{chen2022augnerf,\ntitle={Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations},\nauthor={Chen, Tianlong and Wang, Peihao and Fan, Zhiwen and Wang, Zhangyang},\nbooktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},\nyear={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVITA-Group%2FAug-NeRF","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FVITA-Group%2FAug-NeRF","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVITA-Group%2FAug-NeRF/lists"}