{"id":18614368,"url":"https://github.com/thomasneff/AdaNeRF","last_synced_at":"2025-04-11T00:30:39.978Z","repository":{"id":47111446,"uuid":"515850897","full_name":"thomasneff/AdaNeRF","owner":"thomasneff","description":"AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields","archived":false,"fork":false,"pushed_at":"2022-11-08T11:49:31.000Z","size":8809,"stargazers_count":244,"open_issues_count":5,"forks_count":10,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-11-07T03:31:09.346Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thomasneff.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-20T05:51:39.000Z","updated_at":"2024-07-26T12:43:49.000Z","dependencies_parsed_at":"2023-01-22T05:03:34.074Z","dependency_job_id":null,"html_url":"https://github.com/thomasneff/AdaNeRF","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomasneff%2FAdaNeRF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomasneff%2FAdaNeRF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomasneff%2FAdaNeRF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomasneff%2FAdaNeRF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thomasneff","download_url":"https://codeload.github.com/thomasneff/AdaNeRF/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322208,"owners_count":21084333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T03:25:55.501Z","updated_at":"2025-04-11T00:30:35.718Z","avatar_url":"https://github.com/thomasneff.png","language":"C++","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF"],"readme":"# AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields\n [Andreas Kurz](https://online.tugraz.at/tug_online/visitenkarte.show_vcard?pPersonenGruppe=3\u0026pPersonenId=D715516087483BD3)\\*\u003csup\u003e1\u003c/sup\u003e,\n [Thomas Neff](https://thomasneff.github.io/)\\*\u003csup\u003e1\u003c/sup\u003e,\n [Zhaoyang Lv](https://lvzhaoyang.github.io/)\u003csup\u003e2\u003c/sup\u003e,\n [Michael Zollhöfer](https://zollhoefer.com/)\u003csup\u003e2\u003c/sup\u003e,\n [Markus Steinberger](https://www.markussteinberger.net/)\u003csup\u003e1\u003c/sup\u003e\n\n \u003csup\u003e1\u003c/sup\u003eGraz University of Technology, \u003csup\u003e2\u003c/sup\u003eReality Labs Research  \n  \\*denotes equal contribution  \nin ECCV 2022 (to appear)\n\n\u003cimg src='adanerf_teaser.png'/\u003e\n\n### [Project Page](https://thomasneff.github.io/adanerf/) | [Paper (ArXiV)](https://arxiv.org/abs/2207.10312)\n\n### Licensing\nThe majority of this project is licensed under CC-BY-NC, except for adapted third-party code, which is available under separate license terms:\n\n* [nerf](https://github.com/bmild/nerf) is licensed under the MIT license\n* [nerf-pytorch](https://github.com/yenchenlin/nerf-pytorch) is licensed under the MIT license\n* [FLIP](https://github.com/NVlabs/flip) is licensed under the BSD-3 license\n* [Python-IW-SSIM](https://github.com/Jack-guo-xy/Python-IW-SSIM) is licensed under the BSD license\n* [DONeRF](https://github.com/facebookresearch/DONERF) is licensed under the CC-BY-NC license.\n\n\n### General\nThis repository contains the source code for the paper \"AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields\", as well as a customized/partial port of the [nerf-pytorch](https://github.com/yenchenlin/nerf-pytorch) codebase by Yen-Chen Lin.\n\nThe codebase has been tested on Ubuntu 20.04 and Arch using an RTX3090 with 24 GB of VRAM, and should also work on other distributions.\n\n### Repo Structure\n`configs/` contains example configuration files to get started with experiments.\n\n`src/` contains the pytorch training/inference framework that handles training of all supported network types.\n\n`requirements.txt` lists the required python packages for the code base. We recommend conda to setup the development environment. Note that PyTorch 1.8 is the minimum working version due to earlier versions having issues with the parallel dataloaders.\n\n\n### Datasets\nWe used the umodified DONeRF dataset in our project. More information on the dataset and the dataset itself can be found on the official DONeRF repository [here](https://github.com/facebookresearch/DONERF).\n\nThe standard LLFF dataset can be modified to work with our codebase with the included `convert_llff.py` script. The (unconverted) LLFF dataset can be found [here](https://www.matthewtancik.com/nerf).\n\n### Training / Example Commands\nTo train a network with a given configuration file, you can adapt the following examplary command, executed from within the `src/` directory. All things in angle brackets `\u003cEXAMPLE\u003e` need to be replaced by specific values depending on your use case, please refer to `src/util/config.py` for all valid configutation options. All configuration options can also be supplied via the command line.\n\nThe following basic command performs dense training of an AdaNeRF network on the given dataset. The dense training uses 128 samples in the shading network.\n\n```bash\npython train.py -c ../configs/dense_training.ini --data \u003cPATH_TO_DATASET_DIRECTORY\u003e --logDir \u003cPATH_TO_OUTPUT_DIRECTORY\u003e \n```\n\nA specific CUDA device can be chosen for training by supplying the `--device` argument:\n\n```bash\npython train.py -c ../configs/dense_training.ini --data \u003cPATH_TO_DATASET_DIRECTORY\u003e --logDir \u003cPATH_TO_OUTPUT_DIRECTORY\u003e --device \u003cDEVICE_ID\u003e\n```\n\nAs the weight for our adaptive sampling network needs to be tuned per scene, it can be supplied to the network with the `--lossWeights` argument:\n\n```bash\npython train.py -c ../configs/dense_training.ini --data \u003cPATH_TO_DATASET_DIRECTORY\u003e --logDir \u003cPATH_TO_OUTPUT_DIRECTORY\u003e --device \u003cDEVICE_ID\u003e --lossWeights \u003cLOSS_WEIGHT_PER_SCENE\u003e --lossWeights 1.0\n```\n\n(Important to note here is that we pass a second lossWeights with a value of 1.0 for our shading network.)\n\nBy default, our dataloader loads images on-demand by using 8 parallel workers. To store all data on the GPU at all times (for faster training), supply the `--storeFullData` argument:\n\n```bash\npython train.py -c ../configs/dense_training.ini --data \u003cPATH_TO_DATASET_DIRECTORY\u003e --logDir \u003cPATH_TO_OUTPUT_DIRECTORY\u003e --device \u003cDEVICE_ID\u003e --lossWeights \u003cLOSS_WEIGHT_PER_SCENE\u003e --lossWeights 1.0 --storeFullData\n```\n\n(Due to both networks being trained simultaneously, it may not be possible to store the image data on the GPU as well, unless a high-end GPU is utilized.)\n\nTo perform the fine training of the above densely trained AdaNeRF network, the following command can be used:\n\n```bash\npython train.py -c ../configs/fine_training.ini --data \u003cPATH_TO_DATASET_DIRECTORY\u003e --logDir \u003cPATH_TO_OUTPUT_DIRECTORY\u003e --lossWeights \u003cLOSS_WEIGHT_PER_SCENE\u003e --lossWeights 1.0 --preTrained \u003cPATH_TO_DENSE_NETWORK\u003e --preTrained \u003cPATH_TO_DENSE_NETWORK\u003e\n```\n\nTo change the threshold used for adaptive sampling, simply supply the `adaptiveSamplingThreshold` argument for fine training:\n\n```bash\npython train.py -c ../configs/fine_training.ini --data \u003cPATH_TO_DATASET_DIRECTORY\u003e --logDir \u003cPATH_TO_OUTPUT_DIRECTORY\u003e --lossWeights \u003cLOSS_WEIGHT_PER_SCENE\u003e --lossWeights 1.0 --preTrained \u003cPATH_TO_DENSE_NETWORK\u003e --preTrained \u003cPATH_TO_DENSE_NETWORK\u003e --adaptiveSamplingThreshold \u003cTHRESHOLD\u003e\n```\n\nThe complete example commands to perform dense as well as fine training for an AdaNeRF network with 8 samples per ray on the classroom dataset using the CUDA Device 0, a loss weight of 0.01, adaptive sampling threshold of 0.2, and storing the outputs in /data/output_results/ could look like this:\n\n```bash\npython train.py -c ../configs/dense_training.ini --data /data/classroom/ --logDir /data/output_results/  --device 0 --lossWeights 0.01 --lossWeights 1.0\npython train.py -c ../configs/fine_training.ini --data /data/classroom/ --logDir /data/output_results/ --device 0 --lossWeights 0.01 --lossWeights 1.0 --preTrained /data/output_results/ --preTrained /data/output_results/ --adaptiveSamplingThreshold 0.2 --numRaymarchSamples 8 --numRaymarchSamples 8\n```\n\n(Important to note here is that we pass numRaymarchSamples and preTrained twice - the first value is actually ignored for numRaymarchSamples since the first network in this particular config file does not use raymarching, but certain config options, like preTrained, are specified per network.)\n\n#### Memory\nAs AdaNeRF trains both the adaptive sampling network and the shading network concurrently, the memory requirements during the dense training are very high.\n\nTo allow training on lower-end GPUs, the number of images and samples per batch can be reduced.\n\nThis can be done by modifying the `batchImages` and `samples` parameters.\n\nThe supplied config files use `2` and `2048` respectively, which runs on an RTX3090.\n\nDepending on the used hardware, the parameters may have to be reduced to `1` and `1024` respectively.\n\n### Testing / Example Commands\n\nBy default, the framework produces rendered output image every `epochsRender` iterations, and validates on the validation set every `epochsValidate` iterations.\n\nVideos can be generated by supplying json paths for the poses, and `epochsVideo` will produce a video from a predefined path at regular intervals. \n\nFor running just an inference pass for all the test images and for a given video path, you can use `src/test.py`. \n\nThis also takes the same arguments and configuration files as `src/train.py` does, so following the example for the training command, you can use `src/test.py` as follows:\n\n```bash\npython test.py -c ../configs/fine_training.ini --data /data/classroom/ --logDir /data/output_results/ --device 0 --lossWeights 0.01 --lossWeights 1.0 --adaptiveSamplingThreshold 0.2 --numRaymarchSamples 8 --numRaymarchSamples 8 --camPath cam_path_rotate --outputVideoName cam_path_rotate --videoFrames 300\n```\n\n(In this case no preTrained arguments are necessary, as the network should already be trained and does not have to load the model parameters of the densely trained network.)\n\n### Evaluation\n\nTo generate quantitative results (and also output images/videos/diffs similar to what `src/test.py` can also do), you can use `src/evaluate.py`. \nTo directly evaluate after training, supply the `--performEvaluation` flag to any training command.\nThis script only requires the `--data` and `--logDir` options to locate the results of the training procedure, and has some additional evaluation-specific options that can be inspected at the top of `def main()` (such as being able to skip certain evaluation procedures or only evaluate specific things).\n\n`src/evaluate.py` performs the evaluation on all subdirectories (if it hasn't done so already), so you only need to run this script once for a specific dataset and all containing results are evaluated sequentially.\n\nTo aggregate the resulting outputs (MSE, SSIM, FLIP, FLOP / Pixel, Number of Parameters), you can use `src/comparison.py` to generate a resulting `.csv` file.\n\n### Generate new data\n\nWe also provide basic script templates that can be used to generate data using Blender's python scripting engine. `blender_export_per_scene_stub.py` can either be copied into the Blender script editor or opened directly, and is used to configure parameters for view cell generation. It then calls `blender_export.py` to generate the images and depth data (for DONeRF) if desired.\n\nFor the scripts to work properly, the name of the Camera needs to be set correctly. If depth buffers should be rendered, the `RENDER_IMG_NAME` variable needs to be set to the compositor node that exports the depth image in Blender. \nFinally, the script uses the rendering settings (number of samples, rendering backend, CPU/GPU configs) that are set in the Blender project, so make sure that these are correct.\n\n### Citation\n\nIf you find this repository useful in any way or use/modify AdaNeRF in your research, please consider citing our paper:\n\n\n```bibtex\n@article{kurz-adanerf2022,\n title = {AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields},\n author = {Kurz, Andreas and Neff, Thomas and Lv, Zhaoyang and Zollh\\\"{o}fer, Michael and Steinberger, Markus},\n booktitle = {European Conference on Computer Vision (ECCV)},\n year = {2022},\n}\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthomasneff%2FAdaNeRF","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthomasneff%2FAdaNeRF","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthomasneff%2FAdaNeRF/lists"}