{"id":18614381,"url":"https://github.com/sxyu/plenoctree","last_synced_at":"2025-04-05T21:07:17.656Z","repository":{"id":40603635,"uuid":"358003717","full_name":"sxyu/plenoctree","owner":"sxyu","description":"PlenOctrees: NeRF-SH Training \u0026 Conversion","archived":false,"fork":false,"pushed_at":"2022-04-06T05:23:01.000Z","size":164,"stargazers_count":430,"open_issues_count":25,"forks_count":76,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-03-29T20:04:55.850Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://alexyu.net/plenoctrees","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sxyu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-14T18:25:26.000Z","updated_at":"2025-03-17T00:48:05.000Z","dependencies_parsed_at":"2022-08-09T23:50:35.049Z","dependency_job_id":null,"html_url":"https://github.com/sxyu/plenoctree","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sxyu%2Fplenoctree","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sxyu%2Fplenoctree/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sxyu%2Fplenoctree/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sxyu%2Fplenoctree/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sxyu","download_url":"https://codeload.github.com/sxyu/plenoctree/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247399876,"owners_count":20932876,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T03:25:55.738Z","updated_at":"2025-04-05T21:07:17.639Z","avatar_url":"https://github.com/sxyu.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF"],"readme":"# PlenOctrees Official Repo: NeRF-SH training and conversion\n\nThis repository contains code to train NeRF-SH and\nto extract the PlenOctree, constituting part of the code release for:\n\nPlenOctrees for Real Time Rendering of Neural Radiance Fields\u003cbr\u003e\nAlex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, Angjoo Kanazawa\n\nhttps://alexyu.net/plenoctrees\n\n```\n@inproceedings{yu2021plenoctrees,\n      title={{PlenOctrees} for Real-time Rendering of Neural Radiance Fields},\n      author={Alex Yu and Ruilong Li and Matthew Tancik and Hao Li and Ren Ng and Angjoo Kanazawa},\n      year={2021},\n      booktitle={ICCV},\n}\n```\n\nPlease see the following repository for our C++ PlenOctrees volume renderer:\n\u003chttps://github.com/sxyu/volrend\u003e\n\n## Setup\n\nPlease use conda for a replicable environment.\n```\nconda env create -f environment.yml\nconda activate plenoctree\npip install --upgrade pip\n```\n\nOr you can install the dependencies manually by:\n```\nconda install pytorch torchvision cudatoolkit=11.0 -c pytorch\nconda install tqdm\npip install -r requirements.txt\n```\n\n[Optional] Install GPU and TPU support for Jax. This is useful for NeRF-SH training.\nRemember to **change cuda110 to your CUDA version**, e.g. cuda102 for CUDA 10.2.\n```\npip install --upgrade jax jaxlib==0.1.65+cuda110 -f https://storage.googleapis.com/jax-releases/jax_releases.html\n```\n\n## NeRF-SH Training\n\nWe release our trained NeRF-SH models as well as converted plenoctrees at \n[Google Drive](https://drive.google.com/drive/folders/1J0lRiDn_wOiLVpCraf6jM7vvCwDr9Dmx?usp=sharing). \nYou can also use the following commands to reproduce the NeRF-SH models.\n\nTraining and evaluation on the **NeRF-Synthetic dataset** ([Google Drive](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1)):\n```\nexport DATA_ROOT=./data/NeRF/nerf_synthetic/\nexport CKPT_ROOT=./data/Plenoctree/checkpoints/syn_sh16/\nexport SCENE=chair\nexport CONFIG_FILE=nerf_sh/config/blender\n\npython -m nerf_sh.train \\\n    --train_dir $CKPT_ROOT/$SCENE/ \\\n    --config $CONFIG_FILE \\\n    --data_dir $DATA_ROOT/$SCENE/\n\npython -m nerf_sh.eval \\\n    --chunk 4096 \\\n    --train_dir $CKPT_ROOT/$SCENE/ \\\n    --config $CONFIG_FILE \\\n    --data_dir $DATA_ROOT/$SCENE/\n```\nNote for `SCENE=mic`, we adopt a warmup learning rate schedule (`--lr_delay_steps 50000 --lr_delay_mult 0.01`) to avoid unstable initialization.\n\n\nTraining and evaluation on **TanksAndTemple dataset** \n([Download Link](https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip)) from the [NSVF](https://github.com/facebookresearch/NSVF) paper:\n```\nexport DATA_ROOT=./data/TanksAndTemple/\nexport CKPT_ROOT=./data/Plenoctree/checkpoints/tt_sh25/\nexport SCENE=Barn\nexport CONFIG_FILE=nerf_sh/config/tt\n\npython -m nerf_sh.train \\\n    --train_dir $CKPT_ROOT/$SCENE/ \\\n    --config $CONFIG_FILE \\\n    --data_dir $DATA_ROOT/$SCENE/\n\npython -m nerf_sh.eval \\\n    --chunk 4096 \\\n    --train_dir $CKPT_ROOT/$SCENE/ \\\n    --config $CONFIG_FILE \\\n    --data_dir $DATA_ROOT/$SCENE/\n```\n\n## PlenOctrees Conversion and Optimization\n\nBefore converting the NeRF-SH models into plenoctrees, you should already have the \nNeRF-SH models trained/downloaded and placed at `./data/Plenoctree/checkpoints/{syn_sh16, tt_sh25}/`. \nAlso make sure you have the training data placed at \n`./data/NeRF/nerf_synthetic` and/or `./data/TanksAndTemple`.\n\nTo reproduce our results in the paper, you can simplly run:\n```\n# NeRF-Synthetic dataset\npython -m octree.task_manager octree/config/syn_sh16.json --gpus=\"0 1 2 3\"\n\n# TanksAndTemple dataset\npython -m octree.task_manager octree/config/tt_sh25.json --gpus=\"0 1 2 3\"\n```\nThe above command will parallel all scenes in the dataset across the gpus you set. The json files \ncontain dedicated hyper-parameters towards better performance (PSNR, SSIM, LPIPS). So in this setting, a 24GB GPU is\nneeded for each scene and in averange the process takes about 15 minutes to finish. The converted plenoctree\nwill be saved to `./data/Plenoctree/checkpoints/{syn_sh16, tt_sh25}/$SCENE/octrees/`.\n\n\nBelow is a more straight-forward script for demonstration purpose:\n```\nexport DATA_ROOT=./data/NeRF/nerf_synthetic/\nexport CKPT_ROOT=./data/Plenoctree/checkpoints/syn_sh16\nexport SCENE=chair\nexport CONFIG_FILE=nerf_sh/config/blender\n\npython -m octree.extraction \\\n    --train_dir $CKPT_ROOT/$SCENE/ --is_jaxnerf_ckpt \\\n    --config $CONFIG_FILE \\\n    --data_dir $DATA_ROOT/$SCENE/ \\\n    --output $CKPT_ROOT/$SCENE/octrees/tree.npz\n\npython -m octree.optimization \\\n    --input $CKPT_ROOT/$SCENE/tree.npz \\\n    --config $CONFIG_FILE \\\n    --data_dir $DATA_ROOT/$SCENE/ \\\n    --output $CKPT_ROOT/$SCENE/octrees/tree_opt.npz\n\npython -m octree.evaluation \\\n    --input $CKPT_ROOT/$SCENE/octrees/tree_opt.npz \\\n    --config $CONFIG_FILE \\\n    --data_dir $DATA_ROOT/$SCENE/\n\n# [Optional] Only used for in-browser viewing.\npython -m octree.compression \\\n    $CKPT_ROOT/$SCENE/octrees/tree_opt.npz \\\n    --out_dir $CKPT_ROOT/$SCENE/ \\\n    --overwrite\n```\n\n## MISC\n\n### Project Vanilla NeRF to PlenOctree\n\nA vanilla trained NeRF can also be converted to a plenoctree for fast inference. To mimic the \nview-independency propertity as in a NeRF-SH model, we project the vanilla NeRF model to SH basis functions\nby sampling view directions for every points in the space. Though this makes converting vanilla NeRF to\na plenoctree possible, the projection process inevitability loses the quality of the model, even with a large amount \nof sampling view directions (which takes hours to finish). So we recommend to just directly train a NeRF-SH model end-to-end.\n\nBelow is a example of projecting a trained vanilla NeRF model from \n[JaxNeRF repo](https://github.com/google-research/google-research/tree/master/jaxnerf) \n([Download Link](http://storage.googleapis.com/gresearch/jaxnerf/jaxnerf_pretrained_models.zip)) to a plenoctree. \nAfter extraction, you can optimize \u0026 evaluate \u0026 compress the plenoctree just like usual:\n```\nexport DATA_ROOT=./data/NeRF/nerf_synthetic/ \nexport CKPT_ROOT=./data/JaxNeRF/jaxnerf_models/blender/ \nexport SCENE=drums\nexport CONFIG_FILE=nerf_sh/config/misc/proj\n\npython -m octree.extraction \\\n    --train_dir $CKPT_ROOT/$SCENE/ --is_jaxnerf_ckpt \\\n    --config $CONFIG_FILE \\\n    --data_dir $DATA_ROOT/$SCENE/ \\\n    --output $CKPT_ROOT/$SCENE/octrees/tree.npz \\\n    --projection_samples 100 \\\n    --radius 1.3\n```\nNote `--projection_samples` controls how many sampling view directions are used. More sampling view directions give better\nprojection quality but takes longer time to finish. For example, for the `drums` scene \nin the NeRF-Synthetic dataset, `100 / 10000` sampling view directions takes about `2 mins / 2 hours` to finish the plenoctree extraction. \nIt produce *raw* plenoctrees with `PSNR=22.49 / 23.84` (before optimization). Note that extraction from a NeRF-SH model produce \na *raw* plenoctree with `PSNR=25.01`.\n\n### List of possible improvements\n\nIn the interst reproducibility, the parameters used in the paper are also used here.\nFor future work we recommend trying the changes in mip-NeRF \u003chttps://jonbarron.info/mipnerf/\u003e \nfor improved stability and quality:\n\n- Centered pixels (+ 0.5 on x, y) when generating rays\n- Use shifted SoftPlus instead of ReLU for density (including for octree optimization)\n- Pad the RGB sigmoid output (avoid low gradient region near 0/1 color)\n- Multi-scale training from mip-NeRF\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsxyu%2Fplenoctree","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsxyu%2Fplenoctree","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsxyu%2Fplenoctree/lists"}