{"id":19728659,"url":"https://github.com/ashawkey/torch-ngp","last_synced_at":"2025-05-15T13:06:36.283Z","repository":{"id":37362572,"uuid":"450447497","full_name":"ashawkey/torch-ngp","owner":"ashawkey","description":"A pytorch CUDA extension implementation of instant-ngp (sdf and nerf), with a GUI.","archived":false,"fork":false,"pushed_at":"2023-11-10T03:06:57.000Z","size":1092,"stargazers_count":2164,"open_issues_count":84,"forks_count":282,"subscribers_count":39,"default_branch":"main","last_synced_at":"2025-04-22T03:30:54.426Z","etag":null,"topics":["d-nerf","gui","instant-ngp","nerf","pytorch","real-time","sdf","tensorf"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ashawkey.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-21T10:26:50.000Z","updated_at":"2025-04-15T13:54:43.000Z","dependencies_parsed_at":"2025-01-04T05:14:01.964Z","dependency_job_id":"ab5202f4-18c8-4866-a3f3-537e9788854a","html_url":"https://github.com/ashawkey/torch-ngp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashawkey%2Ftorch-ngp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashawkey%2Ftorch-ngp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashawkey%2Ftorch-ngp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashawkey%2Ftorch-ngp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ashawkey","download_url":"https://codeload.github.com/ashawkey/torch-ngp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254346624,"owners_count":22055808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["d-nerf","gui","instant-ngp","nerf","pytorch","real-time","sdf","tensorf"],"created_at":"2024-11-12T00:06:54.178Z","updated_at":"2025-05-15T13:06:36.234Z","avatar_url":"https://github.com/ashawkey.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# torch-ngp\n\nThis repository contains:\n* A pytorch implementation of the SDF and NeRF part (grid encoder, density grid ray sampler) in [instant-ngp](https://github.com/NVlabs/instant-ngp), as described in [_Instant Neural Graphics Primitives with a Multiresolution Hash Encoding_](https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.pdf).\n* A pytorch implementation of [TensoRF](https://github.com/apchenstu/TensoRF), as described in [_TensoRF: Tensorial Radiance Fields_](https://arxiv.org/abs/2203.09517), adapted to instant-ngp's NeRF framework.\n* A pytorch implementation of [CCNeRF](https://github.com/ashawkey/CCNeRF), as described in [_Compressible-composable NeRF via Rank-residual Decomposition_](https://arxiv.org/abs/2205.14870).\n* [New!] An implementation of [D-NeRF](https://github.com/albertpumarola/D-NeRF) adapted to instant-ngp's framework, as described in [_D-NeRF: Neural Radiance Fields for Dynamic Scenes_](https://openaccess.thecvf.com/content/CVPR2021/papers/Pumarola_D-NeRF_Neural_Radiance_Fields_for_Dynamic_Scenes_CVPR_2021_paper.pdf).\n* Some experimental features in the NeRF framework (e.g., text-guided NeRF editig similar to [CLIP-NeRF](https://arxiv.org/abs/2112.05139)).\n* A GUI for training/visualizing NeRF!\n\n**News**: A clean and improved version focusing on static NeRF reconstruction of realistic scenes has been separated into [nerf_template](https://github.com/ashawkey/nerf_template), as this repository has been hard to maintain.\n\n### [Gallery](assets/gallery.md) | [Update Logs](assets/update_logs.md)\n\nInstant-ngp interactive training/rendering on lego:\n\nhttps://user-images.githubusercontent.com/25863658/176174011-e7b7c4ab-9b6f-4f65-9952-7eceafe609b7.mp4\n\nAlso the first interactive deformable-nerf implementation:\n\nhttps://user-images.githubusercontent.com/25863658/175821784-63ba79f6-29be-47b5-b3fc-dab5282fce7a.mp4\n\n\n### Other related projects\n\n* [ngp_pl](https://github.com/kwea123/ngp_pl): PyTorch+CUDA trained with pytorch-lightning.\n\n* [JNeRF](https://github.com/Jittor/JNeRF): An NeRF benchmark based on Jittor.\n\n* [HashNeRF-pytorch](https://github.com/yashbhalgat/HashNeRF-pytorch): A pure PyTorch implementation.\n\n* [dreamfields-torch](https://github.com/ashawkey/dreamfields-torch): PyTorch+CUDA implementation of [_Zero-Shot Text-Guided Object Generation with Dream Fields_](https://arxiv.org/abs/2112.01455) based on this repository.\n\n# Install\n```bash\ngit clone --recursive https://github.com/ashawkey/torch-ngp.git\ncd torch-ngp\n```\n\n### Install with pip\n```bash\npip install -r requirements.txt\n\n# (optional) install the tcnn backbone\npip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch\n```\n\n### Install with conda\n```bash\nconda env create -f environment.yml\nconda activate torch-ngp\n```\n\n### Build extension (optional)\nBy default, we use [`load`](https://pytorch.org/docs/stable/cpp_extension.html#torch.utils.cpp_extension.load) to build the extension at runtime.\nHowever, this may be inconvenient sometimes.\nTherefore, we also provide the `setup.py` to build each extension:\n```bash\n# install all extension modules\nbash scripts/install_ext.sh\n\n# if you want to install manually, here is an example:\ncd raymarching\npython setup.py build_ext --inplace # build ext only, do not install (only can be used in the parent directory)\npip install . # install to python path (you still need the raymarching/ folder, since this only install the built extension.)\n```\n\n### Tested environments\n* Ubuntu 20 with torch 1.10 \u0026 CUDA 11.3 on a TITAN RTX.\n* Ubuntu 16 with torch 1.8 \u0026 CUDA 10.1 on a V100.\n* Windows 10 with torch 1.11 \u0026 CUDA 11.3 on a RTX 3070.\n\nCurrently, `--ff` only supports GPUs with CUDA architecture `\u003e= 70`.\nFor GPUs with lower architecture, `--tcnn` can still be used, but the speed will be slower compared to more recent GPUs.\n\n\n# Usage\n\nWe use the same data format as instant-ngp, e.g., [armadillo](https://github.com/NVlabs/instant-ngp/blob/master/data/sdf/armadillo.obj) and [fox](https://github.com/NVlabs/instant-ngp/tree/master/data/nerf/fox). \nPlease download and put them under `./data`.\n\nWe also support self-captured dataset and converting other formats (e.g., LLFF, Tanks\u0026Temples, Mip-NeRF 360) to the nerf-compatible format, with details in the following code block.\n\n\u003cdetails\u003e\n  \u003csummary\u003e Supported datasets \u003c/summary\u003e\n\n  * [nerf_synthetic](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) \n\n  * [Tanks\u0026Temples](https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip): [[conversion script]](./scripts/tanks2nerf.py)\n\n  * [LLFF](https://drive.google.com/drive/folders/14boI-o5hGO9srnWaaogTU5_ji7wkX2S7): [[conversion script]](./scripts/llff2nerf.py)\n\n  * [Mip-NeRF 360](http://storage.googleapis.com/gresearch/refraw360/360_v2.zip): [[conversion script]](./scripts/llff2nerf.py)\n\n  * (dynamic) [D-NeRF](https://www.dropbox.com/s/0bf6fl0ye2vz3vr/data.zip?dl=0)\n\n  * (dynamic) [Hyper-NeRF](https://github.com/google/hypernerf/releases/tag/v0.1): [[conversion script]](./scripts/hyper2nerf.py)\n\n\u003c/details\u003e\n\nFirst time running will take some time to compile the CUDA extensions.\n\n```bash\n### Instant-ngp NeRF\n# train with different backbones (with slower pytorch ray marching)\n# for the colmap dataset, the default dataset setting `--bound 2 --scale 0.33` is used.\npython main_nerf.py data/fox --workspace trial_nerf # fp32 mode\npython main_nerf.py data/fox --workspace trial_nerf --fp16 # fp16 mode (pytorch amp)\npython main_nerf.py data/fox --workspace trial_nerf --fp16 --ff # fp16 mode + FFMLP (this repo's implementation)\npython main_nerf.py data/fox --workspace trial_nerf --fp16 --tcnn # fp16 mode + official tinycudann's encoder \u0026 MLP\n\n# use CUDA to accelerate ray marching (much more faster!)\npython main_nerf.py data/fox --workspace trial_nerf --fp16 --cuda_ray # fp16 mode + cuda raymarching\n\n# preload data into GPU, accelerate training but use more GPU memory.\npython main_nerf.py data/fox --workspace trial_nerf --fp16 --preload\n\n# one for all: -O means --fp16 --cuda_ray --preload, which usually gives the best results balanced on speed \u0026 performance.\npython main_nerf.py data/fox --workspace trial_nerf -O\n\n# test mode\npython main_nerf.py data/fox --workspace trial_nerf -O --test\n\n# construct an error_map for each image, and sample rays based on the training error (slow down training but get better performance with the same number of training steps)\npython main_nerf.py data/fox --workspace trial_nerf -O --error_map\n\n# use a background model (e.g., a sphere with radius = 32), can supress noises for real-world 360 dataset\npython main_nerf.py data/firekeeper --workspace trial_nerf -O --bg_radius 32\n\n# start a GUI for NeRF training \u0026 visualization\n# always use with `--fp16 --cuda_ray` for an acceptable framerate!\npython main_nerf.py data/fox --workspace trial_nerf -O --gui\n\n# test mode for GUI\npython main_nerf.py data/fox --workspace trial_nerf -O --gui --test\n\n# for the blender dataset, you should add `--bound 1.0 --scale 0.8 --dt_gamma 0`\n# --bound means the scene is assumed to be inside box[-bound, bound]\n# --scale adjusts the camera locaction to make sure it falls inside the above bounding box. \n# --dt_gamma controls the adaptive ray marching speed, set to 0 turns it off.\npython main_nerf.py data/nerf_synthetic/lego --workspace trial_nerf -O --bound 1.0 --scale 0.8 --dt_gamma 0\npython main_nerf.py data/nerf_synthetic/lego --workspace trial_nerf -O --bound 1.0 --scale 0.8 --dt_gamma 0 --gui\n\n# for the LLFF dataset, you should first convert it to nerf-compatible format:\npython scripts/llff2nerf.py data/nerf_llff_data/fern # by default it use full-resolution images, and write `transforms.json` to the folder\npython scripts/llff2nerf.py data/nerf_llff_data/fern --images images_4 --downscale 4 # if you prefer to use the low-resolution images\n# then you can train as a colmap dataset (you'll need to tune the scale \u0026 bound if necessary):\npython main_nerf.py data/nerf_llff_data/fern --workspace trial_nerf -O\npython main_nerf.py data/nerf_llff_data/fern --workspace trial_nerf -O --gui\n\n# for the Tanks\u0026Temples dataset, you should first convert it to nerf-compatible format:\npython scripts/tanks2nerf.py data/TanksAndTemple/Family # write `trainsforms_{split}.json` for [train, val, test]\n# then you can train as a blender dataset (you'll need to tune the scale \u0026 bound if necessary)\npython main_nerf.py data/TanksAndTemple/Family --workspace trial_nerf_family -O --bound 1.0 --scale 0.33 --dt_gamma 0\npython main_nerf.py data/TanksAndTemple/Family --workspace trial_nerf_family -O --bound 1.0 --scale 0.33 --dt_gamma 0 --gui\n\n# for custom dataset, you should:\n# 1. take a video / many photos from different views \n# 2. put the video under a path like ./data/custom/video.mp4 or the images under ./data/custom/images/*.jpg.\n# 3. call the preprocess code: (should install ffmpeg and colmap first! refer to the file for more options)\npython scripts/colmap2nerf.py --video ./data/custom/video.mp4 --run_colmap # if use video\npython scripts/colmap2nerf.py --images ./data/custom/images/ --run_colmap # if use images\npython scripts/colmap2nerf.py --video ./data/custom/video.mp4 --run_colmap --dynamic # if the scene is dynamic (for D-NeRF settings), add the time for each frame.\n# 4. it should create the transform.json, and you can train with: (you'll need to try with different scale \u0026 bound \u0026 dt_gamma to make the object correctly located in the bounding box and render fluently.)\npython main_nerf.py data/custom --workspace trial_nerf_custom -O --gui --scale 2.0 --bound 1.0 --dt_gamma 0.02\n\n### Instant-ngp SDF\npython main_sdf.py data/armadillo.obj --workspace trial_sdf\npython main_sdf.py data/armadillo.obj --workspace trial_sdf --fp16\npython main_sdf.py data/armadillo.obj --workspace trial_sdf --fp16 --ff\npython main_sdf.py data/armadillo.obj --workspace trial_sdf --fp16 --tcnn\n\npython main_sdf.py data/armadillo.obj --workspace trial_sdf --fp16 --test\n\n### TensoRF\n# almost the same as Instant-ngp NeRF, just replace the main script.\npython main_tensoRF.py data/fox --workspace trial_tensoRF -O\npython main_tensoRF.py data/nerf_synthetic/lego --workspace trial_tensoRF -O --bound 1.0 --scale 0.8 --dt_gamma 0 \n\n### CCNeRF\n# training on single objects, turn on --error_map for better quality.\npython main_CCNeRF.py data/nerf_synthetic/chair --workspace trial_cc_chair -O --bound 1.0 --scale 0.67 --dt_gamma 0 --error_map\npython main_CCNeRF.py data/nerf_synthetic/ficus --workspace trial_cc_ficus -O --bound 1.0 --scale 0.67 --dt_gamma 0 --error_map\npython main_CCNeRF.py data/nerf_synthetic/hotdog --workspace trial_cc_hotdog -O --bound 1.0 --scale 0.67 --dt_gamma 0 --error_map\n# compose, use a larger bound and more samples per ray for better quality.\npython main_CCNeRF.py data/nerf_synthetic/hotdog --workspace trial_cc_hotdog -O --bound 2.0 --scale 0.67 --dt_gamma 0 --max_steps 2048 --test --compose\n# compose + gui, only about 1 FPS without dynamic resolution... just for quick verification of composition results.\npython main_CCNeRF.py data/nerf_synthetic/hotdog --workspace trial_cc_hotdog -O --bound 2.0 --scale 0.67 --dt_gamma 0 --test --compose --gui\n\n### D-NeRF\n# almost the same as Instant-ngp NeRF, just replace the main script.\n# use deformation to model dynamic scene\npython main_dnerf.py data/dnerf/jumpingjacks --workspace trial_dnerf_jumpingjacks -O --bound 1.0 --scale 0.8 --dt_gamma 0\npython main_dnerf.py data/dnerf/jumpingjacks --workspace trial_dnerf_jumpingjacks -O --bound 1.0 --scale 0.8 --dt_gamma 0 --gui\n# use temporal basis to model dynamic scene\npython main_dnerf.py data/dnerf/jumpingjacks --workspace trial_dnerf_basis_jumpingjacks -O --bound 1.0 --scale 0.8 --dt_gamma 0 --basis\npython main_dnerf.py data/dnerf/jumpingjacks --workspace trial_dnerf_basis_jumpingjacks -O --bound 1.0 --scale 0.8 --dt_gamma 0 --basis --gui\n# for the hypernerf dataset, first convert it into nerf-compatible format:\npython scripts/hyper2nerf.py data/split-cookie --downscale 2 # will generate transforms*.json\npython main_dnerf.py data/split-cookie/ --workspace trial_dnerf_cookies -O --bound 1 --scale 0.3 --dt_gamma 0\n```\n\ncheck the `scripts` directory for more provided examples.\n\n# Performance Reference\n\nTested with the default settings on the Lego dataset.\nHere the speed refers to the `iterations per second` on a V100.\n\n| Model | Split | PSNR | Train Speed | Test Speed |\n| - | - | - | - | - |\n| instant-ngp (paper)            | trainval?            | 36.39  |  -   | -    |\n| instant-ngp (`-O`)             | train (30K steps)    | 34.15  |  97  | 7.8  |\n| instant-ngp (`-O --error_map`) | train (30K steps)    | 34.88  |  50  | 7.8  |\n| instant-ngp (`-O`)             | trainval (40k steps) | 35.22  |  97  | 7.8  |\n| instant-ngp (`-O --error_map`) | trainval (40k steps) | 36.00  |  50  | 7.8  |\n| TensoRF (paper)                | train (30K steps)    | 36.46  |  -   | -    |\n| TensoRF (`-O`)                 | train (30K steps)    | 35.05  |  51  | 2.8  |\n| TensoRF (`-O --error_map`)     | train (30K steps)    | 35.84  |  14  | 2.8  |\n\n# Tips\n\n**Q**: How to choose the network backbone? \n\n**A**: The `-O` flag which uses pytorch's native mixed precision is suitable for most cases. I don't find very significant improvement for `--tcnn` and `--ff`, and they require extra building. Also, some new features may only be available for the default `-O` mode.\n\n**Q**: CUDA Out Of Memory for my dataset.\n\n**A**: You could try to turn off `--preload` which loads all images in to GPU for acceleration (if use `-O`, change it to `--fp16 --cuda_ray`). Another solution is to manually set `downscale` in `NeRFDataset` to lower the image resolution.\n\n**Q**: How to adjust `bound` and `scale`? \n\n**A**: You could start with a large `bound` (e.g., 16) or a small `scale` (e.g., 0.3) to make sure the object falls into the bounding box. The GUI mode can be used to interactively shrink the `bound` to find the suitable value. Uncommenting [this line](https://github.com/ashawkey/torch-ngp/blob/main/nerf/provider.py#L219) will visualize the camera poses, and some good examples can be found in [this issue](https://github.com/ashawkey/torch-ngp/issues/59).\n\n**Q**: Noisy novel views for realistic datasets.\n\n**A**: You could try setting `bg_radius` to a large value, e.g., 32. It trains an extra environment map to model the background in realistic photos. A larger `bound` will also help.\nAn example for `bg_radius` in the [firekeeper](https://drive.google.com/file/d/19C0K6_crJ5A9ftHijUmJysxmY-G4DMzq/view?usp=sharing) dataset:\n![bg_model](./assets/bg_model.jpg)\n\n\n# Difference from the original implementation\n\n* Instead of assuming the scene is bounded in the unit box `[0, 1]` and centered at `(0.5, 0.5, 0.5)`, this repo assumes **the scene is bounded in box `[-bound, bound]`, and centered at `(0, 0, 0)`**. Therefore, the functionality of `aabb_scale` is replaced by `bound` here.\n* For the hashgrid encoder, this repo only implements the linear interpolation mode.\n* For TensoRF, we don't implement regularizations other than L1, and use `trunc_exp` as the density activation instead of `softplus`. The alpha mask pruning is replaced by the density grid sampler from instant-ngp, which shares the same logic for acceleration.\n\n\n# Citation\n\nIf you find this work useful, a citation will be appreciated via:\n```\n@misc{torch-ngp,\n    Author = {Jiaxiang Tang},\n    Year = {2022},\n    Note = {https://github.com/ashawkey/torch-ngp},\n    Title = {Torch-ngp: a PyTorch implementation of instant-ngp}\n}\n\n@article{tang2022compressible,\n    title = {Compressible-composable NeRF via Rank-residual Decomposition},\n    author = {Tang, Jiaxiang and Chen, Xiaokang and Wang, Jingbo and Zeng, Gang},\n    journal = {arXiv preprint arXiv:2205.14870},\n    year = {2022}\n}\n```\n\n# Acknowledgement\n\n* Credits to [Thomas Müller](https://tom94.net/) for the amazing [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn) and [instant-ngp](https://github.com/NVlabs/instant-ngp):\n    ```\n    @misc{tiny-cuda-nn,\n        Author = {Thomas M\\\"uller},\n        Year = {2021},\n        Note = {https://github.com/nvlabs/tiny-cuda-nn},\n        Title = {Tiny {CUDA} Neural Network Framework}\n    }\n\n    @article{mueller2022instant,\n        title = {Instant Neural Graphics Primitives with a Multiresolution Hash Encoding},\n        author = {Thomas M\\\"uller and Alex Evans and Christoph Schied and Alexander Keller},\n        journal = {arXiv:2201.05989},\n        year = {2022},\n        month = jan\n    }\n    ```\n\n* The framework of NeRF is adapted from [nerf_pl](https://github.com/kwea123/nerf_pl):\n    ```\n    @misc{queianchen_nerf,\n        author = {Quei-An, Chen},\n        title = {Nerf_pl: a pytorch-lightning implementation of NeRF},\n        url = {https://github.com/kwea123/nerf_pl/},\n        year = {2020},\n    }\n    ```\n\n* The official TensoRF [implementation](https://github.com/apchenstu/TensoRF):\n    ```\n    @article{TensoRF,\n      title={TensoRF: Tensorial Radiance Fields},\n      author={Chen, Anpei and Xu, Zexiang and Geiger, Andreas and Yu, Jingyi and Su, Hao},\n      journal={arXiv preprint arXiv:2203.09517},\n      year={2022}\n    }\n    ```\n\n* The NeRF GUI is developed with [DearPyGui](https://github.com/hoffstadt/DearPyGui).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashawkey%2Ftorch-ngp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fashawkey%2Ftorch-ngp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashawkey%2Ftorch-ngp/lists"}