{"id":17632729,"url":"https://github.com/kwea123/ngp_pl","last_synced_at":"2025-05-16T08:05:06.303Z","repository":{"id":41269171,"uuid":"508950989","full_name":"kwea123/ngp_pl","owner":"kwea123","description":"Instant-ngp in pytorch+cuda trained with pytorch-lightning (high quality with high speed, with only few lines of legible code)","archived":false,"fork":false,"pushed_at":"2023-06-16T01:03:45.000Z","size":15437,"stargazers_count":1278,"open_issues_count":41,"forks_count":158,"subscribers_count":23,"default_branch":"master","last_synced_at":"2025-04-08T19:23:56.826Z","etag":null,"topics":["3d-reconstruction","cuda","instant-ngp","nerf","novel-view-synthesis","pytorch","pytorch-lightning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kwea123.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"kwea123","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2022-06-30T05:53:43.000Z","updated_at":"2025-04-06T13:09:12.000Z","dependencies_parsed_at":"2024-10-23T10:17:52.630Z","dependency_job_id":null,"html_url":"https://github.com/kwea123/ngp_pl","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwea123%2Fngp_pl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwea123%2Fngp_pl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwea123%2Fngp_pl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwea123%2Fngp_pl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kwea123","download_url":"https://codeload.github.com/kwea123/ngp_pl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254493378,"owners_count":22080126,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","cuda","instant-ngp","nerf","novel-view-synthesis","pytorch","pytorch-lightning"],"created_at":"2024-10-23T01:45:19.060Z","updated_at":"2025-05-16T08:05:01.293Z","avatar_url":"https://github.com/kwea123.png","language":"Jupyter Notebook","funding_links":["https://github.com/sponsors/kwea123"],"categories":[],"sub_categories":[],"readme":"# ngp_pl\n\n### Advertisement: Check out the latest integrated project [nerfstudio](https://github.com/nerfstudio-project/nerfstudio)! There are a lot of recent improvements on nerf related methods, including instant-ngp!\n\n\u003c!-- ### Update 2022 July 29th: GUI prototype is available now (see following video)!\n\n### Update 2022 July 24th: Training on custom data is possible now!\n\n### Update 2022 July 14th: Multi-GPU training is available now! With multiple GPUs, now you can achieve high quality under a minute! --\u003e\n\nInstant-ngp (only NeRF) in pytorch+cuda trained with pytorch-lightning (**high quality with high speed**). This repo aims at providing a concise pytorch interface to facilitate future research, and am grateful if you can share it (and a citation is highly appreciated)!\n\n*  [Official CUDA implementation](https://github.com/NVlabs/instant-ngp/tree/master)\n*  [torch-ngp](https://github.com/ashawkey/torch-ngp) another pytorch implementation that I highly referenced.\n\n# :paintbrush: Gallery\n\nhttps://user-images.githubusercontent.com/11364490/181671484-d5e154c8-6cea-4d52-94b5-1e5dd92955f2.mp4\n\nOther representative videos are in [GALLERY.md](GALLERY.md)\n\n# :computer: Installation\n\nThis implementation has **strict** requirements due to dependencies on other libraries, if you encounter installation problem due to hardware/software mismatch, I'm afraid there is **no intention** to support different platforms (you are welcomed to contribute).\n\n## Hardware\n\n* OS: Ubuntu 20.04\n* NVIDIA GPU with Compute Compatibility \u003e= 75 and memory \u003e 6GB (Tested with RTX 2080 Ti), CUDA 11.3 (might work with older version)\n* 32GB RAM (in order to load full size images)\n\n## Software\n\n* Clone this repo by `git clone https://github.com/kwea123/ngp_pl`\n* Python\u003e=3.8 (installation via [anaconda](https://www.anaconda.com/distribution/) is recommended, use `conda create -n ngp_pl python=3.8` to create a conda environment and activate it by `conda activate ngp_pl`)\n* Python libraries\n    * Install pytorch by `pip install torch==1.11.0 --extra-index-url https://download.pytorch.org/whl/cu113`\n    * Install `torch-scatter` following their [instruction](https://github.com/rusty1s/pytorch_scatter#installation)\n    * Install `tinycudann` following their [instruction](https://github.com/NVlabs/tiny-cuda-nn#pytorch-extension) (pytorch extension)\n    * Install `apex` following their [instruction](https://github.com/NVIDIA/apex#linux)\n    * Install core requirements by `pip install -r requirements.txt`\n\n* Cuda extension: Upgrade `pip` to \u003e= 22.1 and run `pip install models/csrc/` (please run this each time you `pull` the code)\n\n# :books: Supported Datasets\n\n1.  NSVF data\n\nDownload preprocessed datasets (`Synthetic_NeRF`, `Synthetic_NSVF`, `BlendedMVS`, `TanksAndTemples`) from [NSVF](https://github.com/facebookresearch/NSVF#dataset). **Do not change the folder names** since there is some hard-coded fix in my dataloader.\n\n2.  NeRF++ data\n\nDownload data from [here](https://github.com/Kai-46/nerfplusplus#data).\n\n3.  Colmap data\n\nFor custom data, run `colmap` and get a folder `sparse/0` under which there are `cameras.bin`, `images.bin` and `points3D.bin`. The following data with colmap format are also supported:\n\n  *  [nerf_llff_data](https://drive.google.com/file/d/16VnMcF1KJYxN9QId6TClMsZRahHNMW5g/view?usp=sharing) \n  *  [mipnerf360 data](http://storage.googleapis.com/gresearch/refraw360/360_v2.zip)\n  *  [HDR-NeRF data](https://drive.google.com/drive/folders/1OTDLLH8ydKX1DcaNpbQ46LlP0dKx6E-I). Additionally, download my colmap pose estimation from [here](https://drive.google.com/file/d/1TXxgf_ZxNB4o67FVD_r0aBUIZVRgZYMX/view?usp=sharing) and extract to the same location.\n\n4. RTMV data\n\nDownload data from [here](http://www.cs.umd.edu/~mmeshry/projects/rtmv/). To convert the hdr images into ldr images for training, run `python misc/prepare_rtmv.py \u003cpath/to/RTMV\u003e`, it will create `images/` folder under each scene folder, and will use these images to train (and test).\n\n# :key: Training\n\nQuickstart: `python train.py --root_dir \u003cpath/to/lego\u003e --exp_name Lego`\n\nIt will train the Lego scene for 30k steps (each step with 8192 rays), and perform one testing at the end. The training process should finish within about 5 minutes (saving testing image is slow, add `--no_save_test` to disable). Testing PSNR will be shown at the end.\n\nMore options can be found in [opt.py](opt.py).\n\nFor other public dataset training, please refer to the scripts under `benchmarking`.\n\n# :mag_right: Testing\n\nUse `test.ipynb` to generate images. Lego pretrained model is available [here](https://github.com/kwea123/ngp_pl/releases/tag/v1.0)\n\nGUI usage: run `python show_gui.py` followed by the **same** hyperparameters used in training (`dataset_name`, `root_dir`, etc) and **add the checkpoint path** with `--ckpt_path \u003cpath/to/.ckpt\u003e`\n\n# Comparison with torch-ngp and the paper\n\nI compared the quality (average testing PSNR on `Synthetic-NeRF`) and the inference speed (on `Lego` scene) v.s. the concurrent work torch-ngp (default settings) and the paper, all trained for about 5 minutes:\n\n| Method    | avg PSNR | FPS   | GPU     |\n| :---:     | :---:    | :---: | :---:   |\n| torch-ngp | 31.46    | 18.2  | 2080 Ti |\n| mine      | 32.96    | 36.2  | 2080 Ti |\n| instant-ngp paper | **33.18** | **60** | 3090 |\n\nAs for quality, mine is slightly better than torch-ngp, but the result might fluctuate across different runs.\n\nAs for speed, mine is faster than torch-ngp, but is still only half fast as instant-ngp. Speed is dependent on the scene (if most of the scene is empty, speed will be faster).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/11364490/176800109-38eb35f3-e145-4a09-8304-1795e3a4e8cd.png\", width=\"45%\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/11364490/176800106-fead794f-7e70-4459-b99e-82725fe6777e.png\", width=\"45%\"\u003e\n  \u003cbr\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/11364490/180444355-444676cf-2af2-49ad-9fe2-16eb1e6c4ef1.png\", width=\"45%\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/11364490/180444337-3df9f245-f7eb-453f-902b-0cb9dae60144.png\", width=\"45%\"\u003e\n  \u003cbr\u003e\n  \u003csup\u003eLeft: torch-ngp. Right: mine.\u003c/sup\u003e\n\u003c/p\u003e\n\n# :chart: Benchmarks\n\nTo run benchmarks, use the scripts under `benchmarking`.\n\nFollowings are my results trained using 1 RTX 2080 Ti (qualitative results [here](https://github.com/kwea123/ngp_pl/issues/7)):\n\n\u003cdetails\u003e\n  \u003csummary\u003eSynthetic-NeRF\u003c/summary\u003e\n\n|       | Mic   | Ficus | Chair | Hotdog | Materials | Drums | Ship  | Lego  | AVG   |\n| :---: | :---: | :---: | :---: | :---:  | :---:     | :---: | :---: | :---: | :---: |\n| PSNR  | 35.59 | 34.13 | 35.28 | 37.35  | 29.46     | 25.81 | 30.32 | 35.76 | 32.96 |\n| SSIM  | 0.988 | 0.982 | 0.984 | 0.980  | 0.944     | 0.933 | 0.890 | 0.979 | 0.960 |\n| LPIPS | 0.017 | 0.024 | 0.025 | 0.038  | 0.070     | 0.076 | 0.133 | 0.022 | 0.051 |\n| FPS   | 40.81 | 34.02 | 49.80 | 25.06  | 20.08     | 37.77 | 15.77 | 36.20 | 32.44 |\n| Training time | 3m9s | 3m12s | 4m17s | 5m53s | 4m55s | 4m7s | 9m20s | 5m5s | 5m00s |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eSynthetic-NSVF\u003c/summary\u003e\n\n|       | Wineholder | Steamtrain | Toad | Robot | Bike | Palace | Spaceship | Lifestyle | AVG | \n| :---: | :---: | :---: | :---: | :---: | :---:  | :---:  | :---: | :---: | :---: |\n| PSNR  | 31.64 | 36.47 | 35.57 | 37.10 | 37.87 | 37.41 | 35.58 | 34.76 | 35.80 |\n| SSIM  | 0.962 | 0.987 | 0.980 | 0.994 | 0.990 | 0.977 | 0.980 | 0.967 | 0.980 |\n| LPIPS | 0.047 | 0.023 | 0.024 | 0.010 | 0.015 | 0.021 | 0.029 | 0.044 | 0.027 |\n| FPS   | 47.07 | 75.17 | 50.42 | 64.87 | 66.88 | 28.62 | 35.55 | 22.84 | 48.93 |\n| Training time | 3m58s | 3m44s | 7m22s | 3m25s | 3m11s | 6m45s | 3m25s | 4m56s | 4m36s |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eTanks and Temples\u003c/summary\u003e\n\n|      | Ignatius | Truck | Barn  | Caterpillar | Family | AVG   | \n|:---: | :---:    | :---: | :---: | :---:       | :---:  | :---: |\n| PSNR | 28.30    | 27.67 | 28.00 | 26.16       | 34.27  | 28.78 |\n| *FPS | 10.04    |  7.99 | 16.14 | 10.91       | 6.16   | 10.25 |\n\n*Evaluated on `test-traj`\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eBlendedMVS\u003c/summary\u003e\n\n|       | *Jade  | *Fountain | Character | Statues | AVG   | \n|:---:  | :---:  | :---:     | :---:     | :---:   | :---: |\n| PSNR  | 25.43  | 26.82     | 30.43     | 26.79   | 27.38 |\n| **FPS | 26.02  | 21.24     | 35.99     | 19.22   | 25.61 |\n| Training time | 6m31s | 7m15s | 4m50s | 5m57s | 6m48s |\n\n*I manually switch the background from black to white, so the number isn't directly comparable to that in the papers.\n\n**Evaluated on `test-traj`\n\n\u003c/details\u003e\n\n# TODO\n\n- [ ] use super resolution in GUI to improve FPS\n- [ ] multi-sphere images as background\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkwea123%2Fngp_pl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkwea123%2Fngp_pl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkwea123%2Fngp_pl/lists"}