{"id":13583897,"url":"https://github.com/kwea123/nerf_pl","last_synced_at":"2025-05-15T09:07:14.413Z","repository":{"id":37675059,"uuid":"255096636","full_name":"kwea123/nerf_pl","owner":"kwea123","description":"NeRF (Neural Radiance Fields) and NeRF in the Wild using pytorch-lightning","archived":false,"fork":false,"pushed_at":"2023-08-03T15:02:39.000Z","size":23906,"stargazers_count":2784,"open_issues_count":80,"forks_count":480,"subscribers_count":51,"default_branch":"master","last_synced_at":"2025-04-14T15:01:14.552Z","etag":null,"topics":["3d-reconstruction","colab","mesh","mixed-reality","nerf","nerf-in-the-wild","nerf-w","neural-radiance-fields","pytorch","pytorch-lightning","ray-tracing","unity3d","view-synthesis","volume-rendering"],"latest_commit_sha":null,"homepage":"https://www.youtube.com/playlist?list=PLDV2CyUo4q-K02pNEyDr7DYpTQuka3mbV","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kwea123.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null},"funding":{"github":"kwea123","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2020-04-12T14:12:06.000Z","updated_at":"2025-04-10T09:01:29.000Z","dependencies_parsed_at":"2024-01-18T19:11:27.528Z","dependency_job_id":null,"html_url":"https://github.com/kwea123/nerf_pl","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwea123%2Fnerf_pl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwea123%2Fnerf_pl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwea123%2Fnerf_pl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwea123%2Fnerf_pl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kwea123","download_url":"https://codeload.github.com/kwea123/nerf_pl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254310515,"owners_count":22049469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","colab","mesh","mixed-reality","nerf","nerf-in-the-wild","nerf-w","neural-radiance-fields","pytorch","pytorch-lightning","ray-tracing","unity3d","view-synthesis","volume-rendering"],"created_at":"2024-08-01T15:03:52.867Z","updated_at":"2025-05-15T09:07:09.404Z","avatar_url":"https://github.com/kwea123.png","language":"Jupyter Notebook","readme":"# nerf_pl\n\n### Update: NVIDIA open-sourced a lightning-fast version of NeRF: [NGP](https://github.com/NVlabs/instant-ngp). I re-implemented in pytorch [here](https://github.com/kwea123/ngp_pl). This version is ~100x faster than this repo with also better quality!\n\n### Update: an improved [NSFF](https://www.cs.cornell.edu/~zl548/NSFF/) implementation to handle dynamic scene is [open](https://github.com/kwea123/nsff_pl)!\n\n### Update: [NeRF-W](https://nerf-w.github.io/) (NeRF in the Wild) implementation is added to [nerfw](https://github.com/kwea123/nerf_pl/tree/nerfw) branch!\n\n### Update: The lastest code (using the latest libraries) will be updated to [dev](https://github.com/kwea123/nerf_pl/tree/dev) branch. The master branch remains to support the colab files. If you don't use colab, it is recommended to switch to dev branch. Only issues of the dev and nerfw branch will be considered currently.\n\n### :gem: [**Project page**](https://kwea123.github.io/nerf_pl/) (live demo!)\n\nUnofficial implementation of [NeRF](https://arxiv.org/pdf/2003.08934.pdf) (Neural Radiance Fields) using pytorch ([pytorch-lightning](https://github.com/PyTorchLightning/pytorch-lightning)). This repo doesn't aim at reproducibility, but aim at providing a simpler and faster training procedure (also simpler code with detailed comments to help to understand the work). Moreover, I try to extend much more opportunities by integrating this algorithm into game engine like Unity.\n\nOfficial implementation: [nerf](https://github.com/bmild/nerf) .. Reference pytorch implementation: [nerf-pytorch](https://github.com/yenchenlin/nerf-pytorch)\n\n### Recommend to read: A detailed NeRF extension list: [awesome-NeRF](https://github.com/yenchenlin/awesome-NeRF)\n\n## :milky_way: Features\n\n* Multi-gpu training: Training on 8 GPUs finishes within 1 hour for the synthetic dataset!\n* [Colab](#mortar_board-colab) notebooks to allow easy usage!\n* [Reconstruct](#ribbon-mesh) **colored** mesh!\n* [Mixed Reality](https://youtu.be/S5phWFTs2iM) in Unity!\n* [REAL TIME volume rendering](https://youtu.be/w9qTbVzCdWk) in Unity!\n* [Portable Scenes](#portable-scenes) to let you play with other people's scenes!\n\n### You can find the Unity project including mesh, mixed reality and volume rendering [here](https://github.com/kwea123/nerf_Unity)! See [README_Unity](README_Unity.md) for generating your own data for Unity rendering!\n\n## :beginner: Tutorial\n\n### What can NeRF do?\n\u003cimg src=\"https://user-images.githubusercontent.com/11364490/82124460-1ccbbb80-97da-11ea-88ad-25e22868a5c1.png\" style=\"max-width:100%\"\u003e\n\n### Tutorial videos\n\u003ca href=\"https://www.youtube.com/playlist?list=PLDV2CyUo4q-K02pNEyDr7DYpTQuka3mbV\"\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/11364490/80913471-d5781080-8d7f-11ea-9f72-9d68402b8271.png\"\u003e\n\u003c/a\u003e\n   \n# :computer: Installation\n\n## Hardware\n\n* OS: Ubuntu 18.04\n* NVIDIA GPU with **CUDA\u003e=10.1** (tested with 1 RTX2080Ti)\n\n## Software\n\n* Clone this repo by `git clone --recursive https://github.com/kwea123/nerf_pl`\n* Python\u003e=3.6 (installation via [anaconda](https://www.anaconda.com/distribution/) is recommended, use `conda create -n nerf_pl python=3.6` to create a conda environment and activate it by `conda activate nerf_pl`)\n* Python libraries\n    * Install core requirements by `pip install -r requirements.txt`\n    * Install `torchsearchsorted` by `cd torchsearchsorted` then `pip install .`\n    \n# :key: Training\n\nPlease see each subsection for training on different datasets. Available training datasets:\n\n* [Blender](#blender) (Realistic Synthetic 360)\n* [LLFF](#llff) (Real Forward-Facing)\n* [Your own data](#your-own-data) (Forward-Facing/360 inward-facing)\n\n## Blender\n\u003cdetails\u003e\n  \u003csummary\u003eSteps\u003c/summary\u003e\n   \n### Data download\n\nDownload `nerf_synthetic.zip` from [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1)\n\n### Training model\n\nRun (example)\n```\npython train.py \\\n   --dataset_name blender \\\n   --root_dir $BLENDER_DIR \\\n   --N_importance 64 --img_wh 400 400 --noise_std 0 \\\n   --num_epochs 16 --batch_size 1024 \\\n   --optimizer adam --lr 5e-4 \\\n   --lr_scheduler steplr --decay_step 2 4 8 --decay_gamma 0.5 \\\n   --exp_name exp\n```\n\nThese parameters are chosen to best mimic the training settings in the original repo. See [opt.py](opt.py) for all configurations.\n\nNOTE: the above configuration doesn't work for some scenes like `drums`, `ship`. In that case, consider increasing the `batch_size` or change the `optimizer` to `radam`. I managed to train on all scenes with these modifications.\n\nYou can monitor the training process by `tensorboard --logdir logs/` and go to `localhost:6006` in your browser.\n\u003c/details\u003e\n\n## LLFF\n\u003cdetails\u003e\n  \u003csummary\u003eSteps\u003c/summary\u003e\n   \n### Data download\n\nDownload `nerf_llff_data.zip` from [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1)\n\n### Training model\n\nRun (example)\n```\npython train.py \\\n   --dataset_name llff \\\n   --root_dir $LLFF_DIR \\\n   --N_importance 64 --img_wh 504 378 \\\n   --num_epochs 30 --batch_size 1024 \\\n   --optimizer adam --lr 5e-4 \\\n   --lr_scheduler steplr --decay_step 10 20 --decay_gamma 0.5 \\\n   --exp_name exp\n```\n\nThese parameters are chosen to best mimic the training settings in the original repo. See [opt.py](opt.py) for all configurations.\n\nYou can monitor the training process by `tensorboard --logdir logs/` and go to `localhost:6006` in your browser.\n\u003c/details\u003e\n\n## Your own data\n\u003cdetails\u003e\n  \u003csummary\u003eSteps\u003c/summary\u003e\n   \n1. Install [COLMAP](https://github.com/colmap/colmap) following [installation guide](https://colmap.github.io/install.html)\n2. Prepare your images in a folder (around 20 to 30 for forward facing, and 40 to 50 for 360 inward-facing)\n3. Clone [LLFF](https://github.com/Fyusion/LLFF) and run `python img2poses.py $your-images-folder`\n4. Train the model using the same command as in [LLFF](#llff). If the scene is captured in a 360 inward-facing manner, add `--spheric` argument.\n\nFor more details of training a good model, please see the video [here](#colab).\n\u003c/details\u003e\n\n## Pretrained models and logs\nDownload the pretrained models and training logs in [release](https://github.com/kwea123/nerf_pl/releases).\n\n## Comparison with other repos\n\n|           | training GPU memory in GB | Speed (1 step) |\n| :---:     |  :---:     | :---:   | \n| [Original](https://github.com/bmild/nerf)  |  8.5 | 0.177s |\n| [Ref pytorch](https://github.com/yenchenlin/nerf-pytorch)  |  6.0 | 0.147s |\n| This repo | 3.2 | 0.12s |\n\nThe speed is measured on 1 RTX2080Ti. Detailed profile can be found in [release](https://github.com/kwea123/nerf_pl/releases).\nTraining memory is largely reduced, since the original repo loads the whole data to GPU at the beginning, while we only pass batches to GPU every step.\n\n# :mag_right: Testing\n\nSee [test.ipynb](test.ipynb) for a simple view synthesis and depth prediction on 1 image.\n\nUse [eval.py](eval.py) to create the whole sequence of moving views.\nE.g.\n```\npython eval.py \\\n   --root_dir $BLENDER \\\n   --dataset_name blender --scene_name lego \\\n   --img_wh 400 400 --N_importance 64 --ckpt_path $CKPT_PATH\n```\n**IMPORTANT** : Don't forget to add `--spheric_poses` if the model is trained under `--spheric` setting!\n\nIt will create folder `results/{dataset_name}/{scene_name}` and run inference on all test data, finally create a gif out of them.\n\nExample of lego scene using pretrained model and the reconstructed **colored** mesh: (PSNR=31.39, paper=32.54)\n\n\u003cp\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/11364490/79932648-f8a1e680-8488-11ea-98fe-c11ec22fc8a1.gif\" width=\"200\"\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/11364490/80813179-822d8300-8c04-11ea-84e6-142f04714c58.png\" width=\"200\"\u003e\n\u003c/p\u003e\n\nExample of fern scene using pretrained model:\n\n![fern](https://user-images.githubusercontent.com/11364490/79932650-f9d31380-8488-11ea-8dad-b70a6a3daa6e.gif)\n\nExample of own scene ([Silica GGO figure](https://www.youtube.com/watch?v=hVQIvEq_Av0)) and the reconstructed **colored** mesh. Click to link to youtube video.\n\n\u003cp\u003e\n\u003ca href=\"https://youtu.be/yH1ZBcdNsUY\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/11364490/80279695-324d4880-873a-11ea-961a-d6350e149ece.gif\" height=\"252\"\u003e\n\u003c/a\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/11364490/80813184-83f74680-8c04-11ea-8606-40580f753355.png\" height=\"252\"\u003e\n\u003c/p\u003e\n\n## Portable scenes\nThe concept of NeRF is that the whole scene is compressed into a NeRF model, then we can render from any pose we want. To render from plausible poses, we can leverage the training poses; therefore, you can generate video with **only** the trained model and the poses (hence the name of portable scenes). I provided my silica model in [release](https://github.com/kwea123/nerf_pl/releases), feel free to play around with it!\n\nIf you trained some interesting scenes, you are also welcomed to share the model (and the `poses_bounds.npy`) by sending me an email, or post in issues! After all, a model is just around **5MB**! Please run `python utils/save_weights_only.py --ckpt_path $YOUR_MODEL_PATH` to extract the final model.\n\n# :ribbon: Mesh\n\nSee [README_mesh](README_mesh.md) for reconstruction of **colored** mesh. Only supported for blender dataset and 360 inward-facing data!\n\n# :warning: Notes on differences with the original repo\n\n*  The learning rate decay in the original repo is **by step**, which means it decreases every step, here I use learning rate decay **by epoch**, which means it changes only at the end of 1 epoch.\n*  The validation image for LLFF dataset is chosen as the most centered image here, whereas the original repo chooses every 8th image.\n*  The rendering spiral path is slightly different from the original repo (I use approximate values to simplify the code).\n\n# :mortar_board: COLAB\n\nI also prepared colab notebooks that allow you to run the algorithm on any machine without GPU requirement.\n\n*  [colmap](https://gist.github.com/kwea123/f0e8f38ff2aa94495dbfe7ae9219f75c) to prepare camera poses for your own training data\n*  [nerf](https://gist.github.com/kwea123/a3c541a325e895ef79ecbc0d2e6d7221) to train on your data\n*  [extract_mesh](https://gist.github.com/kwea123/77ed1640f9bc9550136dc13a6a419e88) to extract colored mesh\n\nPlease see [this playlist](https://www.youtube.com/playlist?list=PLDV2CyUo4q-K02pNEyDr7DYpTQuka3mbV) for the detailed tutorials.\n\n# :jack_o_lantern: SHOWOFF\n\nWe can incorporate *ray tracing* techniques into the volume rendering pipeline, and realize realistic scene editing (following is the `materials` scene with an object removed, and a mesh is inserted and rendered with ray tracing). The code **will not** be released.\n\n![add](https://user-images.githubusercontent.com/11364490/90312710-92face00-df41-11ea-9eea-10f24849b407.gif)\n![add2](https://user-images.githubusercontent.com/11364490/90360796-92744b80-e097-11ea-859d-159aa2519375.gif)\n\nWith my integration in Unity, I can realize realistic mixed reality photos (note my character casts shadow on the scene, **zero** post- image editing required):\n![defer](https://user-images.githubusercontent.com/11364490/140264589-295acebe-8ace-4d61-b871-26eb8ae10ab0.png)\n![defer2](https://user-images.githubusercontent.com/11364490/140264596-59daebe5-b88d-48e7-82bd-5ccaaff2283f.png)\nBTW, I would like to visit the museum one day...\n\n# :book: Citation\nIf you use (part of) my code or find my work helpful, please consider citing\n```\n@misc{queianchen_nerf,\n  author={Quei-An, Chen},\n  title={Nerf_pl: a pytorch-lightning implementation of NeRF},\n  url={https://github.com/kwea123/nerf_pl/},\n  year={2020},\n}\n```\n","funding_links":["https://github.com/sponsors/kwea123"],"categories":["Jupyter Notebook","Implementations"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkwea123%2Fnerf_pl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkwea123%2Fnerf_pl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkwea123%2Fnerf_pl/lists"}