{"id":19132444,"url":"https://github.com/taichi-dev/taichi-nerfs","last_synced_at":"2025-05-16T19:03:59.721Z","repository":{"id":122034703,"uuid":"610072946","full_name":"taichi-dev/taichi-nerfs","owner":"taichi-dev","description":"Implementations of NeRF variants based on Taichi + PyTorch","archived":false,"fork":false,"pushed_at":"2024-05-27T04:22:04.000Z","size":157335,"stargazers_count":781,"open_issues_count":23,"forks_count":52,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-04-12T17:46:41.461Z","etag":null,"topics":["3d-reconstruction","computer-graphics","computer-vision","dreamfusion","machine-learning","nerf","neural-network","neural-radiance-field","real-time","real-time-rendering","taichi"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/taichi-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-06T02:58:21.000Z","updated_at":"2025-04-11T02:11:40.000Z","dependencies_parsed_at":null,"dependency_job_id":"c0a7ccb6-4bc4-4cd7-816e-4b876a093af1","html_url":"https://github.com/taichi-dev/taichi-nerfs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taichi-dev%2Ftaichi-nerfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taichi-dev%2Ftaichi-nerfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taichi-dev%2Ftaichi-nerfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taichi-dev%2Ftaichi-nerfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/taichi-dev","download_url":"https://codeload.github.com/taichi-dev/taichi-nerfs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254592367,"owners_count":22097010,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","computer-graphics","computer-vision","dreamfusion","machine-learning","nerf","neural-network","neural-radiance-field","real-time","real-time-rendering","taichi"],"created_at":"2024-11-09T06:18:18.935Z","updated_at":"2025-05-16T19:03:59.668Z","avatar_url":"https://github.com/taichi-dev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Taichi NeRFs\n\nA PyTorch + Taichi implementation of [instant-ngp](https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.pdf) NeRF training pipeline. For more details about modeling, please checkout this [article on our blog site](https://docs.taichi-lang.org/blog/taichi-instant-ngp).\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/office.gif\" width=\"200\"\u003e\n\u003c/p\u003e\n\n## Installation\n\n1. Install PyTorch by `python -m pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu116` (update the url with your installed CUDA Toolkit version number).\n2. Install taichi nightly via `pip install -U pip \u0026\u0026 pip install -i https://pypi.taichi.graphics/simple/ taichi-nightly`.\n3. Install requirements by `pip install -r requirements.txt`.\n4. If you plan to train with your own video, please install `colmap` via `sudo apt install colmap` or follow instructions at https://colmap.github.io/install.html.\n\n## Train with preprocessed datasets\n\n### Synthetic NeRF\n\nDownload [Synthetic NeRF dataset](https://dl.fbaipublicfiles.com/nsvf/dataset/Synthetic_NeRF.zip) and unzip it. Please keep the folder name unchanged.\n\n\nWe also provide a script to train the Lego scene from scratch, and display an interactive GUI at the end of the training.\n\n```bash\n./scripts/train_nsvf_lego.sh\n```\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/ngp_gui.gif\" width=\"400\"\u003e\n\u003c/p\u003e\n\nPerformance is measured on a Ubuntu 20.04 with an RTX3090 GPU.\n\n| Scene | avg PSNR | Training Time(20 epochs) |   GPU   |\n| :---: | :------: | :----------------------: | :-----: |\n| Lego  |   35.0   |           208s           | RTX3090 |\n\nTo reach the best performance, here are the steps to follow:\n1. Your work station is running on Linux and has RTX 3090 Graphics card\n2. Follow the steps in [Installation Section](https://github.com/taichi-dev/taichi-nerfs#installation)\n3. Uncomment `--half2_opt` to enable half2 optimization in the script, then `./scripts/train_nsvf_lego.sh`. For now, half2 optimization is only supported on Linux with Graphics Card Architecture \u003ePascal.\n\n\n### 360_v2 dataset\n\nDownload [360 v2 dataset](http://storage.googleapis.com/gresearch/refraw360/360_v2.zip) and unzip it. Please keep the folder name unchanged. The default `batch_size=8192` takes up to 18GB RAM on a RTX3090. Please adjust `batch_size` according to your hardware spec. \n\n```bash\n./scripts/train_360_v2_garden.sh\n```\n\n## Train with your own video\n\nPlace your video in `data` folder and pass the video path to the script. There are several key parameters for producing a sound dataset for NeRF training. For a real scene, `scale`  is recommended to set to 16. `video_fps` determines the number of images generated from the video, typically 150~200 images are sufficient. For a one minute video, 2 is a suitable number. Running this script will preprocess your video and start training a NeRF out of it:\n\n```\n./scripts/train_from_video.sh -v {your_video_name} -s {scale} -f {video_fps}\n```\n\n## Mobile Deployment\n\n[Taichi NGP Deployment](deployment/InstantNGP)\n\nUsing [Taichi AOT](https://docs.taichi-lang.org/docs/tutorial), you can easily deploy a NeRF rendering pipeline on any mobile devices!\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/NeRF_on_iPad.gif\" width=\"200\"\u003e \n\u003cimg src=\"assets/NeRF_iPhone14_Pro_Max.gif\" width=\"200\"\u003e\n\u003c/p\u003e\n\nWe're able to achieve real-time interactive on iOS devices.\n\n| Performance | iPad Pro (M1) | iPhone 14 Pro Max | iPhone 14 |\n| --- | --- | --- | --- |\n| Taichi Instant NGP | 22.4 fps | 18 fps | 13.5 fps |\n\nStay tuned, more cool demos are on the way! For business inquiries, please reach out us at `contact@taichi.graphics`.\n\n## Text to 3D \nTaichi-nerf serves as a new backend of the text-to-3D project [stable-dreamfusion](https://github.com/ashawkey/stable-dreamfusion).\n\n## Frequently asked questions (FAQ)\n\n__Q:__ Is CUDA the only supported Taichi backend? How about vulkan backend?\n\n__A:__ For the most efficient interop with PyTorch CUDA backend, training is mostly tested with Taichi CUDA backend. However it's pretty straightforward to switch to Taichi vulkan backend if interop is removed, check out this awesome [taichi-ngp inference demo](https://github.com/taichi-dev/taichi/blob/master/python/taichi/examples/rendering/taichi_ngp.py)!\n\n__Q:__ I got OOM(Out of Memory) error on my GPU, what can I do?\n\n__A:__ Reduce `batch_size` passed to `train.py`! By default it's `8192` which fits a RTX3090, you should reduce this accordingly. For instance, `batch_size=2048` is recommended on a RTX3060Ti.\n\n# Acknowledgement\n\nThe PyTorch interface of the training pipeline and colmap preprocessing are highly referred to:\n\n*  [ngp_pl](https://github.com/kwea123/ngp_pl)\n*  [instant-ngp CUDA implementation](https://github.com/NVlabs/instant-ngp/tree/master)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaichi-dev%2Ftaichi-nerfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftaichi-dev%2Ftaichi-nerfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaichi-dev%2Ftaichi-nerfs/lists"}