{"id":22563726,"url":"https://github.com/j3soon/omni-nerf-extension","last_synced_at":"2025-04-10T11:37:57.981Z","repository":{"id":235362862,"uuid":"711417906","full_name":"j3soon/omni-nerf-extension","owner":"j3soon","description":"Neural Radiance Field (NeRF) extension for Omniverse","archived":false,"fork":false,"pushed_at":"2025-02-28T09:39:42.000Z","size":1915,"stargazers_count":31,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-24T10:21:19.270Z","etag":null,"topics":["3d","3d-reconstruction","computer-graphics","isaac-sim","nerf","nerfstudio","neural-radiance-fields","omniverse","omniverse-kit-extension","robotics"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/j3soon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-29T07:47:45.000Z","updated_at":"2025-03-20T08:00:12.000Z","dependencies_parsed_at":"2024-04-24T12:30:30.375Z","dependency_job_id":"5e19c3ce-1cac-4bc2-97fb-01dfc6b0a950","html_url":"https://github.com/j3soon/omni-nerf-extension","commit_stats":null,"previous_names":["j3soon/omni-nerf-extension"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3soon%2Fomni-nerf-extension","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3soon%2Fomni-nerf-extension/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3soon%2Fomni-nerf-extension/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3soon%2Fomni-nerf-extension/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/j3soon","download_url":"https://codeload.github.com/j3soon/omni-nerf-extension/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248209037,"owners_count":21065271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","3d-reconstruction","computer-graphics","isaac-sim","nerf","nerfstudio","neural-radiance-fields","omniverse","omniverse-kit-extension","robotics"],"created_at":"2024-12-07T23:12:34.822Z","updated_at":"2025-04-10T11:37:57.969Z","avatar_url":"https://github.com/j3soon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Omniverse NeRF Extension\n\nNeural Radiance Field (NeRF) extension for Omniverse.\n\n\u003e For 3D Gaussian Splatting (3DGS) support, please refer to the [Omniverse 3DGS Extension](https://github.com/j3soon/omni-3dgs-extension).\n\n## Prerequisites\n\n- **Hardware**:\n  - CPU: x86\n  - GPU: NVIDIA RTX GPU\n  - See [this page](https://docs.omniverse.nvidia.com/isaacsim/latest/installation/requirements.html#system-requirements) for more details.\n- **Operating System**: Ubuntu 20.04/22.04.\n- **Software**:\n  - [NVIDIA Driver](https://ubuntu.com/server/docs/nvidia-drivers-installation)\n  - [Docker](https://docs.docker.com/engine/install/ubuntu/)\n  - [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)\n  - Omniverse Isaac Sim (through NGC Container)\n\n## Demo\n\nhttps://github.com/j3soon/omni-nerf-extension/assets/20457146/4b022624-a172-40b5-a5ad-c0ae678412f8\n\n\u003e At the exterior of the National Tsing Hua University (NTHU) swimming pool. Collected by [@muyishen2040](https://github.com/muyishen2040), which is also used in the [DriveEnv-NeRF project](https://github.com/muyishen2040/DriveEnvNeRF).\n\nhttps://github.com/j3soon/omni-nerf-extension/assets/20457146/40835ce5-5834-4631-aa17-1b591e402f27\n\n\u003e Exterior of the Delta Building in National Tsing Hua University (NTHU). Collected by [@Howardkhh](https://github.com/Howardkhh).\n\nhttps://github.com/j3soon/omni-nerf-extension/assets/20457146/049ff215-7a68-477b-9bc3-57b4cb3a889e\n\n\u003e The UR5 Robotic Arm in [ElsaLab](https://github.com/elsa-lab).\n\nhttps://github.com/j3soon/omni-nerf-extension/assets/20457146/3c9d657c-772d-4a83-9ac0-7af8c301d24f\n\n\u003e The Husky UGV in [ElsaLab](https://github.com/elsa-lab).\n\n**Note**: The datasets for these demos are not provided in this repository as they are casually collected. The purpose of these demos is to illustrate that this repository can be readily applied to a variety of custom datasets with extremely low effort. The following guide will use the `Poster` dataset for simplicity.\n\n## Setup\n\n```sh\ngit clone https://github.com/j3soon/omni-nerf-extension\ncd omni-nerf-extension\n```\n\nDownload assets:\n\n```sh\nwget https://github.com/j3soon/omni-nerf-extension/releases/download/v0.0.3/assets.zip\nunzip assets.zip\n```\n\nPrepare assets for `nerfstudio_renderer`:\n\n```sh\n# change the DATE_TIME to the name of the placeholder\nDATE_TIME=2023-12-30_111633\nCHECKPOINT_NAME=step-000029999\ncp -r ./assets/outputs/poster/nerfacto/$DATE_TIME ./assets/outputs/poster/nerfacto/DATE_TIME\nmv ./assets/outputs/poster/nerfacto/DATE_TIME/nerfstudio_models/$CHECKPOINT_NAME.ckpt ./assets/outputs/poster/nerfacto/DATE_TIME/nerfstudio_models/CHECKPOINT_NAME.ckpt\n```\n\nThe following assumes that you are running the commands from the root of the repository.\n\n## Managing Containers\n\nBuild the docker images for the extension:\n\n```sh\ndocker compose build\n```\n\nLaunch the containers:\n\n```sh\n# You might want to use `tmux` for exec-ing into the containers later\nxhost +local:docker\ndocker compose up\n```\n\nThen follow the remaining sections.\n\nTo remove and stop the containers, run:\n\n```sh\ndocker compose down\n```\n\n### Nerfstudio Renderer\n\nCode: [`nerfstudio_renderer`](./nerfstudio_renderer)\n\nThe renderer server would be listening on port `10001` upon successful startup:\n\n```\nINFO SLAVE/10001[MainThread]: server started on [0.0.0.0]:10001\n```\n\nAfter seeing the above logs, no additional steps are required for the renderer server.\n\n### PyGame Viewer\n\nCode: [`pygame_viewer`](./pygame_viewer)\n\nAttach to the container and run the testing script:\n\n```sh\ndocker exec -it pygame-viewer /src/run.sh\n```\n\nThe script may fail at the first run due to the cold start of the renderer server. If it fails, try run the script again.\n\nhttps://github.com/j3soon/omni-nerf-extension/assets/20457146/c971d623-30d1-4cfc-ba47-666dd99e021d\n\n### Isaac Sim Viewer\n\nCode: [`extension`](./extension)\n\n```sh\ndocker exec -it isaac-sim-viewer bash\n# in container\n/isaac-sim/runapp.sh --ext-folder /src/exts --enable omni.nerf.viewport\n```\n\n\u003e Alternatively, you can use WebRTC by running:\n\u003e \n\u003e ```sh\n\u003e # in container\n\u003e /isaac-sim/runheadless.webrtc.sh --ext-folder /src/exts --enable omni.nerf.viewport\n\u003e ```\n\u003e \n\u003e Wait for the `Isaac Sim Headless WebRTC App is loaded.` message,\n\u003e and then visit \u003chttp://127.0.0.1:8211/streaming/webrtc-demo/?server=127.0.0.1\u003e in Google Chrome.\n\n![](docs/media/isaac-sim-steps.png)\n\n1. Select the folder `/workspace/usd`\n2. Open the file `example_scene.usd`\n3. Click the mesh that you added in Step 2.\n4. Press the button in NeRF Viewport to update the input mesh of NeRF.\n\nhttps://github.com/j3soon/omni-nerf-extension/assets/20457146/5203061a-3b23-4d72-8103-5e3a6e9923a7\n\nhttps://github.com/user-attachments/assets/a1640eba-2cd0-497e-b47e-529190154677\n\n**Known Issues**:\n- Cannot correctly handling non-uniform scaling of the object mesh yet.\n\n## Development Notes\n\n### Nerfstudio Renderer\n\nAfter modifying code, you need to remove and recreate the container to apply changes. This is because the container will copy and install the code upon startup.\n\n### PyGame Viewer\n\nAfter modifying code, you need to re-run the testing script. The docker container can be re-used since the code is mounted as a volume.\n\n### Isaac Sim Viewer\n\nSetup VSCode intellisense for the extension:\n\n```sh\ncd extension\n./link_app.sh --path \"$HOME/.local/share/ov/pkg/code-2022.3.3\"\n# open the `extension` folder in VSCode\n```\n\nAfter modifying code, you can restart Isaac Sim to apply changes. The docker container can be re-used since the code is mounted as a volume. If the change is small, it is often faster to disable and re-enable the extension in the Isaac Sim UI. This can be done through `Window \u003e Extensions \u003e NVIDIA \u003e General`, search `nerf`, and then un-toggle and re-toggle the extension.\n\n## Future Directions\n\n- Include 3D Gaussian Splatting (3DGS) backend for real-time rendering, which is supported by a newer version of Nerfstudio, see [Splatfacto](https://docs.nerf.studio/nerfology/methods/splat.html) for more details.\n- Include [SDFStudio Neuralangelo](https://github.com/autonomousvision/sdfstudio?tab=readme-ov-file#updates) for improved mesh extraction. However, I've heard that SDFStudio does not perform as well as the original [Neuralangelo](https://research.nvidia.com/labs/dir/neuralangelo/) implementation.\n- Support multiple Nerf renderings in a single scene potentially through [Compositioning](https://docs.nerf.studio/extensions/blender_addon.html#compositing-nerf-objects-in-nerf-environments) or [Block-NeRF](https://waymo.com/intl/zh-tw/research/block-nerf/).\n\n## Related Works\n\nThis project focuses on the simplest integration of a NeRF renderer with Omniverse, which diverges from more advanced usages, such as those proposed in [a talk from GTC Spring 2023](https://www.nvidia.com/en-us/on-demand/session/gtcspring23-s52163/). The formal way to integrate NeRF with Omniverse is to first prototype with [custom](https://learnusd.github.io/chapter4/primspecs_and_attributes.html) [USD attributes](https://docs.omniverse.nvidia.com/dev-guide/latest/programmer_ref/usd/properties/create-attribute.html). The NeRF rendering can then be done by reading the custom USD attributes, using special [MDL](https://developer.nvidia.com/rendering-technologies/mdl-sdk) materials, or other clever techniques. Although this allows more advanced usages, this approach is much more difficult to implement. Moreover, the rasterization techniques used in 3DGS is fundamentally different from the volume-rendering techniques used in NeRF, which may require using two different rendering engines in the same scene.\n\n## Acknowledgement\n\nThis project has been made possible through the support of [ElsaLab][elsalab] and [NVIDIA AI Technology Center (NVAITC)][nvaitc].\n\nSpecial thanks to [@tigerpaws01](https://github.com/tigerpaws01) for the initial implementation of the NeRF renderer backend and PyGame viewer. This project wouldn't have come to light if it weren't for his early contributions. Fun fact: this project was initiated during one of our dinner conversation.\n\nI would also like to thank the NeRF Study Group members, [@muyishen2040](https://github.com/muyishen2040), [@AndreaHsu](https://github.com/AndreaHsu), [@Howardkhh](https://github.com/Howardkhh), and [VickyHuang1113](https://github.com/VickyHuang1113). Numerous insights and experiences have been gained through the collaboration on the [DriveEnv-NeRF project](https://github.com/muyishen2040/DriveEnvNeRF), which has significantly accelerated the development process of this project.\n\nFor a complete list of contributors to the code of this repository, please visit the [contributor list](https://github.com/j3soon/omni-nerf-extension/graphs/contributors).\n\n[![](docs/media/logos/elsalab.png)][elsalab]\n[![](docs/media/logos/nvaitc.png)][nvaitc]\n\n[elsalab]: https://github.com/elsa-lab\n[nvaitc]: https://github.com/NVAITC\n\nDisclaimer: this is not an official NVIDIA product.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj3soon%2Fomni-nerf-extension","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fj3soon%2Fomni-nerf-extension","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj3soon%2Fomni-nerf-extension/lists"}