{"id":17938864,"url":"https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_stereo_depth","last_synced_at":"2025-03-24T10:31:43.631Z","repository":{"id":45029915,"uuid":"509288029","full_name":"NVIDIA-ISAAC-ROS/isaac_ros_dnn_stereo_depth","owner":"NVIDIA-ISAAC-ROS","description":"NVIDIA-accelerated, deep learned stereo disparity estimation","archived":false,"fork":false,"pushed_at":"2025-02-28T01:47:08.000Z","size":386,"stargazers_count":99,"open_issues_count":8,"forks_count":9,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-28T08:14:55.040Z","etag":null,"topics":["deep-learning","gpu","jetson","nvidia","ros2","ros2-humble","stereo-camera"],"latest_commit_sha":null,"homepage":"https://developer.nvidia.com/isaac-ros-gems","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVIDIA-ISAAC-ROS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-01T01:51:50.000Z","updated_at":"2025-02-28T00:39:19.000Z","dependencies_parsed_at":"2024-06-14T23:27:12.971Z","dependency_job_id":"f3228fa6-25ea-426e-85df-9204b469d52c","html_url":"https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_stereo_depth","commit_stats":null,"previous_names":["nvidia-isaac-ros/isaac_ros_dnn_stereo_depth"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-ISAAC-ROS%2Fisaac_ros_dnn_stereo_depth","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-ISAAC-ROS%2Fisaac_ros_dnn_stereo_depth/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-ISAAC-ROS%2Fisaac_ros_dnn_stereo_depth/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-ISAAC-ROS%2Fisaac_ros_dnn_stereo_depth/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVIDIA-ISAAC-ROS","download_url":"https://codeload.github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_stereo_depth/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245252300,"owners_count":20585007,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","gpu","jetson","nvidia","ros2","ros2-humble","stereo-camera"],"created_at":"2024-10-29T00:06:24.129Z","updated_at":"2025-03-24T10:31:43.615Z","avatar_url":"https://github.com/NVIDIA-ISAAC-ROS.png","language":"C++","funding_links":[],"categories":["Research-Grade Frameworks"],"sub_categories":[],"readme":"# Isaac ROS DNN Stereo Depth\n\nNVIDIA-accelerated, deep learned stereo disparity estimation\n\n\u003cdiv align=\"center\"\u003e\u003ca class=\"reference internal image-reference\" href=\"https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_dnn_stereo_depth/ess3.0_conf0_r2b_storage_576p.gif/\"\u003e\u003cimg alt=\"image\" src=\"https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_dnn_stereo_depth/ess3.0_conf0_r2b_storage_576p.gif/\" width=\"800px\"/\u003e\u003c/a\u003e\u003c/div\u003e\n\n---\n\n## Webinar Available\n\nLearn how to use this package by watching our on-demand webinar:\n[Using ML Models in ROS 2 to Robustly Estimate Distance to Obstacles](https://gateway.on24.com/wcc/experience/elitenvidiabrill/1407606/3998202/isaac-ros-webinar-series)\n\n---\n\n## Overview\n\nThe vision depth perception problem is generally useful in many fields of robotics such as estimating\nthe pose of a robotic arm in an object manipulation task, estimating distance of static or moving targets\nin autonomous robot navigation, tracking targets in delivery robots and so on.\nIsaac ROS DNN Stereo Depth is targeted at two Isaac applications,\nIsaac Manipulator and Isaac Perceptor. In Isaac Manipulator application, ESS is deployed in\n[Isaac ROS cuMotion](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_cumotion/index.html)\npackage as a plug-in node to provide depth perception maps for robot arm motion planning and control.\nIn this scenario, multi-camera stereo streams of industrial robot arms on a table task are passed to ESS to\nobtain corresponding depth streams. The depth streams are used to segment the relative distance of robot arms from\ncorresponding objects on the table; thus providing signals for collision avoidance and fine-grain control.\nSimilarly, the Isaac Perceptor application uses several Isaac ROS packages, namely,\n[Isaac ROS Nova](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_nova/index.html),\n[Isaac ROS Visual Slam](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_visual_slam/index.html),\n[Isaac ROS Stereo Depth (ESS)](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_dnn_stereo_depth/index.html),\n[Isaac ROS Nvblox](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_nvblox/index.html)\nand [Isaac ROS Image Pipeline](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/index.html).\n\nESS is deployed in Isaac Perceptor to enable Nvblox to create\n3D voxelized images of the robot surroundings. Specifically, the Nova developer suite provides 3x stereo-camera\nstreams to Isaac Perceptor. Each stream corresponds to the front, left, and right cameras.\nIn both Isaac Manipulator and Isaac Perceptor, a camera-specific image processing pipeline consisting of\nGPU-accelerated operations, provides rectification and undistortion of the input stereo images. All stereo stream image\npair are time synchronized before before passing them to ESS. ESS node outputs corresponding depth maps for all three\npreprocessed image streams and combines the depth images with motion signals provided by cuVSLAM module.\nThe combined depth and motion integrated signals are fed to Nvblox module to produce a dense 3D volumetric scene\nreconstruction of the surrounding scene.\n\n\u003cdiv align=\"center\"\u003e\u003ca class=\"reference internal image-reference\" href=\"https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_dnn_stereo_depth/isaac_ros_ess_nodegraph.png/\"\u003e\u003cimg alt=\"image\" src=\"https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_dnn_stereo_depth/isaac_ros_ess_nodegraph.png/\" width=\"800px\"/\u003e\u003c/a\u003e\u003c/div\u003e\n\nAbove, ESS node is used in a graph of nodes to provide a disparity prediction from an input left and right stereo image pair.\nThe rectify and resize nodes pre-process the left and right frames to the appropriate resolution.\nThe aspect ratio of the image is recommended to be maintained to avoid degrading the depth output quality.\nThe graph for DNN encode, DNN inference, and DNN decode is included in the ESS node. Inference is performed using\nTensorRT, as the ESS DNN model is designed with optimizations supported by TensorRT.\n\n## Isaac ROS NITROS Acceleration\n\nThis package is powered by [NVIDIA Isaac Transport for ROS (NITROS)](https://developer.nvidia.com/blog/improve-perception-performance-for-ros-2-applications-with-nvidia-isaac-transport-for-ros/), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.\n\n## Performance\n\n| Sample Graph\u003cbr/\u003e\u003cbr/\u003e                                                                                                                                                                                | Input Size\u003cbr/\u003e\u003cbr/\u003e     | AGX Orin\u003cbr/\u003e\u003cbr/\u003e                                                                                                                                                | Orin NX\u003cbr/\u003e\u003cbr/\u003e                                                                                                                                                | x86_64 w/ RTX 4090\u003cbr/\u003e\u003cbr/\u003e                                                                                                                                       |\n|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| [DNN Stereo Disparity Node](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_ess_benchmark/scripts/isaac_ros_ess_node.py)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003eFull\u003cbr/\u003e\u003cbr/\u003e          | 576p\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e | [72.5 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_ess_node-agx_orin.json)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e17 ms @ 30Hz\u003cbr/\u003e\u003cbr/\u003e       | [42.1 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_ess_node-orin_nx.json)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e26 ms @ 30Hz\u003cbr/\u003e\u003cbr/\u003e       | [350 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_ess_node-x86-4090.json)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e2.1 ms @ 30Hz\u003cbr/\u003e\u003cbr/\u003e        |\n| [DNN Stereo Disparity Node](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_ess_benchmark/scripts/isaac_ros_light_ess_node.py)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003eLight\u003cbr/\u003e\u003cbr/\u003e   | 288p\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e | [304 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_light_ess_node-agx_orin.json)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e5.9 ms @ 30Hz\u003cbr/\u003e\u003cbr/\u003e | [143 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_light_ess_node-orin_nx.json)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e9.6 ms @ 30Hz\u003cbr/\u003e\u003cbr/\u003e | [350 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_light_ess_node-x86-4090.json)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e1.6 ms @ 30Hz\u003cbr/\u003e\u003cbr/\u003e  |\n| [DNN Stereo Disparity Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_ess_benchmark/scripts/isaac_ros_ess_graph.py)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003eFull\u003cbr/\u003e\u003cbr/\u003e        | 576p\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e | [89.8 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_ess_graph-agx_orin.json)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e19 ms @ 30Hz\u003cbr/\u003e\u003cbr/\u003e      | [35.2 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_ess_graph-orin_nx.json)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e34 ms @ 30Hz\u003cbr/\u003e\u003cbr/\u003e      | [350 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_ess_graph-x86-4090.json)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e5.8 ms @ 30Hz\u003cbr/\u003e\u003cbr/\u003e       |\n| [DNN Stereo Disparity Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_ess_benchmark/scripts/isaac_ros_light_ess_graph.py)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003eLight\u003cbr/\u003e\u003cbr/\u003e | 288p\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e | [184 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_light_ess_graph-agx_orin.json)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e14 ms @ 30Hz\u003cbr/\u003e\u003cbr/\u003e | [128 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_light_ess_graph-orin_nx.json)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e14 ms @ 30Hz\u003cbr/\u003e\u003cbr/\u003e | [350 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_light_ess_graph-x86-4090.json)\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e5.2 ms @ 30Hz\u003cbr/\u003e\u003cbr/\u003e |\n\n---\n\n## Documentation\n\nPlease visit the [Isaac ROS Documentation](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_dnn_stereo_depth/index.html) to learn how to use this repository.\n\n---\n\n## Packages\n\n* [`isaac_ros_ess`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_dnn_stereo_depth/isaac_ros_ess/index.html)\n  * [Quickstart](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_dnn_stereo_depth/isaac_ros_ess/index.html#quickstart)\n  * [Try More Examples](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_dnn_stereo_depth/isaac_ros_ess/index.html#try-more-examples)\n  * [Troubleshooting](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_dnn_stereo_depth/isaac_ros_ess/index.html#troubleshooting)\n  * [API](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_dnn_stereo_depth/isaac_ros_ess/index.html#api)\n\n## Latest\n\nUpdate 2024-09-26: Updated for ESS 4.1 trained on additional samples\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNVIDIA-ISAAC-ROS%2Fisaac_ros_dnn_stereo_depth","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNVIDIA-ISAAC-ROS%2Fisaac_ros_dnn_stereo_depth","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNVIDIA-ISAAC-ROS%2Fisaac_ros_dnn_stereo_depth/lists"}