https://github.com/nvidia-isaac-ros/isaac_ros_image_pipeline
NVIDIA-accelerated ROS 2 packages for camera image processing.
https://github.com/nvidia-isaac-ros/isaac_ros_image_pipeline
cv gpu image-processing jetson nvidia ros ros2 ros2-humble stereo-vision
Last synced: 3 months ago
JSON representation
NVIDIA-accelerated ROS 2 packages for camera image processing.
- Host: GitHub
- URL: https://github.com/nvidia-isaac-ros/isaac_ros_image_pipeline
- Owner: NVIDIA-ISAAC-ROS
- License: apache-2.0
- Created: 2021-10-19T00:14:25.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2025-02-28T01:47:03.000Z (7 months ago)
- Last Synced: 2025-06-05T07:07:18.853Z (4 months ago)
- Topics: cv, gpu, image-processing, jetson, nvidia, ros, ros2, ros2-humble, stereo-vision
- Language: C++
- Homepage: https://developer.nvidia.com/isaac-ros-gems
- Size: 1.49 MB
- Stars: 148
- Watchers: 4
- Forks: 28
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# Isaac ROS Image Pipeline
NVIDIA-accelerated Image Pipeline.
![]()
![]()
## Overview
Isaac ROS Image Pipeline is a metapackage of functionality for image
processing. Camera output often needs pre-processing to meet the input
requirements of multiple different perception functions. This can
include cropping, resizing, mirroring, correcting for lens distortion,
and color space conversion. For stereo cameras, additional processing is
required to produce disparity between left + right images and a point
cloud for depth perception.This package is accelerated using the GPU and specialized hardware
engines for image computation, replacing the CPU-based
[image_pipeline metapackage](http://wiki.ros.org/image_pipeline).
Considerable effort has been made to ensure that replacing
`image_pipeline` with `isaac_ros_image_pipeline` on a Jetson or GPU
is as painless a transition as possible.> [!Note]
> Some image pre-processing functions use specialized
> hardware engines, which offload the GPU to make more compute
> available for other tasks.Rectify corrects for lens distortion from the received camera sensor
message. The rectified image is resized to the input resolution for
disparity, using a crop before resizing to maintain image aspect ratio.
The image is color space converted to YUV from RGB using the luma
channel (the Y in YUV) to compute disparity using
[SGM](https://en.wikipedia.org/wiki/Semi-global_matching). This
common graph of nodes can be performed without the CPU processing a
single pixel using `isaac_ros_image_pipeline`; in comparison, using
`image_pipeline`, the CPU would process each pixel ~3 times.The Isaac ROS Image Pipeline metapackage offloads the CPU from common
image processing tasks so it can perform robotics functions best suited
for the CPU.## Isaac ROS NITROS Acceleration
This package is powered by [NVIDIA Isaac Transport for ROS (NITROS)](https://developer.nvidia.com/blog/improve-perception-performance-for-ros-2-applications-with-nvidia-isaac-transport-for-ros/), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.
## Performance
| Sample Graph
| Input Size
| AGX Orin
| Orin NX
| Orin Nano Super 8GB
| x86_64 w/ RTX 4090
|
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Rectify Node](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_image_proc_benchmark/scripts/isaac_ros_rectify_node.py)
| 1080p
| [800 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_rectify_node-agx_orin.json)
2.8 ms @ 30Hz
| [572 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_rectify_node-orin_nx.json)
3.3 ms @ 30Hz
| [595 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_rectify_node-orin_nano.json)
3.8 ms @ 30Hz
| [2500 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_rectify_node-x86-4090.json)
0.57 ms @ 30Hz
|
| [Stereo Disparity Node](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_stereo_image_proc_benchmark/scripts/isaac_ros_disparity_node.py)
| 1080p
| [118 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_disparity_node-agx_orin.json)
11 ms @ 30Hz
| [78.1 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_disparity_node-orin_nx.json)
14 ms @ 30Hz
| [83.8 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_disparity_node-orin_nano.json)
13 ms @ 30Hz
| [943 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_disparity_node-x86-4090.json)
1.6 ms @ 30Hz
|
| [Stereo Disparity Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_stereo_image_proc_benchmark/scripts/isaac_ros_disparity_graph.py)
| 1080p
| [111 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_disparity_graph-agx_orin.json)
15 ms @ 30Hz
| [72.2 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_disparity_graph-orin_nx.json)
18 ms @ 30Hz
| [77.4 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_disparity_graph-orin_nano.json)
18 ms @ 30Hz
| [692 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_disparity_graph-x86-4090.json)
4.6 ms @ 30Hz
|---
## Documentation
Please visit the [Isaac ROS Documentation](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/index.html) to learn how to use this repository.
---
## Packages
* [`isaac_ros_depth_image_proc`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/isaac_ros_depth_image_proc/index.html)
* [API](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/isaac_ros_depth_image_proc/index.html#api)
* [`isaac_ros_image_pipeline`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/isaac_ros_image_pipeline/index.html)
* [Replacing `image_pipeline` with `isaac_ros_image_pipeline`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/isaac_ros_image_pipeline/index.html#replacing-image-pipeline-with-isaac-ros-image-pipeline)
* [`isaac_ros_image_proc`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/isaac_ros_image_proc/index.html)
* [Quickstart](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/isaac_ros_image_proc/index.html#quickstart)
* [`isaac_ros_stereo_image_proc`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/isaac_ros_stereo_image_proc/index.html)
* [Quickstart](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/isaac_ros_stereo_image_proc/index.html#quickstart)
* [Try More Examples](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/isaac_ros_stereo_image_proc/index.html#try-more-examples)
* [API](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/isaac_ros_stereo_image_proc/index.html#api)## Latest
Update 2024-12-10: Added new alpha blending node