https://github.com/nvidia-isaac-ros/isaac_ros_image_segmentation
NVIDIA-accelerated, deep learned semantic image segmentation
https://github.com/nvidia-isaac-ros/isaac_ros_image_segmentation
ai dnn gpu image-segmentation inference jetson nvidia ros2 ros2-humble tensorrt
Last synced: 3 months ago
JSON representation
NVIDIA-accelerated, deep learned semantic image segmentation
- Host: GitHub
- URL: https://github.com/nvidia-isaac-ros/isaac_ros_image_segmentation
- Owner: NVIDIA-ISAAC-ROS
- License: apache-2.0
- Created: 2021-11-04T21:20:41.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2026-03-24T03:51:28.000Z (4 months ago)
- Last Synced: 2026-03-25T04:15:20.486Z (4 months ago)
- Topics: ai, dnn, gpu, image-segmentation, inference, jetson, nvidia, ros2, ros2-humble, tensorrt
- Language: C++
- Homepage:
- Size: 33.9 MB
- Stars: 117
- Watchers: 4
- Forks: 10
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Security: SECURITY.md
Awesome Lists containing this project
README
# Isaac ROS Image Segmentation
NVIDIA-accelerated, deep learned semantic image segmentation
## Overview
Isaac ROS Image Segmentation contains ROS packages for semantic image segmentation.
These packages provide methods for classification of an input image
at the pixel level by running GPU-accelerated inference on a DNN model.
Each pixel of the input image is predicted to belong to a set of defined classes.
The output prediction can be used by perception functions to understand where each
class is spatially in a 2D image or fuse with a corresponding depth location in a 3D scene.
| Package | Model Architecture | Description |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
| [Isaac ROS U-NET](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_unet/index.html#quickstart) | [U-NET](https://en.wikipedia.org/wiki/U-Net) | Convolutional network popular for biomedical imaging segmentation models |
| [Isaac ROS Segformer](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segformer/index.html#quickstart) | [Segformer](https://arxiv.org/abs/2105.15203) | Transformer-based network that works well for objects of varying scale |
| [Isaac ROS Segment Anything](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything/index.html#quickstart) | [Segment Anything](https://github.com/facebookresearch/segment-anything) | Segments any object in an image when given a prompt as to which one |
| [Isaac ROS Segment Anything2](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything2/index.html#quickstart) | [Segment Anything2](https://ai.meta.com/research/publications/sam-2-segment-anything-in-images-and-videos/) | Segments and tracks any object in a video stream when given a prompt as to which one |
Input images may need to be cropped and resized to maintain the aspect ratio and match the input
resolution expected by the DNN model; image resolution may be reduced to improve
DNN inference performance, which typically scales directly with the
number of pixels in the image.
Image segmentation provides more information and uses more compute than
object detection to produce classifications per pixel, whereas object
detection classifies a simpler bounding box rectangle in image
coordinates. Object detection is used to know if, and where spatially in
a 2D image, the object exists. On the other hand, image segmentation is used to know which
pixels belong to the class. One application is using the segmentation result, and fusing it with the corresponding depth
information in order to know an object location in a 3D scene.
## Isaac ROS NITROS Acceleration
This package is powered by [NVIDIA Isaac Transport for ROS (NITROS)](https://developer.nvidia.com/blog/improve-perception-performance-for-ros-2-applications-with-nvidia-isaac-transport-for-ros/), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.
## Performance
| Sample Graph
| Input Size
| AGX Thor T5000
| AGX Thor T4000
| DGX Spark
| x86_64 w/ RTX 5090
|
|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [SAM Image Segmentation Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/benchmarks/isaac_ros_segment_anything_benchmark/scripts/isaac_ros_segment_anything_graph.py)
Full SAM
| 720p
| [2.26 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/results/isaac_ros_sam_graph-agx_thor.json)
350 ms @ 30Hz
| [2.24 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/results/isaac_ros_sam_graph-thor-t4000.json)
290 ms @ 30Hz
| [2.22 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/results/isaac_ros_sam_graph-dgx_spark.json)
280 ms @ 30Hz
| [20.8 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/results/isaac_ros_sam_graph-x86-5090.json)
57 ms @ 30Hz
|
| [SAM Image Segmentation Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/benchmarks/isaac_ros_segment_anything_benchmark/scripts/isaac_ros_mobile_segment_anything_graph.py)
Mobile SAM
| 720p
| [15.0 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/results/isaac_ros_mobile_sam_graph-agx_thor.json)
230 ms @ 30Hz
| [15.0 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/results/isaac_ros_mobile_sam_graph-thor-t4000.json)
200 ms @ 30Hz
| [14.6 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/results/isaac_ros_mobile_sam_graph-dgx_spark.json)
82 ms @ 30Hz
| [70.3 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/results/isaac_ros_mobile_sam_graph-x86-5090.json)
20 ms @ 30Hz
|
| [TensorRT Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/benchmarks/isaac_ros_unet_benchmark/scripts/isaac_ros_unet_graph.py)
PeopleSemSegNet
| 544p
| [449 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/results/isaac_ros_unet_graph-agx_thor.json)
8.1 ms @ 30Hz
| [319 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/results/isaac_ros_unet_graph-thor-t4000.json)
19 ms @ 30Hz
| [562 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/results/isaac_ros_unet_graph-dgx_spark.json)
6.7 ms @ 30Hz
| [1330 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/release-4.4/results/isaac_ros_unet_graph-x86-5090.json)
6.3 ms @ 30Hz
|
---
## Documentation
Please visit the [Isaac ROS Documentation](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/index.html) to learn how to use this repository.
---
## Packages
* [`isaac_ros_segformer`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segformer/index.html)
* [Quickstart](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segformer/index.html#quickstart)
* [Try More Examples](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segformer/index.html#try-more-examples)
* [Troubleshooting](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segformer/index.html#troubleshooting)
* [API](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segformer/index.html#api)
* [`isaac_ros_segment_anything`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything/index.html)
* [Quickstart](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything/index.html#quickstart)
* [Try More Examples](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything/index.html#try-more-examples)
* [Troubleshooting](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything/index.html#troubleshooting)
* [API](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything/index.html#api)
* [`isaac_ros_segment_anything2`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything2/index.html)
* [Quickstart](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything2/index.html#quickstart)
* [Troubleshooting](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything2/index.html#troubleshooting)
* [API](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything2/index.html#api)
* [`isaac_ros_segment_anything2_interfaces`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything2_interfaces/index.html)
* [`isaac_ros_unet`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_unet/index.html)
* [Quickstart](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_unet/index.html#quickstart)
* [Try More Examples](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_unet/index.html#try-more-examples)
* [Troubleshooting](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_unet/index.html#troubleshooting)
* [API](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_unet/index.html#api)
## Latest
Update 2026-04-30: Compatibility and integration updates for the Isaac ROS 4.4.0 release