{"id":13717604,"url":"https://github.com/dusty-nv/ros_deep_learning","last_synced_at":"2025-05-16T06:05:39.537Z","repository":{"id":41579763,"uuid":"69927813","full_name":"dusty-nv/ros_deep_learning","owner":"dusty-nv","description":"Deep learning inference nodes for ROS / ROS2 with support for NVIDIA Jetson and TensorRT","archived":false,"fork":false,"pushed_at":"2024-07-13T03:08:46.000Z","size":101,"stargazers_count":923,"open_issues_count":91,"forks_count":260,"subscribers_count":35,"default_branch":"master","last_synced_at":"2025-04-08T15:14:57.850Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dusty-nv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-04T02:18:28.000Z","updated_at":"2025-03-26T05:50:42.000Z","dependencies_parsed_at":"2024-07-13T04:24:22.333Z","dependency_job_id":null,"html_url":"https://github.com/dusty-nv/ros_deep_learning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dusty-nv%2Fros_deep_learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dusty-nv%2Fros_deep_learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dusty-nv%2Fros_deep_learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dusty-nv%2Fros_deep_learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dusty-nv","download_url":"https://codeload.github.com/dusty-nv/ros_deep_learning/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254478188,"owners_count":22077676,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T00:01:24.655Z","updated_at":"2025-05-16T06:05:38.129Z","avatar_url":"https://github.com/dusty-nv.png","language":"C++","funding_links":[],"categories":["Sensor Processing","ROS2"],"sub_categories":["Image Processing","ROS2 Packages"],"readme":"# DNN Inference Nodes for ROS/ROS2\nThis package contains DNN inference nodes and camera/video streaming nodes for ROS/ROS2 with support for NVIDIA **[Jetson Nano / TX1 / TX2 / Xavier / Orin](https://developer.nvidia.com/embedded-computing)** devices and TensorRT.\n\nThe nodes use the image recognition, object detection, and semantic segmentation DNN's from the [`jetson-inference`](https://github.com/dusty-nv/jetson-inference) library and NVIDIA [Hello AI World](https://github.com/dusty-nv/jetson-inference#hello-ai-world) tutorial, which come with several built-in pretrained networks for classification, detection, and segmentation and the ability to load customized user-trained models.\n\nThe camera \u0026 video streaming nodes support the following [input/output interfaces](https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md):\n\n* MIPI CSI cameras\n* V4L2 cameras\n* RTP / RTSP streams\n* WebRTC streams\n* Videos \u0026 Images\n* Image sequences\n* OpenGL windows\n\nVarious distribution of ROS are supported either from source or through containers (including Melodic, Noetic, Foxy, Galactic, Humble, and Iron).  The same branch supports both ROS1 and ROS2.\n\n### Table of Contents\n\n* [Installation](#installation)\n* [Testing](#testing)\n\t* [Video Viewer](#video-viewer)\n\t* [imagenet Node](#imagenet-node)\n\t* [detectnet Node](#detectnet-node)\n\t* [segnet Node](#segnet-node)\n* [Topics \u0026 Parameters](#topics-messages)\n\t* [imagenet Node](#imagenet-node-1)\n\t* [detectnet Node](#detectnet-node-1)\n\t* [segnet Node](#segnet-node-1) \n\t* [video_source Node](#video_source-node)\n\t* [video_output Node](#video_output-node)\n\n## Installation\n\nThe easiest way to get up and running is by cloning [jetson-inference](https://github.com/dusty-nv/jetson-inference) (which ros_deep_learning is a submodule of) and running the pre-built container, which automatically mounts the required model directories and devices:\n\n``` bash\n$ git clone --recursive --depth=1 https://github.com/dusty-nv/jetson-inference\n$ cd jetson-inference\n$ docker/run.sh --ros=humble  # noetic, foxy, galactic, humble, iron\n```\n\n\u003e **note**: the ros_deep_learning nodes rely on data from the jetson-inference tree for storing models, so clone and mount `jetson-inference/data` if you're using your own container or source installation method.\n\nThe `--ros` argument to the [`docker/run.sh`](https://github.com/dusty-nv/jetson-inference/blob/master/docker/run.sh) script selects the ROS distro to use.  They in turn use the `ros:$ROS_DISTRO-pytorch` container images from [jetson-containers](https://github.com/dusty-nv/jetson-containers), which include jetson-inference and this.\n\nFor previous information about building the ros_deep_learning package for an uncontainerized ROS installation, expand the section below (the parts about installing ROS may require adapting for the particular version of ROS/ROS2 that you want to install)\n\n\u003cdetails\u003e\n\u003csummary\u003eLegacy Install Instructions\u003c/summary\u003e\n\n### jetson-inference\n\nThese ROS nodes use the DNN objects from the [`jetson-inference`](https://github.com/dusty-nv/jetson-inference) project (aka Hello AI World).  To build and install jetson-inference, see [this page](https://github.com/dusty-nv/jetson-inference/blob/master/docs/building-repo-2.md) or run the commands below:\n\n```bash\n$ cd ~\n$ sudo apt-get install git cmake\n$ git clone --recursive --depth=1 https://github.com/dusty-nv/jetson-inference\n$ cd jetson-inference\n$ mkdir build\n$ cd build\n$ cmake ../\n$ make -j$(nproc)\n$ sudo make install\n$ sudo ldconfig\n```\nBefore proceeding, it's worthwhile to test that `jetson-inference` is working properly on your system by following this step of the Hello AI World tutorial:\n* [Classifying Images with ImageNet](https://github.com/dusty-nv/jetson-inference/blob/master/docs/imagenet-console-2.md)\n\n### ROS/ROS2\n\nInstall the `ros-melodic-ros-base` or `ros-eloquent-ros-base` package on your Jetson following these directions:\n\n* ROS Melodic - [ROS Install Instructions](http://wiki.ros.org/melodic/Installation/Ubuntu)\n* ROS2 Eloquent - [ROS2 Install Instructions](https://index.ros.org/doc/ros2/Installation/Eloquent/Linux-Install-Debians/)\n\nDepending on which version of ROS you're using, install some additional dependencies and create a workspace:\n\n#### ROS Melodic\n```bash\n$ sudo apt-get install ros-melodic-image-transport ros-melodic-vision-msgs\n```\n\nFor ROS Melodic, create a Catkin workspace (`~/ros_workspace`) using these steps:  \nhttp://wiki.ros.org/ROS/Tutorials/InstallingandConfiguringROSEnvironment#Create_a_ROS_Workspace\n\n#### ROS Eloquent\n```bash\n$ sudo apt-get install ros-eloquent-vision-msgs \\\n                       ros-eloquent-launch-xml \\\n                       ros-eloquent-launch-yaml \\\n                       python3-colcon-common-extensions\n```\n\nFor ROS Eloquent, create a workspace (`~/ros_workspace`) to use:\n\n```bash\n$ mkdir -p ~/ros2_example_ws/src\n```\n\n### ros_deep_learning\n\nNext, navigate into your ROS workspace's `src` directory and clone `ros_deep_learning`:\n\n```bash\n$ cd ~/ros_workspace/src\n$ git clone https://github.com/dusty-nv/ros_deep_learning\n```\n\nThen build it - if you are using ROS Melodic, use `catkin_make`.  If you are using ROS2 Eloquent, use `colcon build`:\n\n```bash\n$ cd ~/ros_workspace/\n\n# ROS Melodic\n$ catkin_make\n$ source devel/setup.bash \n\n# ROS2 Eloquent\n$ colcon build\n$ source install/local_setup.bash \n```\n\nThe nodes should now be built and ready to use.  Remember to source the overlay as shown above so that ROS can find the nodes.\n\n\u003c/details\u003e\n\n## Testing\n\nBefore proceeding, if you're using ROS Melodic make sure that `roscore` is running first:\n\n```bash\n$ roscore\n```\n\nIf you're using ROS2, running the core service is no longer required.\n\n### Video Viewer\n\nFirst, it's recommended to test that you can stream a video feed using the [`video_source`](#video-source-node) and [`video_output`](#video-output-node) nodes.  See [Camera Streaming \u0026 Multimedia](https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md) for valid input/output streams, and substitute your desired `input` and `output` argument below.  For example, you can use video files for the input or output, or use V4L2 cameras instead of MIPI CSI cameras.  You can also use RTP/RTSP streams over the network.\n\n```bash\n# ROS\n$ roslaunch ros_deep_learning video_viewer.ros1.launch input:=csi://0 output:=display://0\n\n# ROS2\n$ ros2 launch ros_deep_learning video_viewer.ros2.launch input:=csi://0 output:=display://0\n```\n\n### imagenet Node\n\nYou can launch a classification demo with the following commands - substitute your desired camera or video path to the `input` argument below (see [here](https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md) for valid input/output streams).  \n\nNote that the `imagenet` node also publishes classification metadata on the `imagenet/classification` topic in a [`vision_msgs/Detection2DArray`](http://docs.ros.org/melodic/api/vision_msgs/html/msg/Detection2DArray.html) message -- see the [Topics \u0026 Parameters](#imagenet-node-1) section below for more info.\n\n```bash\n# ROS\n$ roslaunch ros_deep_learning imagenet.ros1.launch input:=csi://0 output:=display://0\n\n# ROS2\n$ ros2 launch ros_deep_learning imagenet.ros2.launch input:=csi://0 output:=display://0\n```\n\n### detectnet Node\n\nTo launch an object detection demo, substitute your desired camera or video path to the `input` argument below (see [here](https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md) for valid input/output streams).  Note that the `detectnet` node also publishes the metadata in a `vision_msgs/Detection2DArray` message -- see the [Topics \u0026 Parameters](#detectnet-node-1) section below for more info.\n\n#### \n\n```bash\n# ROS\n$ roslaunch ros_deep_learning detectnet.ros1.launch input:=csi://0 output:=display://0\n\n# ROS2\n$ ros2 launch ros_deep_learning detectnet.ros2.launch input:=csi://0 output:=display://0\n```\n\n### segnet Node\n\nTo launch a semantic segmentation demo, substitute your desired camera or video path to the `input` argument below (see [here](https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md) for valid input/output streams).  Note that the `segnet` node also publishes raw segmentation results to the `segnet/class_mask` topic -- see the [Topics \u0026 Parameters](#segnet-node-1) section below for more info.\n\n```bash\n# ROS\n$ roslaunch ros_deep_learning segnet.ros1.launch input:=csi://0 output:=display://0\n\n# ROS2\n$ ros2 launch ros_deep_learning segnet.ros2.launch input:=csi://0 output:=display://0\n```\n\n## Topics \u0026 Parameters\n\nBelow are the message topics and parameters that each node implements.\n\n### imagenet Node\n\n| Topic Name     |   I/O  | Message Type                                                                                                 | Description                                           |\n|----------------|:------:|--------------------------------------------------------------------------------------------------------------|-------------------------------------------------------|\n| image_in       |  Input | [`sensor_msgs/Image`](http://docs.ros.org/melodic/api/sensor_msgs/html/msg/Image.html)                       | Raw input image                                       |\n| classification | Output | [`vision_msgs/Classification2D`](http://docs.ros.org/melodic/api/vision_msgs/html/msg/Classification2D.html) | Classification results (class ID + confidence)        |\n| vision_info    | Output | [`vision_msgs/VisionInfo`](http://docs.ros.org/melodic/api/vision_msgs/html/msg/VisionInfo.html)             | Vision metadata (class labels parameter list name)         |\n| overlay        | Output | [`sensor_msgs/Image`](http://docs.ros.org/melodic/api/sensor_msgs/html/msg/Image.html)                       | Input image overlayed with the classification results |\n\n| Parameter Name    |       Type       |    Default    | Description                                                                                                        |\n|-------------------|:----------------:|:-------------:|--------------------------------------------------------------------------------------------------------------------|\n| model_name        |     `string`     | `\"googlenet\"` | Built-in model name (see [here](https://github.com/dusty-nv/jetson-inference#image-recognition) for valid values)  |\n| model_path        |     `string`     |      `\"\"`     | Path to custom caffe or ONNX model                                                                                 |\n| prototxt_path     |     `string`     |      `\"\"`     | Path to custom caffe prototxt file                                                                                 |\n| input_blob        |     `string`     |    `\"data\"`   | Name of DNN input layer                                                                                            |\n| output_blob       |     `string`     |    `\"prob\"`   | Name of DNN output layer                                                                                           |\n| class_labels_path |     `string`     |      `\"\"`     | Path to custom class labels file                                                                                   |\n| class_labels_HASH | `vector\u003cstring\u003e` |  class names  | List of class labels, where HASH is model-specific (actual name of parameter is found via the `vision_info` topic) |\n\n### detectnet Node\n\n| Topic Name  |   I/O  | Message Type                                                                                                 | Description                                                |\n|-------------|:------:|--------------------------------------------------------------------------------------------------------------|------------------------------------------------------------|\n| image_in    |  Input | [`sensor_msgs/Image`](http://docs.ros.org/melodic/api/sensor_msgs/html/msg/Image.html)                       | Raw input image                                            |\n| detections  | Output | [`vision_msgs/Detection2DArray`](http://docs.ros.org/melodic/api/vision_msgs/html/msg/Detection2DArray.html) | Detection results (bounding boxes, class IDs, confidences) |\n| vision_info | Output | [`vision_msgs/VisionInfo`](http://docs.ros.org/melodic/api/vision_msgs/html/msg/VisionInfo.html)             | Vision metadata (class labels parameter list name)         |\n| overlay     | Output | [`sensor_msgs/Image`](http://docs.ros.org/melodic/api/sensor_msgs/html/msg/Image.html)                       | Input image overlayed with the detection results      |\n\n| Parameter Name    |       Type       |        Default       | Description                                                                                                        |\n|-------------------|:----------------:|:--------------------:|--------------------------------------------------------------------------------------------------------------------|\n| model_name        |     `string`     | `\"ssd-mobilenet-v2\"` | Built-in model name (see [here](https://github.com/dusty-nv/jetson-inference#object-detection) for valid values)   |\n| model_path        |     `string`     |         `\"\"`         | Path to custom caffe or ONNX model                                                                                 |\n| prototxt_path     |     `string`     |         `\"\"`         | Path to custom caffe prototxt file                                                                                 |\n| input_blob        |     `string`     |       `\"data\"`       | Name of DNN input layer                                                                                            |\n| output_cvg        |     `string`     |     `\"coverage\"`     | Name of DNN output layer (coverage/scores)                                                                         |\n| output_bbox       |     `string`     |      `\"bboxes\"`      | Name of DNN output layer (bounding boxes)                                                                          |\n| class_labels_path |     `string`     |         `\"\"`         | Path to custom class labels file                                                                                   |\n| class_labels_HASH | `vector\u003cstring\u003e` |      class names     | List of class labels, where HASH is model-specific (actual name of parameter is found via the `vision_info` topic) |\n| overlay_flags     |     `string`     |  `\"box,labels,conf\"` | Flags used to generate the overlay (some combination of `none,box,labels,conf`)                                    |\n| mean_pixel_value  |      `float`     |          0.0         | Mean pixel subtraction value to be applied to input (normally 0)                                                   |\n| threshold         |      `float`     |          0.5         | Minimum confidence value for positive detections (0.0 - 1.0)                                                       |\n\n### segnet Node\n\n| Topic Name  |   I/O  | Message Type                                                                                     | Description                                              |\n|-------------|:------:|--------------------------------------------------------------------------------------------------|----------------------------------------------------------|\n| image_in    |  Input | [`sensor_msgs/Image`](http://docs.ros.org/melodic/api/sensor_msgs/html/msg/Image.html)           | Raw input image                                          |\n| vision_info | Output | [`vision_msgs/VisionInfo`](http://docs.ros.org/melodic/api/vision_msgs/html/msg/VisionInfo.html) | Vision metadata (class labels parameter list name)       |\n| overlay     | Output | [`sensor_msgs/Image`](http://docs.ros.org/melodic/api/sensor_msgs/html/msg/Image.html)           | Input image overlayed with the classification results    |\n| color_mask  | Output | [`sensor_msgs/Image`](http://docs.ros.org/melodic/api/sensor_msgs/html/msg/Image.html)           | Colorized segmentation class mask out                    |\n| class_mask  | Output | [`sensor_msgs/Image`](http://docs.ros.org/melodic/api/sensor_msgs/html/msg/Image.html)           | 8-bit single-channel image where each pixel is a classID |\n\n| Parameter Name    |       Type       |                Default               | Description                                                                                                           |\n|-------------------|:----------------:|:------------------------------------:|-----------------------------------------------------------------------------------------------------------------------|\n| model_name        |     `string`     | `\"fcn-resnet18-cityscapes-1024x512\"` | Built-in model name (see [here](https://github.com/dusty-nv/jetson-inference#semantic-segmentation) for valid values) |\n| model_path        |     `string`     |                 `\"\"`                 | Path to custom caffe or ONNX model                                                                                    |\n| prototxt_path     |     `string`     |                 `\"\"`                 | Path to custom caffe prototxt file                                                                                    |\n| input_blob        |     `string`     |               `\"data\"`               | Name of DNN input layer                                                                                               |\n| output_blob       |     `string`     |        `\"score_fr_21classes\"`        | Name of DNN output layer                                                                                              |\n| class_colors_path |     `string`     |                 `\"\"`                 | Path to custom class colors file                                                                                      |\n| class_labels_path |     `string`     |                 `\"\"`                 | Path to custom class labels file                                                                                      |\n| class_labels_HASH | `vector\u003cstring\u003e` |              class names             | List of class labels, where HASH is model-specific (actual name of parameter is found via the `vision_info` topic)    |\n| mask_filter       |     `string`     |              `\"linear\"`              | Filtering to apply to color_mask topic (`linear` or `point`)                                                          |\n| overlay_filter    |     `string`     |              `\"linear\"`              | Filtering to apply to overlay topic (`linear` or `point`)                                                             |\n| overlay_alpha     |      `float`     |                `180.0`               | Alpha blending value used by overlay topic (0.0 - 255.0)                                                              |\n\n### video_source Node\n\n| Topic Name |   I/O  | Message Type                                                                           | Description             |\n|------------|:------:|----------------------------------------------------------------------------------------|-------------------------|\n| raw        | Output | [`sensor_msgs/Image`](http://docs.ros.org/melodic/api/sensor_msgs/html/msg/Image.html) | Raw output image (BGR8) |\n\n| Parameter      |   Type   |   Default   | Description                                                                                                                                                               |\n|----------------|:--------:|:-----------:|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| resource       | `string` | `\"csi://0\"` | Input stream URI (see [here](https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md#input-streams) for valid protocols)                           |\n| codec          | `string` |     `\"\"`    | Manually specify codec for compressed streams (see [here](https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md#input-options) for valid values) |\n| width          |   `int`  |      0      | Manually specify desired width of stream (0 = stream default)                                                                                                             |\n| height         |   `int`  |      0      | Manually specify desired height of stream (0 = stream default)                                                                                                            |\n| framerate      |   `int`  |      0      | Manually specify desired framerate of stream (0 = stream default)                                                                                                         |\n| loop           |   `int`  |      0      | For video files:  `0` = don't loop, `\u003e0` = # of loops, `-1` = loop forever                                                                                                |\n| flip           | `string` |    `\"\"`     | Set the flip method for MIPI CSI cameras (see [here](https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md#input-options) for valid values)      |\n### video_output Node\n\n| Topic Name |  I/O  | Message Type                                                                           | Description     |\n|------------|:-----:|----------------------------------------------------------------------------------------|-----------------|\n| image_in   | Input | [`sensor_msgs/Image`](http://docs.ros.org/melodic/api/sensor_msgs/html/msg/Image.html) | Raw input image |\n\n| Parameter      |   Type   |     Default     | Description                                                                                                                                                   |\n|----------------|:--------:|:---------------:|---------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| resource       | `string` | `\"display://0\"` | Output stream URI (see [here](https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md#output-streams) for valid protocols)             |\n| codec          | `string` |     `\"h264\"`    | Codec used for compressed streams (see [here](https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md#input-options) for valid values) |\n| bitrate        |   `int`  |     4000000     | Target VBR bitrate of encoded streams (in bits per second)                                                                                                    |\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdusty-nv%2Fros_deep_learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdusty-nv%2Fros_deep_learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdusty-nv%2Fros_deep_learning/lists"}