{"id":28421685,"url":"https://github.com/nvidia-isaac-ros/ros2_benchmark","last_synced_at":"2026-02-20T07:01:39.530Z","repository":{"id":156874168,"uuid":"620536920","full_name":"NVIDIA-ISAAC-ROS/ros2_benchmark","owner":"NVIDIA-ISAAC-ROS","description":"Benchmark the performance of your ROS 2 graphs","archived":false,"fork":false,"pushed_at":"2025-12-11T06:15:49.000Z","size":284,"stargazers_count":121,"open_issues_count":2,"forks_count":17,"subscribers_count":7,"default_branch":"main","last_synced_at":"2026-01-26T19:51:29.535Z","etag":null,"topics":["benchmarking","performance","performance-testing","ros2","ros2-humble"],"latest_commit_sha":null,"homepage":"https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2023","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVIDIA-ISAAC-ROS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-03-28T22:02:31.000Z","updated_at":"2026-01-20T16:01:17.000Z","dependencies_parsed_at":"2024-06-14T23:33:36.026Z","dependency_job_id":"386ac19f-27fb-4d93-9dbb-ba064f247eea","html_url":"https://github.com/NVIDIA-ISAAC-ROS/ros2_benchmark","commit_stats":{"total_commits":17,"total_committers":4,"mean_commits":4.25,"dds":0.5882352941176471,"last_synced_commit":"5f458ea87b2086b325d8e24de77cadcf74083c60"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"purl":"pkg:github/NVIDIA-ISAAC-ROS/ros2_benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-ISAAC-ROS%2Fros2_benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-ISAAC-ROS%2Fros2_benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-ISAAC-ROS%2Fros2_benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-ISAAC-ROS%2Fros2_benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVIDIA-ISAAC-ROS","download_url":"https://codeload.github.com/NVIDIA-ISAAC-ROS/ros2_benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-ISAAC-ROS%2Fros2_benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29643960,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-20T05:21:04.652Z","status":"ssl_error","status_checked_at":"2026-02-20T05:21:04.238Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","performance","performance-testing","ros2","ros2-humble"],"created_at":"2025-06-05T06:09:17.906Z","updated_at":"2026-02-20T07:01:39.524Z","avatar_url":"https://github.com/NVIDIA-ISAAC-ROS.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ros2_benchmark\n\n\u003cdiv align=\"center\"\u003e\u003cimg alt=\"ros2_benchmark introduction\" src=\"resources/ros2_benchmark_intro.png\" width=\"500px\"/\u003e\u003c/div\u003e\n\n## Overview\n\nRobots are real-time systems which require complex graphs of heterogeneous computation to perform perception, planning, and control. These graphs of computation need to perform work deterministically and with known latency. The computing platform has a fixed budget for heterogeneous computation (TOPS) and throughput; computation is typically performed on multiple CPUs, GPUs, and additional special purpose, fixed function hardware accelerators.\n\n`ros2_benchmark` provides the tools for measuring the throughput, latency, and compute utilization of these complex graphs without altering the code under test. The results can be used to make informed design decisions on how best a robotics application can meet its real-time requirements. Results can be used to optimize system performance by tracking results over time against changes in the implementation and can be used in the development of program flow monitors to detect anomalies during operation of the real-time robotics application.\n\nThis tooling allows for realistic assessments of robotics application performance under load including message transport costs in [RCL](https://github.com/ros2/rclcpp) for practical benchmarking indicative of your real-world results. Message transport costs can be measured intra-process or inter-process including DDS overhead with support for [type adaptation](https://ros.org/reps/rep-2007.html). This tooling does not require modification of the graph of nodes under test to measure results, allowing both open source and proprietary solutions to be measured with the same tools in an non-intrusive way. Input for benchmarking is standardized with available rosbag datasets accompanying this package.\n\nDesigned for local developer use or in CI/CD platforms, these tools can be containerized to run on cloud native platforms such as Kubernetes. The tools are commercially hardened over tens of thousands of runs. We use this nightly on 7 hardware platforms using `aarch64` and `x86_64` architectures on multiple graph configurations.\n\n\u003cdiv align=\"center\"\u003e\u003cimg alt=\"ros2_benchmark architecture\" src=\"resources/ros2_benchmark_arch.png\" width=\"800px\"/\u003e\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\u003ci\u003e\u003ccode\u003eros2_benchmark\u003c/code\u003e uses the benchmark controller to orchestrate the data loader, playback and monitor nodes to perform benchmark runs, and calculate performance results into a benchmark report.\n\n\u003cbr\u003e\nThe data loader node fetches input data from rosbag. Input data is pre-processed using a configurable graph of nodes, and buffered into memory in the playback node which supports a plug-in for type adaptation. The graph benchmarked runs unmodified with input from the playback node controlling the data rate to output received at the monitor node.\u003c/i\u003e\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\u003cimg alt=\"ros2_benchmark flow\" src=\"resources/ros2_benchmark_flow.png\" width=\"900px\"/\u003e\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\u003ci\u003e\u003ccode\u003eros2_benchmark\u003c/code\u003e loads data from rosbag(s), performs any data pre-processing using a graph of ros nodes, and buffers the input data for benchmarking. If measuring peak throughput, the auto finder runs the graph under benchmark at multiple publisher rates to find the maximum publisher rate with less than 5% drops through the graph, otherwise it uses the specified fixed publishing rate or the timing from the rosbag.\n\n\u003cbr\u003e\nThe graph under benchmark is measured multiple times, with calculated results in a benchmark report.\u003c/i\u003e\u003c/div\u003e\n\n## Table of Contents\n\n- [ros2\\_benchmark](#ros2_benchmark)\n  - [Overview](#overview)\n  - [Table of Contents](#table-of-contents)\n  - [Latest Update](#latest-update)\n  - [Supported Platforms](#supported-platforms)\n  - [Quickstart](#quickstart)\n  - [Datasets](#datasets)\n    - [r2b Dataset 2024](#r2b-dataset-2024)\n    - [r2b Dataset 2023](#r2b-dataset-2023)\n  - [Results](#results)\n    - [Example Results](#example-results)\n    - [Explanation of the Results JSON Format](#explanation-of-the-results-json-format)\n  - [Creating Custom Benchmark](#creating-custom-benchmark)\n  - [Profiling](#profiling)\n  - [Live Benchmark Mode](#live-benchmark-mode)\n  - [Updates](#updates)\n\n## Latest Update\n\nUpdate 2023-11-16: Add support for live benchmark mode\n\n## Supported Platforms\n\nThis package is designed and tested to be compatible aarch64 and x86_64 platforms using ROS 2 Humble.\n\n| Platform hardware   | Platform software                                   | ROS Version                                               |\n| ------------------- | --------------------------------------------------- | --------------------------------------------------------- |\n| aarch64 \u003cbr\u003e x86_64 | [Ubuntu 20.04+](https://releases.ubuntu.com/20.04/) | [ROS 2 Humble](https://docs.ros.org/en/humble/index.html) |\n\n\u003e **Note**: `ros2_benchmark` has been tested on multiple computing platforms including [Intel NUC Corei7 11th Gen](https://ark.intel.com/content/www/us/en/ark/products/228816/intel-nuc-11-enterprise-edge-compute-nuc11tnhv70l.html) and [Jetson Orin](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/).\n\n## Quickstart\n\nTo use and learn to use `ros2_benchmark`, start by running a sample benchmark. Follow the steps below to start measuring the performance of an AprilTag node with `ros2_benchmark`.\n\n1. Install ROS 2 Humble natively (see [here](https://docs.ros.org/en/humble/Installation.html)) or launch official Docker container with ROS 2 Humble pre-installed:\n\n    ```bash\n    docker run -it ros:humble\n    ```\n\n2. Setup convenience environment variables and install tools.\n    ```bash\n    export R2B_WS_HOME=~/ros_ws \u0026\u0026 \\\n    export ROS2_BENCHMARK_OVERRIDE_ASSETS_ROOT=$R2B_WS_HOME/src/ros2_benchmark/assets \u0026\u0026 \\\n    sudo apt-get update \u0026\u0026 sudo apt-get install -y git git-lfs wget\n    ```\n\n3. Clone this repository along with an available implementation of Apriltag detection and install dependencies.\n\n    ```bash\n    mkdir -p $R2B_WS_HOME/src \u0026\u0026 cd $R2B_WS_HOME/src \u0026\u0026 \\\n        git clone https://github.com/NVIDIA-ISAAC-ROS/ros2_benchmark.git \u0026\u0026 \\\n        git clone https://github.com/christianrauch/apriltag_ros.git \u0026\u0026 \\\n    cd $R2B_WS_HOME \u0026\u0026 \\\n        sudo apt-get update \u0026\u0026 \\\n        rosdep update \u0026\u0026 rosdep install -i -r --from-paths src --rosdistro humble -y\n    ```\n\n4. Clone and build `image_proc` package with patch to fix incompatible QoS settings.\n\n    ```bash\n    cd $R2B_WS_HOME/src \u0026\u0026 \\\n      git clone https://github.com/ros-perception/vision_opencv.git \u0026\u0026 cd vision_opencv \u0026\u0026 git checkout humble \u0026\u0026 \\\n    cd $R2B_WS_HOME/src \u0026\u0026 \\\n      git clone https://github.com/ros-perception/image_pipeline.git \u0026\u0026 cd image_pipeline \u0026\u0026 git checkout humble \u0026\u0026 \\\n      git config user.email \"benchmarking@ros2_benchmark.com\" \u0026\u0026 git config user.name \"ROS 2 Developer\" \u0026\u0026 \\\n      wget https://raw.githubusercontent.com/NVIDIA-ISAAC-ROS/ros2_benchmark/main/resources/patch/resize_qos_profile.patch \u0026\u0026 \\\n      git apply resize_qos_profile.patch \u0026\u0026 \\\n    cd $R2B_WS_HOME \u0026\u0026 \\\n      sudo apt-get update \u0026\u0026 \\\n      rosdep update \u0026\u0026 rosdep install -i -r --from-paths src --rosdistro humble -y \u0026\u0026 \\\n      colcon build --packages-up-to image_proc\n    ```\n\n5. Pull down `r2b dataset 2023` by following the instructions [here](#datasets) or fetch just the rosbag used in this Quickstart with the following command.\n\n    ```bash\n    mkdir -p $R2B_WS_HOME/src/ros2_benchmark/assets/datasets/r2b_dataset/r2b_storage \u0026\u0026 \\\n    cd $R2B_WS_HOME/src/ros2_benchmark/assets/datasets/r2b_dataset/r2b_storage \u0026\u0026 \\\n      wget 'https://api.ngc.nvidia.com/v2/resources/nvidia/isaac/r2bdataset2023/versions/2/files/r2b_storage/metadata.yaml' \u0026\u0026 \\\n      wget 'https://api.ngc.nvidia.com/v2/resources/nvidia/isaac/r2bdataset2023/versions/2/files/r2b_storage/r2b_storage_0.db3'\n    ```\n\n6. Build `ros2_benchmark` and source the workspace:\n\n    ```bash\n    cd $R2B_WS_HOME \u0026\u0026 \\\n      colcon build --packages-up-to ros2_benchmark apriltag_ros \u0026\u0026 \\\n      source install/setup.bash\n    ```\n\n7. (Optional) Run tests to verify complete and correct installation:\n\n    ```bash\n    colcon test --packages-select ros2_benchmark\n    ```\n\n8. Start the AprilTag benchmark:\n\n    ```bash\n    launch_test src/ros2_benchmark/scripts/apriltag_ros_apriltag_node.py\n    ```\n\nOnce the benchmark is finished, the final performance measurements are displayed in the terminal.\nAdditionally, the final results and benchmark metadata (e.g., system information, benchmark configurations) are also exported as a JSON file.\n\n## Datasets\n\nInput data for benchmarking is provided in a rosbag.\n\nTo provide consistency of results, we have provided multiple dataset sequences in rosbag for use with `ros2_benchmark`; input data in other rosbag(s) can be used. These dataset sequences were captured on a robot, using very high precision time synchronization between sensors.  Captured sensor data includes [HAWK (2mp RGB stereo camera with IMU)](https://www.leopardimaging.com/li-ar0234cs-stereo-gmsl2-hawk/), [D455](https://realsenseai.com/stereo-depth-cameras/real-sense-depth-camera-d455/) and [XT32](https://www.hesaitech.com/product/xt32/).\n\nThese datasets are explicitly **not** provided inside this repository. Instead, visit NGC to download the dataset [here](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/collections/r2bdataset).\n\nYou can also download the datasets with command-line tools as follows by first installing the NGC CLI.\n\n```bash\nwget --content-disposition https://ngc.nvidia.com/downloads/ngccli_linux.zip \u0026\u0026 unzip ngccli_linux.zip \u0026\u0026 chmod u+x ngc-cli/ngc\n```\n\nWith the NGC CLI available, you can download the datasets with the following commands:\n\n```bash\n./ngc-cli/ngc registry resource download-version \"nvidia/isaac/r2bdataset2023:3\"\n./ngc-cli/ngc registry resource download-version \"nvidia/isaac/r2bdataset2024:1\"\n```\n\nThen, move the datasets to their required location:\n\n```bash\nmkdir assets/datasets/r2b_dataset\nmv r2bdataset2023_v3/* assets/datasets/r2b_dataset\nmv r2bdataset2024_v1/* assets/datasets/r2b_dataset\n```\n\n### r2b Dataset 2024\n| Sequence                                                                                           | Size | Visual                                                                                                                                                                                    | Contents                                                                                                                          | Description                                                                                            |\n| -------------------------------------------------------------------------------------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |\n| [r2b_galileo](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2024)     | 471M | \u003cimg alt=\"galileo sequence\" src=\"https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ngc/r2bdataset/r2b_galileo_sequence.gif\" width=\"300px\"/\u003e         | 4x LI HAWK stereo \u003cbr\u003e L+R 1920x1200 RGB 30fps\u003cbr\u003e\u003cbr\u003e IMU \u003cbr\u003e Wheel odometry                                                    | Room sequence including robots, shelving, boxes, pallets, with moving obstacles.                       |\n| [r2b_galileo2](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2024)    | 986M | \u003cimg alt=\"galileo2 sequence\" src=\"https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ngc/r2bdataset/r2b_galileo2_sequence.gif\" width=\"300px\"/\u003e       | RealSense D455 \u003cbr\u003e L+R 1280x720 IR Mono 30fps \u003cbr\u003e Depth 1280x780 30fps \u003cbr\u003e 1280x800 RGB 30fps \u003cbr\u003e\u003cbr\u003e IMU \u003cbr\u003e Wheel odometry | Room sequence including robots, shelving, boxes, pallets, with moving obstacles.                       |\n| [r2b_robotarm](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2024)    | 1.4G | \u003cimg alt=\"robotarm sequence\" src=\"https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ngc/r2bdataset/r2b_robotarm_sequence.gif\" width=\"300px\"/\u003e       | RealSense D455 \u003cbr\u003e L+R 1280x720 IR Mono 30fps \u003cbr\u003e Depth 1280x780 30fps \u003cbr\u003e 1280x800 RGB 30fps \u003cbr\u003e\u003cbr\u003e Robot joint states      | Robot arm sequence containing objects on a tray.                                                       |\n| [r2b_whitetunnel](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2024) | 503M | \u003cimg alt=\"whitetunnel sequence\" src=\"https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ngc/r2bdataset/r2b_whitetunnel_sequence.gif\" width=\"300px\"/\u003e | 4x LI HAWK stereo \u003cbr\u003e L+R 1920x1200 RGB 30fps\u003cbr\u003e\u003cbr\u003e IMU \u003cbr\u003e Wheel odometry                                                    | Hallway sequence with low to no feature not-perpendicular walls, specular highlights, and reflections. |\n\n### r2b Dataset 2023\n| Sequence                                                                                          | Size | Visual                                                                                     | Contents                                                                                                                                                                       | Description                                                                                                                                                                                        |\n| ------------------------------------------------------------------------------------------------- | ---- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| [r2b_lounge](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2023)     | 3.9G | \u003cimg alt=\"lounge sequence\" src=\"resources/r2b_lounge_sequence.gif\" width=\"300px\"/\u003e         | LI HAWK stereo \u003cbr\u003e L+R 1920x1200 RGB 30fps\u003cbr\u003e\u003cbr\u003e RealSense D455 \u003cbr\u003e L+R 1280x720 Mono IR 30fps \u003cbr\u003e Depth 1280x780 30fps \u003cbr\u003e 1280x800 RGB 30fps \u003cbr\u003e Hesai XT32 10Hz \u003cbr\u003e | Lounge sequence containing couch, table, chairs, and staircase with natural planted background wall.                                                                                               |\n| [r2b_storage](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2023)    | 2.9G | \u003cimg alt=\"storage sequence\" src=\"resources/r2b_storage_sequence.gif\" width=\"300px\"/\u003e       | LI HAWK stereo \u003cbr\u003e L+R 1920x1200 RGB 30fps\u003cbr\u003e\u003cbr\u003e RealSense D455 \u003cbr\u003e L+R 1280x720 IR Mono 30fps \u003cbr\u003e Depth 1280x780 30fps \u003cbr\u003e 1280x800 RGB 30fps \u003cbr\u003e Hesai XT32 10Hz \u003cbr\u003e | Storage sequence including person, AprilTag, shoe, shelving, boxes, pallets                                                                                                                        | skids, dollys, robots, boundary floor tape, calibration target, and color checker, with 50% reflective background grey walls. |\n| [r2b_hallway](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2023)    | 1.3G | \u003cimg alt=\"hallway sequence\" src=\"resources/r2b_hallway_sequence.gif\" width=\"300px\"/\u003e       | LI HAWK stereo \u003cbr\u003e L+R 1920x1200 RGB 30fps\u003cbr\u003e\u003cbr\u003e RealSense D455 \u003cbr\u003e L+R 1280x720 Mono 30fps \u003cbr\u003e Depth 1280x780 30fps \u003cbr\u003e 1280x800 RGB 30fps \u003cbr\u003e Hesai XT32 10Hz \u003cbr\u003e    | Hallway sequence with walking persons, low to no feature not-perpendicular walls, specular highlights, and reflections.                                                                            |\n| [r2b_datacenter](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2023) | 1.7G | \u003cimg alt=\"datacenter sequence\" src=\"resources/r2b_datacenter_sequence.gif\" width=\"300px\"/\u003e | LI HAWK stereo \u003cbr\u003e L+R 1920x1200 RGB 30fps\u003cbr\u003e\u003cbr\u003e RealSense D455 \u003cbr\u003e L+R 1280x720 Mono 30fps \u003cbr\u003e Depth 1280x780 30fps \u003cbr\u003e 1280x800 RGB 30fps \u003cbr\u003e Hesai XT32 10Hz \u003cbr\u003e    | Datacenter sequence with tall vertical corridor repetitive low-feature surfaces, little color.                                                                                                     |\n| [r2b_cafe](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2023)       | 1.2G | \u003cimg alt=\"cafe sequence\" src=\"resources/r2b_cafe_sequence.gif\" width=\"300px\"/\u003e             | LI HAWK stereo \u003cbr\u003e L+R 1920x1200 RGB 30fps\u003cbr\u003e\u003cbr\u003e RealSense D455 \u003cbr\u003e L+R 1280x720 Mono 30fps \u003cbr\u003e Depth 1280x780 IR 30fps \u003cbr\u003e 1280x800 RGB 30fps \u003cbr\u003e Hesai XT32 10Hz \u003cbr\u003e | Café sequence including table, chairs, stools, reflective flooring, dark reflective glass walls, specular highlights, low wall features, and vibration from floor surface.                         |\n| [r2b_hope](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2023)       | 30M  | \u003cimg alt=\"hope sequence\" src=\"resources/r2b_hope_sequence.gif\" width=\"300px\"/\u003e             | D415 RGB                                                                                                                                                                       | Image from [HOPE dataset](https://github.com/swtyree/hope-dataset/) for 6-DoF pose estimation from [scene 0005](https://github.com/swtyree/hope-dataset/tree/master/hope-image-preview/scene_0005) |\n| [r2b_hideaway](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2023)   | 1.8G | \u003cimg alt=\"hideaway sequence\" src=\"resources/r2b_hideaway_sequence.gif\" width=\"300px\"/\u003e     | LI HAWK stereo \u003cbr\u003e L+R 1920x1200 RGB 30fps\u003cbr\u003e\u003cbr\u003e RealSense D455 \u003cbr\u003e L+R 1280x720 IR Mono 30fps \u003cbr\u003e Depth 1280x780 30fps \u003cbr\u003e 1280x800 RGB 30fps \u003cbr\u003e Hesai XT32 10Hz \u003cbr\u003e | Hideaway sequence including table, chairs, seated and moving persons specular highlights, low wall features, and vibration from floor surface.                                                     |\n| [r2b_mezzanine](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2023)  | 2.0G | \u003cimg alt=\"mezzanine sequence\" src=\"resources/r2b_mezzanine_sequence.gif\" width=\"300px\"/\u003e   | LI HAWK stereo \u003cbr\u003e L+R 1920x1200 RGB 30fps\u003cbr\u003e\u003cbr\u003e RealSense D455 \u003cbr\u003e L+R 1280x720 IR Mono 30fps \u003cbr\u003e Depth 1280x780 30fps \u003cbr\u003e 1280x800 RGB 30fps \u003cbr\u003e Hesai XT32 10Hz \u003cbr\u003e | Mezzanine sequence including staircase, railings, table, chairs, highlights, low wall features, and vibration from floor surface.                                                                  |\n\n## Results\n\nPerformance measurements are output to a results JSON file. JSON provides a human-readable format which allows for traceable independently verifiable results and can be conveniently imported into your visualization tool of choice.\n\nDefault measurements include throughput, latency, and jitter.  These measurements can be performed for the sample rate from the dataset sequence of the rosbag input, at peak throughput, and for fixed frequencies often to represent a sensor capture rate; for example 10hz for LIDAR or 30fps for camera.\n\nIncluded in the log is information on the host system on which results were measured, when they were measured, and the corresponding software version. The input data used, and a hash of the input data file is reported for traceability of results.  The input YAML configuration used for the benchmark run is reported allowing results to be independent reproduced with the same configuration.\n\n\u003e **Note**: We use the naming convention `_node` to represent a graph under test that contains a single node (for example, `stereo_image_proc_node.py`) and `_graph` to represent a graph of multiple nodes (for example, `stereo_image_proc_graph.py`).\n\n### Example Results\n\nThe following are the performance results measured with `ros2_benchmark` on `aarch64` and `x86_64` platforms, using ROS 2 Humble in March 2023. The table below also contains links to the packages for the nodes used in the benchmark and to the complete results JSON files.\n\n| Node                                                                                  | Input Size | Intel NUC Corei7 11th Gen                                                      | AGX Orin (CPU only)                                                          |\n| ------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- |\n| [AprilTag Node](scripts/apriltag_ros_apriltag_node.py)                                | 720p       | [90.8 fps](results/apriltag_ros_apriltag_node-nuc_4060ti.json)\u003cbr\u003e11 ms        | [56.3 fps](results/apriltag_ros_apriltag_node-agx_orin.json)\u003cbr\u003e18 ms        |\n| [Rectify Node](scripts/image_proc_rectify_node.py)                                    | 1080p      | [539 fps](results/image_proc_rectify_node-nuc_4060ti.json)\u003cbr\u003e1.9 ms           | [185 fps](results/image_proc_rectify_node-agx_orin.json)\u003cbr\u003e5.6 ms           |\n| [H.264 Encoder Node\u003cbr\u003eI-frame Support](scripts/image_transport_h264_decoder_node.py) | 1080p      | [60.5 fps](results/image_transport_h264_decoder_node-nuc_4060ti.json)\u003cbr\u003e19 ms | [28.0 fps](results/image_transport_h264_decoder_node-agx_orin.json)\u003cbr\u003e37 ms |\n| [H.264 Encoder Node\u003cbr\u003eP-frame Support](scripts/image_transport_h264_encoder_node.py) | 1080p      | [43.4 fps](results/image_transport_h264_encoder_node-nuc_4060ti.json)\u003cbr\u003e24 ms | [10.2 fps](results/image_transport_h264_encoder_node-agx_orin.json)\u003cbr\u003e95 ms |\n| [Stereo Disparity Node](scripts/stereo_image_proc_node.py)                            | 1080p      | [99.5 fps](results/stereo_image_proc_node-nuc_4060ti.json)\u003cbr\u003e6.4 ms           | [66.5 fps](results/stereo_image_proc_node-agx_orin.json)\u003cbr\u003e15 ms            |\n\n| Graph                                                        | Input Size | Intel NUC Corei7 11th Gen                                                | AGX Orin (CPU only)                                                    |\n| ------------------------------------------------------------ | ---------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------- |\n| [AprilTag Graph](scripts/apriltag_ros_apriltag_graph.py)     | 720p       | [88.1 fps](results/apriltag_ros_apriltag_graph-nuc_4060ti.json)\u003cbr\u003e12 ms | [56.3 fps](results/apriltag_ros_apriltag_graph-agx_orin.json)\u003cbr\u003e22 ms |\n| [Stereo Disparity Graph](scripts/stereo_image_proc_graph.py) | 1080p      | [99.4 fps](results/stereo_image_proc_graph-nuc_4060ti.json)\u003cbr\u003e16 ms     | [63.5 fps](results/stereo_image_proc_graph-agx_orin.json)\u003cbr\u003e28 ms     |\n\n\n\u003e **Note**: All results above are using ROS 2 nodes from open source that run computation on the CPU only. For GPU-accelerated equivalent packages, see [Isaac ROS](https://github.com/NVIDIA-ISAAC-ROS/).\n\n### Explanation of the Results JSON Format\n\nAfter a `ros2_benchmark`-based benchmark is complete, the framework will output a detailed log of the results in a JSON format. This section explains that JSON format through the use of an example.\n\nThe first section of the output JSON file presents the results achieved running at the peak throughput, as identified by the harness' auto-tune search process. That peak throughput is logged with the key `MEAN_FRAME_RATE`; in this sample, the corresponding value is about ~6.09fps. This iteration of the test was run for `RECEIVED_DURATION = 4919.10` milliseconds. There were `NUM_MISSED_FRAMES = 20.0` frames dropped somewhere in `rclcpp` transport over the path that originates from the playback node, goes through the graph under test, and terminates in the monitor node.\n\n```json\n{\n  \"BasicPerformanceMetrics.RECEIVED_DURATION\": 4919.0,\n  \"BasicPerformanceMetrics.MEAN_PLAYBACK_FRAME_RATE\": 10.20338761907538,\n  \"BasicPerformanceMetrics.MEAN_FRAME_RATE\": 6.098802584821505,\n  \"BasicPerformanceMetrics.NUM_MISSED_FRAMES\": 20.0,\n  \"BasicPerformanceMetrics.NUM_FRAMES_SENT\": 50.0,\n  \"BasicPerformanceMetrics.FIRST_SENT_RECEIVED_LATENCY\": 29.333333333333332,\n  \"BasicPerformanceMetrics.LAST_SENT_RECEIVED_LATENCY\": 48.0,\n  \"BasicPerformanceMetrics.MAX_JITTER\": 231.0,\n  \"BasicPerformanceMetrics.MIN_JITTER\": 0.0,\n  \"BasicPerformanceMetrics.MEAN_JITTER\": 106.96428571428571,\n  \"BasicPerformanceMetrics.STD_DEV_JITTER\": 54.942484366766024,\n  \"CPUProfilingMetrics.MAX_CPU_UTIL\": 7.633333333333334,\n  \"CPUProfilingMetrics.MIN_CPU_UTIL\": 0.05555555555555555,\n  \"CPUProfilingMetrics.MEAN_CPU_UTIL\": 1.5159932659932658,\n  \"CPUProfilingMetrics.STD_DEV_CPU_UTIL\": 1.4090849674792143,\n  \"CPUProfilingMetrics.BASELINE_CPU_UTIL\": 2.1425925925925924,\n}\n```\n\nThe next section of results measures latency at a fixed input throughput of `10.0fps`, as indicated by the section header. In this case, `MEAN_PLAYBACK_FRAME_RATE = 10.2` fps indicates that the true input throughput achieved during the test was close but not exactly equal to the nominal value. The `MEAN_JITTER = 115` ms and `MAX_JITTER = 251` ms indicate the mean and max jitter, respectively. The value `MEAN_FRAME_RATE = 6.12` fps indicates that the output throughput of the graph was significantly slower than the input throughput; this corroborates the previous section's conclusion that the max sustainable frame rate is about ~6.09fps. This iteration of the test was run for `RECEIVED_DURATION = 4900` ms with `NUM_MISSED_FRAMES = 20` frames dropped somewhere in the `rclcpp` transport process.\n\nLatency tests are often run at the processing rate for the graph under test. The playback rate can be tied to the sensor input rate; many LIDARs run at 10Hz, while cameras may run at 30fps or 60fps.\n\nIf desired, additional fixed playback rates can be specified to calculate additional latency measurements at multiple processing rates.\n\n```json\n{\n  \"10.0fps\": {\n    \"BasicPerformanceMetrics.RECEIVED_DURATION\": 4900,\n    \"BasicPerformanceMetrics.MEAN_PLAYBACK_FRAME_RATE\": 10.204081632653061,\n    \"BasicPerformanceMetrics.MEAN_FRAME_RATE\": 6.122448979591836,\n    \"BasicPerformanceMetrics.NUM_MISSED_FRAMES\": 20,\n    \"BasicPerformanceMetrics.NUM_FRAMES_SENT\": 50,\n    \"BasicPerformanceMetrics.FIRST_SENT_RECEIVED_LATENCY\": 23,\n    \"BasicPerformanceMetrics.LAST_SENT_RECEIVED_LATENCY\": 23,\n    \"BasicPerformanceMetrics.MAX_JITTER\": 251.0,\n    \"BasicPerformanceMetrics.MIN_JITTER\": 2.0,\n    \"BasicPerformanceMetrics.MEAN_JITTER\": 115.64285714285714,\n    \"BasicPerformanceMetrics.STD_DEV_JITTER\": 58.463183949824526,\n    \"CPUProfilingMetrics.MAX_CPU_UTIL\": 7.855555555555556,\n    \"CPUProfilingMetrics.MIN_CPU_UTIL\": 0.16666666666666666,\n    \"CPUProfilingMetrics.MEAN_CPU_UTIL\": 5.264444444444445,\n    \"CPUProfilingMetrics.STD_DEV_CPU_UTIL\": 2.9738969321351676,\n    \"CPUProfilingMetrics.BASELINE_CPU_UTIL\": 7.805555555555555,\n  }\n}\n```\n\nFinally, the metadata provided at the end of the JSON file contains system and file information to provide a transparent and reproducible record of how the benchmark results were obtained. `BenchmarkMetadata.CONFIG` contains a copy of the configuration file YAML as a string. This configuration can be used to run a benchmark with identical parameters to those from the results file.\n\nThe name and checksum of the dataset used for the benchmark are also provided, ensuring that the same dataset has been used when comparing or reproducing independent results.\n\n```json\n{\n  \"metadata\": {\n    \"BenchmarkMetadata.NAME\": \"reference AprilTagNode benchmark\",\n    \"BenchmarkMetadata.TEST_FILE_PATH\": \"/workspaces/ros-dev/src/ros2_benchmark/scripts/reference_apriltag_node_test.py\",\n    \"BenchmarkMetadata.TEST_DATETIME\": \"2023-03-15T22:07:43Z\",\n    \"BenchmarkMetadata.DEVICE_HOSTNAME\": \"neuromancer\",\n    \"BenchmarkMetadata.DEVICE_ARCH\": \"x86_64\",\n    \"BenchmarkMetadata.DEVICE_OS\": \"Linux 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022\",\n    \"BenchmarkMetadata.BENCHMARK_MODE\": 1,\n    \"BenchmarkMetadata.PEAK_THROUGHPUT_PREDICTION\": 10.0,\n    \"BenchmarkMetadata.INPUT_DATA_PATH\": \"assets/r2b_storage/r2b_storage.db3\",\n    \"BenchmarkMetadata.INPUT_DATA_HASH\": \"b7e276d5105397dfb19a6f2c6db7672f\",\n    \"BenchmarkMetadata.CONFIG\": [copy of input config as stringified YAML]\n  }\n}\n```\n\n\u003e **Note**: The peak throughput of this sample run was capped at 10fps by the input configuation setting `\"BenchmarkMetadata.PEAK_THROUGHPUT_PREDICTION\": 10.0`.\n\n## Creating Custom Benchmark\n\nBenchmark your own graphs using `ros2_benchmark` framework by creating custom benchmark scripts from the minimum template shown below:\n\n```python\nfrom launch_ros.actions import ComposableNodeContainer\nfrom launch_ros.descriptions import ComposableNode\n\nfrom ros2_benchmark import ImageResolution\nfrom ros2_benchmark import ROS2BenchmarkConfig, ROS2BenchmarkTest\n\ndef launch_setup(container_prefix, container_sigterm_timeout):\n    \"\"\"Graph setup for benchmarking your custom graph.\"\"\"\n\n    # Insert your composable node declarations\n\n    # Required DataLoaderNode\n    data_loader_node = ComposableNode(\n        name='DataLoaderNode',\n        namespace=TestCustomGraph.generate_namespace(),\n        package='ros2_benchmark',\n        plugin='ros2_benchmark::DataLoaderNode',\n        # Insert remappings if necessary\n    )\n\n    # Insert your custom preprocessor graph if needed\n\n    # Required PlaybackNode\n    playback_node = ComposableNode(\n        name='PlaybackNode',\n        namespace=TestCustomGraph.generate_namespace(),\n        package='ros2_benchmark',\n        plugin='ros2_benchmark::PlaybackNode',\n        # Revise \"data_formats\" based on your graph\n        parameters=[{\n            'data_formats': [\n                'sensor_msgs/msg/Image',\n                'sensor_msgs/msg/CameraInfo'\n            ],\n        }],\n        # Revise \"remapping\" based on your graph\n        remappings=[\n            ('buffer/input0', 'data_loader_node/image'),\n            ('input0', 'image'),\n            ('buffer/input1', 'data_loader_node/camera_info'),\n            ('input1', 'camera_info')\n        ]\n    )\n\n    # Required MonitorNode\n    # You can add as many monitor nodes as you need to measure performance\n    # for multiple topics.\n    monitor_node = ComposableNode(\n        name='MonitorNode',\n        namespace=TestCustomGraph.generate_namespace(),\n        package='ros2_benchmark',\n        plugin='ros2_benchmark::MonitorNode',\n        parameters=[{\n            # Add \"monitor_index\" parameter to distinguish between various\n            # monitor nodes when multiple monitor nodes are used.\n            'monitor_data_format': 'apriltag_msgs/msg/AprilTagDetectionArray',\n        }],\n        # Revise \"remapping\" based on your graph\n        remappings=[\n            ('output', 'apriltag_detections')\n        ]\n    )\n\n    # Required composable node container\n    # Insert your composable nodes in the \"composable_node_descriptions\" list.\n    composable_node_container = ComposableNodeContainer(\n        name='container',\n        namespace=TestCustomGraph.generate_namespace(),\n        package='rclcpp_components',\n        executable='component_container_mt',\n        prefix=container_prefix,\n        sigterm_timeout=container_sigterm_timeout,\n        composable_node_descriptions=[\n            data_loader_node,\n            playback_node,\n            monitor_node,\n            # Insert custom nodes here\n        ],\n        output='screen'\n    )\n\n    return [composable_node_container]\n\ndef generate_test_description():\n    return TestCustomGraph.generate_test_description_with_nsys(launch_setup)\n\nclass TestCustomGraph(ROS2BenchmarkTest):\n    \"\"\"Performance test for your custom graph.\"\"\"\n\n    # Custom configurations\n    config = ROS2BenchmarkConfig(\n        # Insert your custom benchmark configurations\n        benchmark_name='Custom Graph Benchmark',\n        input_data_path='datasets/your_custom_rosbag_directory_path',\n        publisher_upper_frequency=100.0,\n        publisher_lower_frequency=10.0,\n        playback_message_buffer_size=10\n    )\n\n    def test_benchmark(self):\n        self.run_benchmark()\n```\n\nRevise the existing or insert your code in the template based on your graph to be measured.\n\nFollow these steps to ensure that everything in the template is configured correctly:\n\n1. Insert your custom graph (e.g., composable nodes) in the `launch_setup` method.\n2. Revise `remappings` in the data loader node to connect rosbag topics to either your preprocessor graph or a playback node.\n3. \\[Optional\\] Insert your preprocessor graph in the `launch_setup` method if required.\n4. Revise `data_formats` and `remappings` in the playback node to connect to the loaded/preprocessed data and your custom graph.\n5. Insert your custom nodes declared in step 1 to the composable node container.\n6. Revise/add benchmark configurations under `ROS2BenchmarkConfig` declaration based on your custom graph.\n\nThe full benchmark configuration options can be found [here](ros2_benchmark/ros2_benchmark/default_ros2_benchmark_config.yaml) in the default `ros2_benchmark` configuration file.\n\n## Profiling\n\nWhen seeking to optimize performance, profiling is often used to gain deep insight into the call stack, and where processing time is spent in functions. [ros2_tracing](https://github.com/ros2/ros2_tracing) provides a tracing instrumentation to better understand performance on a CPU, but lacks information on GPU acceleration.\n\n[Nsight Systems](https://developer.nvidia.com/nsight-systems) provides tracing instrumentation for CPU, GPU, and other SOC (system-on-chip) hardware accelerators for both `aarch64` and `x86_64` platforms, and is freely available for download.  We use this tooling to profile our graphs of computation in ROS, to identify areas of improvement for compute optimization, and improvement of synchronization between heterogenous computing hardware. These tools allow for comparison of before and after to inspect profile differences with the benchmark tooling.\n\n## Live Benchmark Mode\n\nBesides benchmarking a graph with data loaded from rosbags, `ros2_benchmark` also supports a live benchmark mode that enables measuring performance outcome of a graph with its data source (e.g., a grph consists of a camera node). A [Realsense benchmark script](scripts/realsense2_node.py) is provided as an example to showcase the use of the live benchmark mode.\n\n## Updates\n\n| Date       | Changes                             |\n| ---------- | ----------------------------------- |\n| 2024-05-30 | Include r2b Dataset 2024            |\n| 2023-11-16 | Add support for live benchmark mode |\n| 2023-04-05 | Initial release                     |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvidia-isaac-ros%2Fros2_benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnvidia-isaac-ros%2Fros2_benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvidia-isaac-ros%2Fros2_benchmark/lists"}