{"id":20099112,"url":"https://github.com/ros2/buildfarm_perf_tests","last_synced_at":"2025-05-06T06:30:59.150Z","repository":{"id":43409297,"uuid":"207922690","full_name":"ros2/buildfarm_perf_tests","owner":"ros2","description":"Performance tests which run regularly on the buildfarm","archived":false,"fork":false,"pushed_at":"2024-05-17T13:53:32.000Z","size":486,"stargazers_count":24,"open_issues_count":7,"forks_count":7,"subscribers_count":20,"default_branch":"master","last_synced_at":"2024-05-17T14:59:56.277Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ros2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-11T23:13:26.000Z","updated_at":"2024-03-22T02:02:17.000Z","dependencies_parsed_at":"2024-02-29T19:49:06.587Z","dependency_job_id":null,"html_url":"https://github.com/ros2/buildfarm_perf_tests","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ros2%2Fbuildfarm_perf_tests","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ros2%2Fbuildfarm_perf_tests/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ros2%2Fbuildfarm_perf_tests/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ros2%2Fbuildfarm_perf_tests/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ros2","download_url":"https://codeload.github.com/ros2/buildfarm_perf_tests/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224490708,"owners_count":17319983,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T17:08:18.787Z","updated_at":"2024-11-13T17:08:19.895Z","avatar_url":"https://github.com/ros2.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# buildfarm perf tests\n\n[![Continuous Integration](https://github.com/ros2/buildfarm_perf_tests/workflows/Continuous%20Integration/badge.svg?branch=master\u0026event=push)](https://github.com/ros2/buildfarm_perf_tests/actions?query=branch%3Amaster+event%3Apush)\n\n## Purpose\n\nThis package defines some tests.\nOn one hand it invokes `perf_test` from Apex.AI's [performance_test](https://gitlab.com/ApexAI/performance_test) package.\nThis allows you to test performance and latency of several ROS 2 RMW implementations.\nOn the other hand we are evaluating the additional overhead caused by a single pub/sub topic or one process spinning and detect potential leaks related to theses activities.\n\n* There is a test for each RMW:\n\n  - CycloneDDS (standalone test)\n  - FastRTPS (standalone test)\n  - rmw_connext_cpp\n  - rmw_cyclonedds_cpp\n  - rmw_fastrtps_cpp\n  - rmw_fastrtps_dynamic_cpp\n  - rmw_opensplice_cpp\n\n### Test 1 - Performance Test  (Apex.AI)\n\nIn this test we are running the Performance Test provided by Apex.AI.\nRight now we have [our own fork](https://github.com/ros2/performance_test) because there are some pending pull requests in the official gitlab repository.\n\nIn this test we are measurement:\n - Average latency\n - CPU usage (provided by Apex.AI tool)\n - Sent/Received messages per second\n - Total lost messages\n - Max resident set size\n\nWe are generating two plots per measurement\n - [One per-build](http://build.ros2.org/view/Rci/job/Rci__nightly-performance_ubuntu_focal_amd64/lastBuild/)\n - [Other over time](http://build.ros2.org/view/Rci/job/Rci__nightly-performance_ubuntu_focal_amd64/plot/)\n\n**The test only measures the latency between the same RMW implementation**\n\n| Publisher/Subscriber     | rmw_fastrtps_cpp         | rmw_opensplice_cpp       | rmw_cyclonedds_cpp       | rmw_fastrtps_dynamic_cpp | rmw_connext_cpp          |\n|--------------------------|--------------------------|--------------------------|--------------------------|--------------------------|--------------------------|\n| rmw_fastrtps_cpp         | :heavy_check_mark:       | :heavy_multiplication_x: | :heavy_multiplication_x: | :heavy_multiplication_x: | :heavy_multiplication_x: |\n| rmw_opensplice_cpp       | :heavy_multiplication_x: | :heavy_check_mark:       | :heavy_multiplication_x: | :heavy_multiplication_x: | :heavy_multiplication_x: |\n| rmw_cyclonedds_cpp       | :heavy_multiplication_x: | :heavy_multiplication_x: | :heavy_check_mark:       | :heavy_multiplication_x: | :heavy_multiplication_x: |\n| rmw_fastrtps_dynamic_cpp | :heavy_multiplication_x: | :heavy_multiplication_x: | :heavy_multiplication_x: | :heavy_check_mark:       | :heavy_multiplication_x: |\n| rmw_connext_cpp          | :heavy_multiplication_x: | :heavy_multiplication_x: | :heavy_multiplication_x: | :heavy_multiplication_x: | :heavy_check_mark:       |\n\n## Test 2 - Simple pub/sub\n\nIn this case we are testing one publisher and one subscriber **in different processes** sending a 1kArray at 5Hz.\nThis will allow us to evaluate additional overhead caused by a single pub/sub topic and detect leaks related to this activity.\n\nWe measure for both publisher and subscriber:\n\n - Average round trip\n - CPU usage ( readed from the filesystem )\n - Total lost messages\n - Received/Sent messages per second\n - Physical memory\n - Resident anonymous memory\n - Virtual memory\n\nAgain we plot measurement:\n - [One per-build](http://build.ros2.org/view/Rci/job/Rci__nightly-performance_ubuntu_focal_amd64/lastBuild/)\n - [Other over time](http://build.ros2.org/view/Rci/job/Rci__nightly-performance_ubuntu_focal_amd64/plot/)\n\n| Publisher/Subscriber     | rmw_fastrtps_cpp   | rmw_opensplice_cpp | rmw_cyclonedds_cpp | rmw_fastrtps_dynamic_cpp | rmw_connext_cpp    |\n|--------------------------|--------------------|--------------------|--------------------|--------------------------|--------------------|\n| rmw_fastrtps_cpp         | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:       | :heavy_check_mark: |\n| rmw_opensplice_cpp       | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:       | :heavy_check_mark: |\n| rmw_cyclonedds_cpp       | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:       | :heavy_check_mark: |\n| rmw_fastrtps_dynamic_cpp | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:       | :heavy_check_mark: |\n| rmw_connext_cpp          | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:       | :heavy_check_mark: |\n\n## Test 3 - Node spinning\n\nThis test creates one process and spins for 1 minute to evaluate ROS 2 overhead and detect obvious leaks.\n\nWe measure:\n\n - CPU usage ( readed from file system )\n - Physical memory\n - Resident anonymous memory\n - Virtual memory\n\nAgain we plot measurement:\n - [One per-build](http://build.ros2.org/view/Rci/job/Rci__nightly-performance_ubuntu_focal_amd64/lastBuild/)\n - [Other over time](http://build.ros2.org/view/Rci/job/Rci__nightly-performance_ubuntu_focal_amd64/plot/Node%20Spinning%20Results/)\n\n| DDS                      | Process 1 |\n|--------------------------|-----------|\n| rmw_fastrtps_cpp         | :heavy_check_mark:   |\n| rmw_opensplice_cpp       |  :heavy_check_mark:   |\n| rmw_cyclonedds_cpp       | :heavy_check_mark:   |\n| rmw_fastrtps_dynamic_cpp |  :heavy_check_mark:   |\n| rmw_connext_cpp          |   :heavy_check_mark:   |\n\n##  Build\n\n1.  Install ROS 2 (https://index.ros.org/doc/ros2/Installation/).\n1.  Source the ROS 2 installation (either `/opt/ros/\u003crosdistro\u003e/setup.bash` if installing from binaries, or `~/ros2_ws/install/setup.bash` if building from source):\n    1.  `source /opt/ros/\u003crosdistro\u003e/setup.bash` or `source ~/ros2_ws/install/setup.bash`\n1.  Make a new workspace and clone this repository into it:\n    1.  `mkdir -p ~/performance_ws/src`\n    1.  `cd ~/performance_ws`\n    1.  `wget https://github.com/ros2/buildfarm_perf_tests/raw/master/tools/ros2_dependencies.repos`\n    1.  `vcs import src \u003c ros2_dependencies.repos`\n    1. `rosdep install --from-path src --ignore-src`\n1.  Build the local workspace:\n    1.  `colcon build`\n1.  Source the local workspace:\n    1.  `source install/local_setup.bash`\n\n### Run\n\n```bash\ncolcon test --packages-select buildfarm_perf_tests --event-handlers console_direct+\n```\n\n## Details\n\n  ***Note: the graphs presented here are for demonstration purposes only. The data in the graphs are not meant to be accurate or current.***\n\n* Each test runs for **30** seconds with a **1k payload**, but this can be changed using CMake variables.\n - `PERF_TEST_RUNTIME`: Maximum number of seconds to run before  exiting. Zero runs forever.\n - `PERF_TEST_TOPICS`: Topics to use. These are all available topics: `Array1k`, `Array4k`, `Array16k`, `Array32k`, `Array60k`, `Array1m`, `Array2m`, `Struct16`, `Struct256`, `Struct4k`, `Struct32k`, `PointCloud512k`, `PointCloud1m`, `PointCloud2m`, `PointCloud4m`, `Range`, `NavSatFix`, `RadarDetection` and `RadarTrack`.\n\nFor example, If we want to run the test during `30` seconds using the topic `Array1k`:\n\n```bash\ncolcon build --packages-select buildfarm_perf_tests --cmake-args -DPERF_TEST_RUNTIME=\"30\" -DPERF_TEST_TOPICS=\"Array1k;Array4k\" --no-warn-unused-cli\n```\n\n* Each test produces a PNG plot of [various measures](http://build.ros2.org/view/Eci/job/Eci__nightly-performance_ubuntu_bionic_amd64/) across time, displayed in Jenkins using the image gallery plugin.\n  - These plots are displayed on the build's summary page, and are part of the output artifacts for the build.\n![](img/latency.png)\n\n* Each test also produces a couple of [aggregated measures](http://build.ros2.org/view/Eci/job/Eci__nightly-performance_ubuntu_bionic_amd64/plot/) in a small csv file, used to plot build-over-build using the Jenkins plot plugin.\n\n   - To see these plots, click the \"Plots\" link on the left side of the JOB summary (not a build summary)\n   - You should be able to click one of those points to jump to the aforementioned PNG plot that produced that aggregated point.\n![](img/agregate_latency.png)\n![](img/size.png)\n![](img/cpu.png)\n![](img/lost_packets.png)\n![](img/received_packets.png)\n![](img/sent_packets.png)\n\n## System metrics collector tool\n\nThis tool allows to create statistics based on the name of the process and arguments. This tool allows to collect the following statistics:\n\n - CPU usage (%): This information is fetched from `/proc/stat`.\n - CPU memory virtual, ResidentAnonymousMemory and physical: This information is fetched from `/proc/meminfo`.\n - Process usage (%):  This information is fetched from `/proc/\u003cpid/\u003estat`.\n - Process memory: a) virtual, b) resident anonymous memory and c) physical:  This information is fetched from  `/proc/\u003cpid/statm` and `/proc/\u003cpid/status`.\n\nThese are the argument to launch the tool:\n\n```\nros2 run buildfarm_perf_tests system_metric_collector -h\nOptions:\n  -h [ --help ]           Help screen\n  --timeout arg (=60)     Test duration\n  --log arg (=out.csv)    Log filename\n  --process_pid arg      process pid\n```\n\nA general overview of what a typical run might do, for example:\n\n1. Start process under test. For example `perf_test`\n2. Launch `system_metrics_collector` using the argument `--process_pid` with the pid of the process (in this case `perf_test`).\n3. Finally `system_metrics_collector` will fetch the data from the files describe above. If you have include the option `--log` then the data it's recorded in the file otherwise the standard output will show the reading.\n\n### Examples\n\n```bash\nros2 run performance_test perf_test -c ROS2 -t Array1k \u0026\nps -e | grep perf_test\n  8621 pts/5    00:00:01 perf_test\nros2 run buildfarm_perf_tests system_metric_collector -process_pid 8621\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fros2%2Fbuildfarm_perf_tests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fros2%2Fbuildfarm_perf_tests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fros2%2Fbuildfarm_perf_tests/lists"}