{"id":51100650,"url":"https://github.com/robotnikautomation/robotnik_sim_benchmark","last_synced_at":"2026-06-24T10:01:53.384Z","repository":{"id":315249202,"uuid":"1058719442","full_name":"RobotnikAutomation/robotnik_sim_benchmark","owner":"RobotnikAutomation","description":null,"archived":false,"fork":false,"pushed_at":"2026-02-03T11:43:14.000Z","size":181429,"stargazers_count":11,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-04T01:26:04.705Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RobotnikAutomation.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-17T13:10:04.000Z","updated_at":"2026-02-02T12:52:29.000Z","dependencies_parsed_at":"2025-10-21T15:35:00.201Z","dependency_job_id":"e0511526-ffba-4c4f-8f7e-048f1ec92d79","html_url":"https://github.com/RobotnikAutomation/robotnik_sim_benchmark","commit_stats":null,"previous_names":["robotnikautomation/robotnik_sim_benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/RobotnikAutomation/robotnik_sim_benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobotnikAutomation%2Frobotnik_sim_benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobotnikAutomation%2Frobotnik_sim_benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobotnikAutomation%2Frobotnik_sim_benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobotnikAutomation%2Frobotnik_sim_benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RobotnikAutomation","download_url":"https://codeload.github.com/RobotnikAutomation/robotnik_sim_benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobotnikAutomation%2Frobotnik_sim_benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34724756,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-24T02:00:07.484Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-24T10:01:52.603Z","updated_at":"2026-06-24T10:01:53.368Z","avatar_url":"https://github.com/RobotnikAutomation.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# robotnik_sim_benchmark\n\n## 1. Simulators\n\nClone repository\n\n```\nmkdir -p ~/benchmark_ws/src \u0026\u0026 cd ~/benchmark_ws/src/\ngit clone https://github.com/RobotnikAutomation/robotnik_sim_benchmark.git\ncd ~/benchmark_ws\ncolcon build\nsource  install/setup.bash\n```\n\nInstall `vcstool`.\n```\nsudo apt update\nsudo apt install python3-vcstool\n```\n\n\u003cdetails\u003e\n\u003csummary style=\"font-size:1.25em; font-weight:bold;\"\u003e1.1 Gazebo\u003c/summary\u003e\n\nInstall Gazebo Harmonic with ROS 2 Humble:\n\n1. Setup sources and keys.\n```\nsudo apt update\nsudo apt-get install curl lsb-release gnupg\nsudo curl https://packages.osrfoundation.org/gazebo.gpg --output /usr/share/keyrings/pkgs-osrf-archive-keyring.gpg\necho \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/pkgs-osrf-archive-keyring.gpg] http://packages.osrfoundation.org/gazebo/ubuntu-stable $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/gazebo-stable.list \u003e /dev/null\n```\n\n2. Install Gazebo Harmonic.\n```\nsudo apt-get update\nsudo apt-get install gz-harmonic\n```\n\n3. Install ROS 2 connectors for Gazebo Harmonic.\n\n\u003e **WARNING**: The package `ros-humble-ros-gzharmonic` conflicts with `ros-humble-ros-gz*`. Remove those packages before installing.\n```\nsudo apt-get update\nsudo apt-get remove ros-humble-ros-gz*\nsudo apt-get autoremove\nsudo apt-get install ros-humble-ros-gzharmonic\n```\n\nSet up workspace and install dependencies:\n\n1. Download required repositories:\n```\nmkdir -p ~/robotnik_benchmark_gazebo_ws/src\ncd ~/robotnik_benchmark_gazebo_ws/src\nvcs import --input https://raw.githubusercontent.com/RobotnikAutomation/robotnik_simulation/refs/heads/jazzy-devel/robotnik_simulation.humble.repos\n```\n\n2. Install closed-source packages:\n```\nsource /opt/ros/humble/setup.bash\nexport GZ_VERSION=harmonic\ncd ~/robotnik_benchmark_gazebo_ws/src/robotnik/robotnik_simulation/debs/\nsudo apt-get install ./ros-${ROS_DISTRO}-*.deb\n```\n\n3. Install missing dependencies:\n```\nsource /opt/ros/humble/setup.bash\nexport GZ_VERSION=harmonic\ncd ~/robotnik_benchmark_gazebo_ws\nrosdep update --rosdistro humble\nrosdep install --from-paths src --ignore-src -r -y --skip-keys=\"gz-plugin2 gz-sim8\"\n```\n\n4. Build workspace:\n```\nsource /opt/ros/humble/setup.bash\nexport GZ_VERSION=harmonic\ncd ~/robotnik_benchmark_gazebo_ws\ncolcon build --symlink-install\n```\n\nRun Gazebo Harmonic simulation:\n\n1. Spawn world:\n```\nsource ~/robotnik_benchmark_gazebo_ws/install/setup.bash\nros2 launch robotnik_gazebo_ignition spawn_world.launch.py\n```\n\n2. Spawn a robot instance (e.g., `robot_a`):\n```\nros2 launch robotnik_gazebo_ignition spawn_robot.launch.py robot_id:=robot_a robot:=rbwatcher\n```\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary style=\"font-size:1.25em; font-weight:bold;\"\u003e1.2 Isaac Sim\u003c/summary\u003e\n\nRequirements: `isaac_sim.sh` located in `$HOME/isaac_sim`\n```\nros2 launch isaac_sim isaac_sim_complete.launch.py num_robots:=1 world_file:=simple_world.usd run_rviz:=false\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary style=\"font-size:1.25em; font-weight:bold;\"\u003e1.3 O3DE\u003c/summary\u003e\n\n## 1. Prerequisites\n\n- ROS 2 Humble installed and sourced. *(Also works on ROS 2 Jazzy.)*\n- O3DE with ROS 2 Gem installed and sourced. Follow the instructions at: [O3DE](https://docs.o3de.org/docs/welcome-guide/setup/installing-linux/) and [ROS 2 Gem](https://docs.o3de.org/docs/user-guide/interactivity/robotics/project-configuration/).\n- Install required ROS 2 packages:\n```bash\nsource /opt/ros/humble/setup.bash\nsudo apt update\nsudo apt install ros-${ROS_DISTRO}-ackermann-msgs ros-${ROS_DISTRO}-control-msgs ros-${ROS_DISTRO}-nav-msgs ros-${ROS_DISTRO}-gazebo-msgs ros-${ROS_DISTRO}-xacro ros-${ROS_DISTRO}-vision-msgs\n```\n- Clone the O3DE simulation repository:\n```bash\nexport O3DE_HOME=${HOME}/o3de\nexport O3DE_EXTRAS_HOME=${HOME}/o3de-extras\nexport PROJECT_NAME=robotnik_roscon25\nexport PROJECT_PATH=${HOME}/projects/robotnik_o3de/project/${PROJECT_NAME}\ncd ${HOME}/projects\ngit clone -b humble-devel https://github.com/RobotnikAutomation/robotnik_o3de.git\ncd ${PROJECT_PATH}\n${O3DE_HOME}/scripts/o3de.sh register --project-path ${PROJECT_PATH}\n```\n- Build the O3DE project (Editor and GameLauncher):\n```bash\ncd ${PROJECT_PATH}\ncmake --build build/linux --config profile --target ${PROJECT_NAME} Editor ${PROJECT_NAME}.Assets\ncmake --build build/linux --config profile --target ${PROJECT_NAME} ${PROJECT_NAME}.Assets ${PROJECT_NAME}.GameLauncher\n```\n\n## 2. Run O3DE Simulation\n\n### 2.1 Editor\nRun the O3DE Editor with the ROS 2 project:\n```bash\ncd ${PROJECT_PATH}\n./build/linux/bin/profile/Editor\n```\n\n### 2.2 Game Launcher\nRun the O3DE Game Launcher with the ROS 2 project:\n```bash\ncd $PROJECT_PATH\n./build/linux/bin/profile/${PROJECT_NAME}.GameLauncher\n```\n\n## 3. ROS 2 Integration\n\nRun O3DE simulation:\n\n1. Spawn world:\n```\nsource /opt/ros/humble/setup.bash\nros2 launch robotnik_o3de spawn_world.launch.py\n```\n\n2. Spawn a robot instance (e.g., `robot_a`):\n```\nros2 launch robotnik_o3de spawn_robot.launch.py robot_id:=robot_a robot:=rbwatcher\n```\n\n\u003e **Note**: Spawn world launch file requires to edit where the Game Launcher binary is located in your system. Edit the `executable` parameter in `spawn_world.launch.py` accordingly.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary style=\"font-size:1.25em; font-weight:bold;\"\u003e1.4 Unity\u003c/summary\u003e\n\nInstall and run the Unity simulation with ROS 2 Humble (also tested on ROS 2 Jazzy):\n\n## 1. Prerequisites\n\n- ROS 2 Humble installed and sourced. *(Also works on Humble or Jazzy.)*\n- ROS–Unity bridge: [ROS-TCP-Endpoint](https://github.com/Unity-Technologies/ROS-TCP-Endpoint). Add to your workspace:\n  ```bash\n  cd ~/workspace/src\n  git clone  --branch=main-ros2 https://github.com/Unity-Technologies/ROS-TCP-Endpoint.git\n  ```\n- The Unity simulation archive placed at:\n  `unity_sim/worlds/unity_simulation.tar.gz`\n  *(The launch auto-extracts it on first run.)*\n\n## 2. Recommended: Run with Launch File\n\nThis launch:\n- Starts **ROS-TCP-Endpoint** listening on **0.0.0.0**.\n- Starts **RViz2** with **use_sim_time:=true** and config `rviz/robot.rviz`.\n- Runs the bootstrap script `utils/load_usd_and_run.py` which **extracts** the tar (if needed) and starts the Unity binary.\n- Optionally **spawns N robots** via `/robot[/_N]/on` services.\n\nRun:\n```bash\n# 1 robot (default)\nros2 launch unity_sim unity_complete.launch.py\n\n# 1 robot in simple_world\nros2 launch unity_sim unity_complete.launch.py world:=simple_world\n\n# 3 robots and RViz on\nros2 launch unity_sim unity_complete.launch.py robot_count:=3 run_rviz:=true\n\n# 2 robots, no RViz, in simple_world\nros2 launch unity_sim unity_complete.launch.py robot_count:=2 run_rviz:=false world:=simple_world\n\n```\n\n**Arguments**\n- `robot_count` (int, 1..5, default: 1) — spawns `robot`, `robot_2`, … `robot_5`.\n- `run_rviz` (bool, default: true) — toggles RViz2.\n- `world` (string, default: `empty_world`) — choose the world to load. Available worlds:\n  - `empty_world` — empty plane.\n  - `simple_world` — simple scene with some objects.\n\n\u003e If the launch prints an error like\n\u003e `Archive not found: .../unity_sim/worlds/unity_simulation.tar.gz`,\n\u003e make sure the archive exists at that exact path (or the extracted binary is already present somewhere under `unity_sim/worlds/`).\n\n## 3. (Optional) Manual Download/Update of the Archive\n\nIf you need to refresh the Unity build:\n\n**A) Using `gdown`**\n```bash\npip install gdown\ngdown https://drive.google.com/uc?id=1NDRtJ9zw5TGTveNKsOdgXoxGDFOHcktZ\nmv unity_simulation.tar.gz unity_sim/worlds/\n```\n\n**B) Using a web browser**\n- Open: [Unity Simulation Binary](https://drive.google.com/file/d/1NDRtJ9zw5TGTveNKsOdgXoxGDFOHcktZ/view?usp=drive_link)\n- Download `unity_simulation.tar.gz` and place it in `unity_sim/worlds/`.\n\n## 4. (Optional) Manual Run Without Launch\n\n1) Extract and run the Unity simulation:\n```bash\ncd unity_sim/worlds\ntar -xvzf unity_simulation.tar.gz\nchmod +x UnitySimulation.x86_64  # or your binary name (e.g., PI_simulation_Unity_Robotnik.x86_64)\n./UnitySimulation.x86_64         # replace with your actual filename if different\n```\n\n2) Start the ROS–Unity bridge on 0.0.0.0:\n```bash\nros2 run ros_tcp_endpoint default_server_endpoint --ros-args -p ROS_IP:=0.0.0.0\n```\n\n## 5. Notes\n\n- **RViz time**: RViz launches with `use_sim_time:=true` because the simulator publishes `/clock`.\n- **Keyboard shortcuts in the simulator**:\n  - `F1`: show performance stats.\n  - `F2`: respawn/destroy robots.\n  - Navigation: arrow keys to move, mouse wheel to zoom; to follow a robot, pick it from the bottom-right dropdown.\n- **Robot services** (provided by the simulation):\n  - Spawn: `robot_{id}/on`   (e.g., `/robot/on`, `/robot_2/on`, …)\n  - Delete: `robot_{id}/off` (e.g., `/robot/off`, `/robot_2/off`, …)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary style=\"font-size:1.25em; font-weight:bold;\"\u003e1.5 Webots\u003c/summary\u003e\n\n1. Download required repositories:\n```\nmkdir -p ~/robotnik_benchmark_webots_ws/src\ncd ~/robotnik_benchmark_webots_ws/src\ngit clone https://github.com/RobotnikAutomation/robotnik_webots.git\ngit clone https://github.com/RobotnikAutomation/robotnik_common.git\n```\n2. Install missing dependencies:\nFollow the guide to install webots from via apt: https://cyberbotics.com/doc/guide/installation-procedure#installation-on-linux\n```\nsudo apt-get install ros-humble-webots-ros2\ncd ~/robotnik_benchmark_webots_ws\nrosdep update --rosdistro humble\nrosdep install --from-paths src --ignore-src -r -y \n```\n\n3. Build workspace:\n```\nsource /opt/ros/humble/setup.bash\ncd ~/robotnik_benchmark_webots_ws\ncolcon build --symlink-install\n```\n\nRun Webots simulation:\n\n1. Spawn world:\n```\nsource ~/robotnik_benchmark_webots_ws/install/setup.bash\nros2 launch robotnik_webots spawn_world.launch.py\n```\n\n2. Spawn a robot instance (e.g., `robot_a`):\n```\nros2 launch robotnik_webots spawn_robot.launch.py robot_id:=robot_a robot:=rbwatcher\n```\n\u003c/details\u003e\n\n\n## 2. Benchmarks\n\n### 2.1 Specifications\n\nThis is the sensor configuration for 1 robot:\n\n| Sensor        | Frequency |\n|---------------|------------|\n| Front RGB camera  | 1920x1080@30fps |\n| Lidar 3D      | 10 Hz, 1800h 16v (15v deg, 360h deg)      | \n| IMU           | 200 Hz     |\n| Top RGB camera    | 1280x720@30fps     |\n\n| Simulator        | Physics |\n|---------------|------------|\n| Any  |   50 Hz (20ms)     |\n\n\n### 2.1 Benchmarking\n\nGo to benchmark repository:\n\n```\ncd ~/benchmark_ws/src/robotnik_sim_benchmark\n```\nUsage of the script:\n\n```\nusage: benchmark_simulator.py [-h] [--image_topic IMAGE_TOPIC] [--csv_file CSV_FILE] [--iterations ITERATIONS]\n                              [--category CATEGORY] [--ros_args [ROS_ARGS ...]] [--iteration_time ITERATION_TIME]\n                              simulator\n\nBenchmark simulator script\n\npositional arguments:\n  simulator             Simulator name (gazebo_harmonic, isaac_sim, webots)\n\noptions:\n  -h, --help            show this help message and exit\n  --image_topic IMAGE_TOPIC\n                        Image topic to subscribe to\n  --csv_file CSV_FILE   CSV file to store results\n  --iterations ITERATIONS\n                        Number of interations\n  --category CATEGORY   Category name for an specific set of benchmarks\n  --ros_args [ROS_ARGS ...]\n                        Additional ROS 2 args to pass to the launch files\n  --iteration_time ITERATION_TIME\n                        Time for each iteration in seconds (time to wait after receiving the first image)\n```\n\n\nThe benchmark iterations is set by `--iteration`: \n\n| Simulator        | Iterations |\n|---------------|------------|\n| Any  |   1 (default)     |\n\n\n\nThe benchmark results are saved under the category folder specified by the `--category` argument.  \n\n\u003e Changing the conditions of the simulation (e.g., number of robots or world type) must be done manually for each simulation.\n\nBy default, this repository launches **one robot in a simple world**, which corresponds to **category 4**.\n\n| Category  | Name                                     | Description \n|----|------| -----------------------------------------| \n| 0  |         No category                             | No folder\n| 1  |         one_robot_emtpy_world                   | One robot without scene results\n| 2  |         two_robot_emtpy_world                   | Two robot without scene results\n| 3  |         three_robot_emtpy_world                 | Three robot without scene results\n| 4  |         one_robot_simple_world                  | One robot in a lightweight scene results\n| 5  |         two_robot_simple_world                  | Two robot in a lightweight scene results\n| 6  |         three_robot_simple_world                | Three robot in a lightweight scene results\n| 7  |         one_robot_emtpy_world_rviz              | One robot without scene results\n| 8  |         two_robot_emtpy_world_rviz              | Two robot without scene results\n| 9  |         three_robot_emtpy_world_rviz            | Three robot without scene results\n| 10 |         one_robot_simple_world_rviz             | One robot in a lightweight scene results\n| 11 |         two_robot_simple_world_rviz             | Two robot in a lightweight scene results\n| 12 |         three_robot_simple_world_rviz           | Three robot in a lightweight scene results\n| 13 |         one_robot_emtpy_world_headless          | One robot without scene results\n| 14 |         two_robot_emtpy_world_headless          | Two robot without scene results\n| 15 |         three_robot_emtpy_world_headless        | Three robot without scene results\n| 16 |         one_robot_simple_world_headless         | One robot in a lightweight scene results\n| 17 |         two_robot_simple_world_headless         | Two robot in a lightweight scene results\n| 18 |         three_robot_simple_world_headless       | Three robot in a lightweight scene results\n| 19 |         one_robot_emtpy_world_rviz_headless     | One robot without scene results\n| 20 |         two_robot_emtpy_world_rviz_headless     | Two robot without scene results\n| 21 |         three_robot_emtpy_world_rviz_headless   | Three robot without scene results\n| 22 |         one_robot_simple_world_rviz_headless    | One robot in a lightweight scene results\n| 23 |         two_robot_simple_world_rviz_headless    | Two robot in a lightweight scene results\n| 14 |         three_robot_simple_world_rviz_headless  | Three robot in a lightweight scene results\n___________________________________________________________________________________________\n\n#### 2.1.1 Benchmarking Gazebo Harmonic\n\n\nRun `gazebo_harmonic` single benchmarks:\n\n```\npython3 ./scripts/benchmark_simulator.py --category 1 --iterations 5 --iteration_time 60 gazebo_harmonic\n```\n\nYou can set any of the categories listed above by changing the `--category` argument.\n\nRun `gazebo_harmonic` all benchmarks:\n\n```\n./scripts/run_all_benchmarks.sh -s gazebo_harmonic\n```\n\nIt will run all the categories from 1 to 12.\n\n#### 2.1.2 Benchmarking Isaac Sim\n\nRun `isaac_sim` single benchmark:\n\n```\npython3 scripts/benchmark_simulator.py isaac_sim --category 1 --iterations 5 --image_topic front_rgbd_camera/color/image_raw --ros_args num_robots:=1 world_file:=simple_world.usd\n```\n\nRun `isaac_sim` all benchmarks:\n\n```\n./scripts/run_all_benchmarks.sh -s isaac_sim\n```\n\nIt will run all the categories from 1 to 12.\n\n\n#### 2.1.3 Benchmarking O3DE\n\nRun `o3de` all benchmarks:\n\n```\n./scripts/run_all_benchmarks.sh -s o3de\n```\n\n#### 2.1.4 Benchmarking Unity\n\nRun `unity` single benchmarks:\n\n```\npython3 scripts/benchmark_simulator.py unity --image_topic /robot/top_rgbd_camera/image_raw --iterations 1 --category 5 --ros_args robot_count:=2 world:=simple_world\n```\n\nRun `unity` all benchmarks:\n\n```\n./scripts/run_all_benchmarks.sh -s unity\n```\n\n#### 2.1.5 Benchmarking Webots\n\nRun `webots` single benchmark:\n\n```\npython3 ./scripts/benchmark_simulator.py --category 1 --iterations 5 --iteration_time 60 webots\n```\n\nRun `webots` all benchmarks:\n\n```\n./scripts/run_all_benchmarks.sh -s webots\n```\n\nIt will run all the categories from 1 to 24.\n\n## 3. Results\n\nFor the performance report, see [performance_report.md](benchmarks/performance_report.md)\n\nGo to benchmark repository:\n\n```\ncd ~/benchmark_ws/src/robotnik_sim_benchmark\n```\n\nGenerate report:\n\n```\npython3 scripts/benchmark_reporter.py\n```\n\n**Note**: Report file is created in `benchmark` folder\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobotnikautomation%2Frobotnik_sim_benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frobotnikautomation%2Frobotnik_sim_benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobotnikautomation%2Frobotnik_sim_benchmark/lists"}