{"id":42964103,"url":"https://github.com/flagopen/sharerobot","last_synced_at":"2026-01-30T23:39:46.892Z","repository":{"id":284513166,"uuid":"955142244","full_name":"FlagOpen/ShareRobot","owner":"FlagOpen","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-01T12:37:39.000Z","size":8447,"stargazers_count":35,"open_issues_count":2,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-06-07T22:41:47.793Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FlagOpen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-26T07:08:59.000Z","updated_at":"2025-05-26T06:34:06.000Z","dependencies_parsed_at":"2025-03-26T09:37:19.448Z","dependency_job_id":"ed05654e-1d1d-4a61-b5ce-3b38d209ab29","html_url":"https://github.com/FlagOpen/ShareRobot","commit_stats":null,"previous_names":["flagopen/sharerobot"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FlagOpen/ShareRobot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlagOpen%2FShareRobot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlagOpen%2FShareRobot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlagOpen%2FShareRobot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlagOpen%2FShareRobot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FlagOpen","download_url":"https://codeload.github.com/FlagOpen/ShareRobot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlagOpen%2FShareRobot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28923551,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T22:32:35.345Z","status":"ssl_error","status_checked_at":"2026-01-30T22:32:31.927Z","response_time":66,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-30T23:39:46.265Z","updated_at":"2026-01-30T23:39:46.887Z","avatar_url":"https://github.com/FlagOpen.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# ShareRobot Dataset\n\n**ShareRobot**, a high-quality heterogeneous dataset that labels multi-dimensional information, including task planning, object affordance, and end-effector trajectory, effectively enhancing various robotic capabilities.\n\n- **Project Website**: [[CVPR 2025] RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete](https://superrobobrain.github.io/  \"可选标题\")\n- **Download Link**: [ShareRobot Dataset](https://huggingface.co/datasets/BAAI/ShareRobot \"可选标题\")\n\n## Overview of ShareRobot\n\n![ee709e8b-6f05-428d-abff-2578914aeb0d](./images/ee709e8b-6f05-428d-abff-2578914aeb0d.png)\n\nFor **planning**, we have 51,403 episodes and each with 30 frames. In the process of data generation, we design 5 different templates for each of the 10 question types in RoboVQA [1]. In the process of data generation, we randomly select 2 templates of each question type to generate question-answer pairs for every instance. This process transforms 51,403 instances into 1,027,990 question-answer pairs, with annotators monitoring data generation to maintain the dataset’s integrity.\n\nFor **Affordance**, we have 6,522 images and each with affordance areas aligned with an instruction. \n\nFor **Trajectory**, we have 6,870 images and each with at least 3 {x, y} coordinates aligned with an instruction.\n\n\n\n## Data Sources🌍\n\n![a608d080-665a-4ab1-bd8f-d5bd121454da](./images/a608d080-665a-4ab1-bd8f-d5bd121454da.png)\n\n**ShareRobot** dataset contains 23 original datasets from Open X-Embodiment dataset [2], 12 embodiments and 107 types of atomic tasks. \n\n\n\n### Raw Dataset for Planning\n\n| Raw Dataset                                                   | Number of Raws |\n|:-------------------------------------------------------------:| --------------:|\n| nyu_door_opening_surprising_effectiveness                     | 421            |\n| bridge                                                        | 15738          |\n| dlr_edan_shared_control_converted_externally_to_rlds          | 63             |\n| utokyo_xarm_pick_and_place_converted_externally_to_rlds       | 92             |\n| cmu_stretch                                                   | 10             |\n| asu_table_top_converted_externally_to_rlds                    | 109            |\n| dlr_sara_pour_converted_externally_to_rlds                    | 51             |\n| utokyo_xarm_bimanual_converted_externally_to_rlds             | 27             |\n| robo_set                                                      | 18164          |\n| dobbe                                                         | 5200           |\n| berkeley_autolab_ur5                                          | 882            |\n| qut_dexterous_manpulation                                     | 192            |\n| aloha_mobile                                                  | 264            |\n| dlr_sara_grid_clamp_converted_externally_to_rlds              | 40             |\n| ucsd_pick_and_place_dataset_converted_externally_to_rlds      | 569            |\n| ucsd_kitchen_dataset_converted_externally_to_rlds             | 39             |\n| jaco_play                                                     | 956            |\n| utokyo_pr2_opening_fridge_converted_externally_to_rlds        | 64             |\n| conq_hose_manipulation                                        | 56             |\n| fmb                                                           | 7836           |\n| plex_robosuite                                                | 398            |\n| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 189            |\n| viola                                                         | 44             |\n\n\n\n### Raw Dataset for Affordance\n\n| Raw Dataset                                                   | Number of Raws |\n|:-------------------------------------------------------------:| -------------:|\n| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 24             |\n| utokyo_xarm_pick_and_place_converted_externally_to_rlds       | 23             |\n| ucsd_kitchen_dataset_converted_externally_to_rlds             | 10             |\n| ucsd_pick_and_place_dataset_converted_externally_to_rlds      | 112            |\n| nyu_door_opening_surprising_effectiveness                     | 85             |\n| jaco_play                                                     | 171            |\n| bridge                                                        | 2610           |\n| utokyo_pr2_opening_fridge_converted_externally_to_rlds        | 12             |\n| asu_table_top_converted_externally_to_rlds                    | 24             |\n| viola                                                         | 1              |\n| berkeley_autolab_ur5                                          | 122            |\n| aloha_mobile                                                  | 23             |\n| conq_hose_manipulation                                        | 1              |\n| dobbe                                                         | 717            |\n| fmb                                                           | 561            |\n| plex_robosuite                                                | 13             |\n| qut_dexterous_manpulation                                     | 16             |\n| robo_set                                                      | 1979           |\n| dlr_edan_shared_control_converted_externally_to_rlds          | 18             |\n| **Summary**                                                   | 6522           |\n\n\n\n### Raw Dataset for Trajectory\n\n| Raw Dataset                                                   | Number of Raws |\n|:-------------------------------------------------------------:| -------------:|\n| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 35             |\n| utokyo_xarm_pick_and_place_converted_externally_to_rlds       | 36             |\n| ucsd_kitchen_dataset_converted_externally_to_rlds             | 19             |\n| dlr_sara_grid_clamp_converted_externally_to_rlds              | 1              |\n| ucsd_pick_and_place_dataset_converted_externally_to_rlds      | 109            |\n| nyu_door_opening_surprising_effectiveness                     | 74             |\n| jaco_play                                                     | 175            |\n| utokyo_xarm_bimanual_converted_externally_to_rlds             | 7              |\n| bridge                                                        | 2986           |\n| utokyo_pr2_opening_fridge_converted_externally_to_rlds        | 12             |\n| asu_table_top_converted_externally_to_rlds                    | 22             |\n| berkeley_autolab_ur5                                          | 164            |\n| dobbe                                                         | 759            |\n| fmb                                                           | 48             |\n| qut_dexterous_manpulation                                     | 29             |\n| robo_set                                                      | 2374           |\n| dlr_sara_pour_converted_externally_to_rlds                    | 3              |\n| dlr_edan_shared_control_converted_externally_to_rlds          | 17             |\n| **Summary**                                                   | 6870           |\n\n\n\n## Data Format\n\n### Planning\n\n![data-demo](./images/data-demo.jpg)\n\n```json\n{\n \"id\"{\n        \"id\": \"/mnt/hpfs/baaiei/jyShi/rt_frames_success/rtx_frames_success_42/62_robo_set#episode_1570\",\n        \"task\": \"Future_Prediction_Task\",\n        \"selected_step\": 3,\n        \"conversations\": [\n            {\n                \"from\": \"human\",\n                \"value\": \"\u003cimage 0-25\u003e After \u003cmove the grasped banana towards the mug\u003e, what's the most probable next event?\"\n            },\n            {\n                \"from\": \"gpt\",\n                \"value\": \"\u003cplace the banana into the mug\u003e\"\n            }\n        ],\n        \"image\": [\n            \"/path/to/image_0-25\"\n        ]\n    }        \n}\n```\n\n     \n\n\n\n### Affordance\n\n\u003c!--![2d94d985-d47e-4899-9760-c1cb8f19cd89](./images/2d94d985-d47e-4899-9760-c1cb8f19cd89.png)![a7817c0b-04b1-4a7c-9535-f9ff7801a689](./images/a7817c0b-04b1-4a7c-9535-f9ff7801a689.png)--\u003e\n\u003cdiv style=\"display: flex; gap: 10px;\"\u003e\n  \u003cimg src=\"./images/2d94d985-d47e-4899-9760-c1cb8f19cd89.png\" style=\"width: 300px;\" /\u003e\n  \u003cimg src=\"./images/a7817c0b-04b1-4a7c-9535-f9ff7801a689.png\" style=\"width: 300px;\" /\u003e\n\u003c/div\u003e\n\n```json\n{\n\n        \"id\": 2486,\n        \"meta_data\": {\n            \"original_dataset\": \"bridge\",\n            \"original_width\": 640,\n            \"original_height\": 480\n        },\n        \"instruction\": \"place the red fork to the left of the left burner\",\n        \"affordance\": {\n            \"x\": 352.87425387858815,\n            \"y\": 186.47871614766484,\n            \"width\": 19.296008229513156,\n            \"height\": 14.472006172134865\n    }\n```\n\n\n\n#### Visualize Code\n\n```python\nimport json\nimport os\nimport cv2\nimport numpy as np\n\nimg_dir = '/path/to/your/original/images/dir'\naffordance_json = '/path/to/your/affordances/json'\noutput_img_dir = '/path/to/your/visualized/images/dir'\n\nwith open(affordance_json, 'r') as f:\n    data = json.load(f)\n    for item in data:\n        filepath = os.path.join(img_dir, item['id'])\n\n        image = cv2.imread(filepath)\n        color = (255, 0, 0)\n        thickness = 2\n\n        x_min,y_min = item['affordance']['x'], item['affordance']['y']\n        x_max,y_max = item['affordance']['x']+item['affordance']['width'], item['affordance']['y']+item['affordance']['height']\n\n        # 定义矩形的四个顶点坐标\n        pts = np.array([\n            [x_min, y_min],  # 左上角\n            [x_max, y_min],  # 右上角\n            [x_max, y_max],  # 右下角\n            [x_min, y_max]   # 左下角\n        ], dtype=np.float32)\n\n        # 绘制矩形框\n        cv2.polylines(image, [pts.astype(int)], isClosed=True, color=color, thickness=thickness)\n\n        # 获取相对路径并拼接目标路径\n        relative_path = os.path.relpath(filepath, img_dir)  # 获取相对于 img_dir 的相对路径\n        output_img_path = os.path.join(output_img_dir, relative_path)  # 拼接目标路径\n\n        # 创建目标文件夹\n        output_directory = os.path.dirname(output_img_path)\n        if not os.path.exists(output_directory):\n            os.makedirs(output_directory)\n\n        # 打印调试信息\n        print(f\"Input filepath: {filepath}\")\n        print(f\"Output image path: {output_img_path}\")\n        print(f\"Output directory: {output_directory}\")\n\n        # 保存图像\n        cv2.imwrite(output_img_path, image)\n\n```\n\n\n\n\n\n### Trajectory\n\n\u003c!-- ![5b923b31-dbbf-470f-af09-5125f5b91ab0](./images/5b923b31-dbbf-470f-af09-5125f5b91ab0.png)![1af4535a-acc3-4417-ae33-675f4301f560](./images/1af4535a-acc3-4417-ae33-675f4301f560.png)--\u003e\n\u003cdiv style=\"display: flex; gap: 10px;\"\u003e\n  \u003cimg src=\"./images/5b923b31-dbbf-470f-af09-5125f5b91ab0.png\" style=\"width: 300px;\" /\u003e\n  \u003cimg src=\"./images/1af4535a-acc3-4417-ae33-675f4301f560.png\" style=\"width: 300px;\" /\u003e\n\u003c/div\u003e\n\n```json\n{\n        \"id\": 456,\n        \"meta_data\": {\n            \"original_dataset\": \"bridge\",\n            \"original_width\": 640,\n            \"original_height\": 480\n        },\n        \"instruction\": \"reach for the carrot\",\n        \"points\": [\n            [\n                265.45454545454544,\n                120.0\n            ],\n            [\n                275.1515151515152,\n                162.42424242424244\n            ],\n            [\n                280.0,\n                213.33333333333331\n            ],\n            [\n                280.0,\n                259.3939393939394\n            ]\n        ]\n    },\n```\n\n#### Visualize Code\n\n```python\nimport json\nimport os\nfrom PIL import Image, ImageDraw\n\ntrajectory_final = '/path/to/your/trajectory_json'\nimg_dir = '/path/to/your/original/images/dir'\noutput_img_dir = '/path/to/your/visualzed/images/dir'\n\nwith open(trajectory_final, 'r') as f:\n    data = json.load(f)\n    for item in data:\n        filepath = os.path.join(img_dir, item['id'])\n        points = item['points']\n\n        image = Image.open(filepath).convert(\"RGB\")  # 确保图像是 RGB 模式\n        draw = ImageDraw.Draw(image)  # 创建绘图对象\n        # 定义颜色和线宽\n        color = (255, 0, 0)  # 红色 (RGB 格式)\n        thickness = 2\n\n\n        scaled_points = [\n                (point[0], point[1])\n                for point in points\n            ]\n        # 按照顺序连接相邻的点\n        for i in range(len(scaled_points) - 1):\n            draw.line([scaled_points[i], scaled_points[i + 1]], fill=color, width=thickness)\n\n        # 获取相对路径并拼接目标路径\n        relative_path = os.path.relpath(filepath, img_dir)\n        output_img_path = os.path.join(output_img_dir, relative_path)\n\n        # 创建目标文件夹\n        output_directory = os.path.dirname(output_img_path)\n        if not os.path.exists(output_directory):\n            os.makedirs(output_directory)\n\n        # 打印调试信息\n        print(f\"Input filepath: {filepath}\")\n        print(f\"Output image path: {output_img_path}\")\n        print(f\"Output directory: {output_directory}\")\n\n        # 保存图像\n        image.save(output_img_path)\n```\n\n\n\n## Evaluation🚀\nPowered by ShareRobot dataset, RoboBrain Model achieves stunning results.🌟\n\n**Task planning capability**: The RoboBrain model trained on ShareRobot achieves a 30.2% improvement in task decomposition accuracy (BLEU-4 reached 55.05%), significantly better than existing methods;  \n\n**Affordance perception capability**: The average accuracy (AP) of object affordance area recognition is 27.1%, which is 14.6% higher than the baseline model.\n\n**Trajectory prediction capability**: End-effector trajectory prediction error reduced by 42.9% (DFD index decreased from 0.191 to 0.109);     \n\n**General capability**: In the OpenEQA benchmark, the scene understanding score surpasses general multimodal models such as GPT-4V. The RoboBrain model trained with ShareRobot did not sacrifice its general ability.\n\n![evaluation_planning](./images/evaluation_planning.png)\n\u003cdiv style=\"display: flex; gap: 10px;\"\u003e\n  \u003cimg src=\"./images/evaluation_affordance.png\" style=\"width: 400px;\" /\u003e\n  \u003cimg src=\"./images/evaluation_trajectory.png\" style=\"width: 400px;\" /\u003e\n\u003c/div\u003e\n\n\n## Reference\n\n[1] Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan,Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi,et al. Robovqa: Multimodal long-horizon reasoning forrobotics. In ICRA, pages 645–652, 2024.\n\n[2] Abby O’Neill, Abdul Rehman, Abhinav Gupta, AbhiramMaddukuri, Abhishek Gupta, Abhishek Padalkar, AbrahamLee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al.Open x-embodiment: Robotic learning datasets and rt-xmodels. arXiv preprint arXiv:2310.08864, 2023.\n\n\n\n## Citation\n```\n@article{ji2025robobrain,\n  title={RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete},\n  author={Ji, Yuheng and Tan, Huajie and Shi, Jiayu and Hao, Xiaoshuai and Zhang, Yuan and Zhang, Hengyuan and Wang, Pengwei and Zhao, Mengdi and Mu, Yao and An, Pengju and others},\n  journal={arXiv preprint arXiv:2502.21257},\n  year={2025}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflagopen%2Fsharerobot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflagopen%2Fsharerobot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflagopen%2Fsharerobot/lists"}