{"id":17938872,"url":"https://github.com/ycheng517/tabletop-handybot","last_synced_at":"2025-03-24T10:31:44.430Z","repository":{"id":240235263,"uuid":"793546445","full_name":"ycheng517/tabletop-handybot","owner":"ycheng517","description":"A low-cost AI powered robotic arm assistant that listens to your voice commands and can carry out a variety of tabletop tasks.","archived":false,"fork":false,"pushed_at":"2024-05-31T12:20:59.000Z","size":1308,"stargazers_count":65,"open_issues_count":0,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-18T00:11:17.434Z","etag":null,"topics":["ai","chatgpt","computer-vision","llm","robot-assistant","robotic-arm","robotics","ros2"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ycheng517.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-29T12:30:26.000Z","updated_at":"2025-03-16T09:39:24.000Z","dependencies_parsed_at":"2024-05-30T04:13:32.758Z","dependency_job_id":"b2a60805-c13d-4c63-8b6b-dba063030a16","html_url":"https://github.com/ycheng517/tabletop-handybot","commit_stats":null,"previous_names":["ycheng517/tabletop-handybot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ycheng517%2Ftabletop-handybot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ycheng517%2Ftabletop-handybot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ycheng517%2Ftabletop-handybot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ycheng517%2Ftabletop-handybot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ycheng517","download_url":"https://codeload.github.com/ycheng517/tabletop-handybot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245252307,"owners_count":20585008,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","chatgpt","computer-vision","llm","robot-assistant","robotic-arm","robotics","ros2"],"created_at":"2024-10-29T00:06:24.255Z","updated_at":"2025-03-24T10:31:44.068Z","avatar_url":"https://github.com/ycheng517.png","language":"Python","funding_links":[],"categories":["Research-Grade Frameworks"],"sub_categories":[],"readme":"# Tabletop HandyBot\n\nA low-cost AI powered robotic arm assistant that listens to your voice commands\nand can carry out a variety of tabletop tasks. Currently it can pick and place\narbitrary objects presented in front of it. The total BOM is around $2300 USD.\n\n## Video Demo\n\n\u003cdiv align=\"center\"\u003e\n\n|                                       Rolling Dice                                       |                                        Picking Markers                                         |\n| :--------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------: |\n| [![Roll Dice](./assets/roll_dice_thumbnail.png)](https://youtube.com/shorts/xAmPpvCZZU4) | [![Pick Markers](./assets/pick_markers_thumbnail.png)](https://youtube.com/shorts/fsCkS7HvgGo) |\n\n\u003c/div\u003e\n\n## System Architecture\n\n![System Architecture](./assets/architecture_diagram.png)\n\n### Technology Used\n\n- [AR4 Robot](https://www.anninrobotics.com/): a $2000 robot arm that can lift 5lb dumbbells\n  - [ar4_ros_driver](https://github.com/ycheng517/ar4_ros_driver): ROS driver for the arm\n- [Realsense D435](https://store.intelrealsense.com/buy-intel-realsense-depth-camera-d435.html?_ga=2.156205052.639062113.1715864879-1357777019.1715864879): a reliable $300 RGBD camera\n  - [realsense-ros](https://github.com/IntelRealSense/realsense-ros): ROS wrapper for the camera\n- [Whisper Mic](https://github.com/mallorbc/whisper_mic): Convenient library for using OpenAI Whisper with a microphone\n- [Grounding DINO](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/GroundingDINO) A SOTA zero-shot object detector that can detect any object.\n- [Segment Anything](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/segment_anything): A SOTA zero-shot object segmentation model that can segment any object.\n- [OpenAI Assistants](https://platform.openai.com/docs/assistants/overview): an API that calls ChatGPT with specific instructions and function calls\n\n## Setup\n\n### Pre-requisites\n\nYou should have [ROS 2 Iron](https://docs.ros.org/en/iron/index.html) installed on Ubuntu 22.04.\n\nThe follow hardware is is used. The use of other hardware will require adapting the codebase.\n\n- [AR4 robot arm](https://www.anninrobotics.com/) with [AR servo gripper](https://www.anninrobotics.com/product-page/servo-gripper-parts-kit)\n- [Intel RealSense D435 Depth Camera](https://www.intelrealsense.com/depth-camera-d435/)\n\nYou should be able to run [ar4_ros_driver](https://github.com/ycheng517/ar4_ros_driver)\nwith the gripper, and successfully perform hand-eye calibration.\n\n### Install\n\nImport dependent repos\n\n```bash\nvcs import . --input tabletop-handybot.repos\n```\n\nCreate a virtual environment, i.e.\n\n```bash\npyenv virtualenv 3.10.12 handybot\n```\n\nGo to the `./Grounded-Segment-Anything/Grounded-Segment-Anything/` sub-directory\nand setup [Grounded-Segment-Anything](https://github.com/IDEA-Research/Grounded-Segment-Anything)\nin the virtualenv. Ensure you can run the [grounded_sam.ipynb](https://github.com/IDEA-Research/Grounded-Segment-Anything/blob/main/grounded_sam.ipynb)\nnotebook.\n\nInstall Python dependencies not in the ROS Index\n\n```bash\npip install -r requirements.txt\n```\n\nBuild and source the project\n\n```bash\ncolcon build\nsource install/setup.bash\n```\n\n### Run\n\nLaunch the AR4\n\n```bash\nros2 launch ar_hardware_interface ar_hardware.launch.py calibrate:=True include_gripper:=True\n```\n\nLaunch all other programs\n\n```bash\nros2 launch tabletop_handybot run.launch.py\n```\n\nIf all things are looking good, then you can try publishing a prompt to the\n`/prompt` topic for the robot to execute, i.e:\n\n```bash\nros2 topic pub --once /prompt std_msgs/msg/String \"data: 'put the marker in the container'\"\n```\n\nIf you have a microphone attached to the computer, you can publish a message\nto the `/listen` topic, and then say your prompt.\n\n```bash\nros2 topic pub --once /listen std_msgs/msg/Empty \"{}\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fycheng517%2Ftabletop-handybot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fycheng517%2Ftabletop-handybot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fycheng517%2Ftabletop-handybot/lists"}