{"id":17133645,"url":"https://github.com/neka-nat/mylangrobot","last_synced_at":"2025-06-20T06:34:02.617Z","repository":{"id":211609100,"uuid":"714523144","full_name":"neka-nat/mylangrobot","owner":"neka-nat","description":"Language instructions to mycobot using GPT-4V","archived":false,"fork":false,"pushed_at":"2023-12-11T23:36:28.000Z","size":3693,"stargazers_count":23,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-13T08:14:08.232Z","etag":null,"topics":["chatgpt","gpt-4-vision","gpt-4-vision-preview","gpt4v","mycobot","segment-anything","whisper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/neka-nat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-11-05T05:08:21.000Z","updated_at":"2025-04-07T05:42:47.000Z","dependencies_parsed_at":"2024-02-09T09:30:48.163Z","dependency_job_id":"d7f189a4-cf25-4314-b3b3-256592d39668","html_url":"https://github.com/neka-nat/mylangrobot","commit_stats":null,"previous_names":["neka-nat/mylangrobot"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/neka-nat/mylangrobot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neka-nat%2Fmylangrobot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neka-nat%2Fmylangrobot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neka-nat%2Fmylangrobot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neka-nat%2Fmylangrobot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/neka-nat","download_url":"https://codeload.github.com/neka-nat/mylangrobot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neka-nat%2Fmylangrobot/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260896048,"owners_count":23078935,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatgpt","gpt-4-vision","gpt-4-vision-preview","gpt4v","mycobot","segment-anything","whisper"],"created_at":"2024-10-14T19:42:40.879Z","updated_at":"2025-06-20T06:33:57.604Z","avatar_url":"https://github.com/neka-nat.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  \u003cbr\u003e\n  \u003cimg height=\"300\" src=\"assets/mylangrobot_logo.png\"\u003e\n  \u003cbr\u003e\n\u003c/h1\u003e\n\n## Introduction\n\nThis is a repository for conducting interactions with robots using language. Using myCobot, pick and place tasks are achieved through verbal interactions with humans.\nBy utilizing ChatGPT for image interpretation, generating pick and place programs, and creating additional questions for ambiguous instructions, we accomplish the targeted tasks.\n\nFor image interpretation, [SoM](https://som-gpt4v.github.io/) is used. This enables more specific object designation by GPT-4V by using images that overlay numbers on segmented images.\n\n## Dependencies\n\n* openai\n* openai-whisper\n* segment-anything\n* pymycobot\n\n## Hardware\n\n* [myCobot](https://shop.elephantrobotics.com/collections/mycobot/products/mycobot-worlds-smallest-and-lightest-six-axis-collaborative-robot)\n* [suction-pump](https://shop.elephantrobotics.com/collections/suction-pumps/products/suction-pump-2-0)\n* [camera-flange](https://shop.elephantrobotics.com/collections/camera-modules/products/camera-flange-2-0)\n\n## Installation\n\n```sh\nsudo apt install portaudio19-dev\ngit clone https://github.com/neka-nat/mylangrobot.git\ncd mylangrobot\npip install -e .  # or poetry install\n```\n\n## Environment Settings\n\n```sh\ncp .env.sample .env\n# Edit .env\n# OPENAI_API_KEY=\u003cyour api key\u003e\n```\n\n## Demo\n\nInteractive demonstration with microphone.\n\n```sh\nsudo chmod 666 /dev/ttyUSBxxx\ncd scripts\npython demo.py\n```\n\nDemonstration of sending a single text command.\n\n```sh\nsudo chmod 666 /dev/ttyUSBxxx\ncd scripts\npython oneshot_demo.py --prompt チョコレートの箱取って。\n```\n\n## Results\n\n### Captured image and Annotated image\n\n![capture](assets/capture.png)\n![annotated](assets/annotated.png)\n\n### Generated program\n\n```\nmove_to_object(4)\ngrab()\nmove_to_place('drop')\nrelease()\n```\n\n### Generated motion\n\nhttps://github.com/neka-nat/mylangrobot/assets/991515/858ae25a-405a-4ad1-afc2-fbf6934adac3\n\n## Settings\n\nYou can configure the robot using the `configs/settings.yml` file.\nPlease set the connection port with the robot, the camera ID, the suction pin number, and other hardware-related settings according to your own environment.\n\n```yaml\npixel_size_on_capture_position: 0.00043  # [m/pixel]\ninterface_type: \"AUDIO\"\ncamera_id: 0\nlanguage: \"Japanese\"\nmycobot_settings:\n  urdf_path: \"../data/mycobot/mycobot.urdf\"\n  end_effector_name: \"camera_flange\"\n  port: \"/dev/ttyACM0\"\n  baud: 115200\n  default_speed: 40\n  default_z_speed: 20\n  suction_pin: 5\n  command_timeout: 5\n  use_gravity_compensation: false\n  end_effector_height: 0.065  # pump head offset\n  object_height: 0.01\n  release_height: 0.05\n  positions:\n    home: [0, 20, -130, 20, 0, 0]\n    capture: [0, 0, -30, -60, 0, -45]\n    drop: [-45, 20, -130, 20, 0, 0]\n```\n\n## Related links\n\n* [Set-of-Mark-Visual-Prompting-for-GPT-4V](https://github.com/microsoft/SoM)\n* [SoM demo](https://github.com/SkalskiP/SoM)\n* [Segment Anything](https://github.com/facebookresearch/segment-anything)\n* [PromptCraft-Robotics](https://github.com/microsoft/PromptCraft-Robotics)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneka-nat%2Fmylangrobot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneka-nat%2Fmylangrobot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneka-nat%2Fmylangrobot/lists"}