{"id":13345747,"url":"https://github.com/microsoft/PromptCraft-Robotics","last_synced_at":"2025-03-12T08:31:02.565Z","repository":{"id":78059817,"uuid":"598872262","full_name":"microsoft/PromptCraft-Robotics","owner":"microsoft","description":"Community for applying LLMs to robotics and a robot simulator with ChatGPT integration","archived":false,"fork":false,"pushed_at":"2024-01-20T20:38:07.000Z","size":18921,"stargazers_count":1969,"open_issues_count":10,"forks_count":212,"subscribers_count":40,"default_branch":"main","last_synced_at":"2025-03-08T17:27:43.490Z","etag":null,"topics":["airsim","chatgpt","llm","prompt-engineering","robotics","simulation"],"latest_commit_sha":null,"homepage":"https://aka.ms/ChatGPT-Robotics","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microsoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-08T01:06:33.000Z","updated_at":"2025-03-08T08:07:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"479f685a-2d45-4008-bde5-0be90570ccee","html_url":"https://github.com/microsoft/PromptCraft-Robotics","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FPromptCraft-Robotics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FPromptCraft-Robotics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FPromptCraft-Robotics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FPromptCraft-Robotics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microsoft","download_url":"https://codeload.github.com/microsoft/PromptCraft-Robotics/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242883733,"owners_count":20200978,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airsim","chatgpt","llm","prompt-engineering","robotics","simulation"],"created_at":"2024-07-29T20:01:07.216Z","updated_at":"2025-03-12T08:31:02.508Z","avatar_url":"https://github.com/microsoft.png","language":"Python","readme":"# PromptCraft-Robotics\n\nThe PromptCraft-Robotics repository serves as a community for people to test and share interesting prompting examples for large language models (LLMs) within the robotics domain. We also provide a sample [robotics simulator](https://github.com/microsoft/PromptCraft-Robotics/tree/main/chatgpt_airsim) (built on Microsoft AirSim) with ChatGPT integration for users to get started.\n\nWe currently focus on OpenAI's [ChatGPT](https://openai.com/blog/chatgpt/), but we also welcome examples from other LLMs (for example open-sourced models or others with API access such as [GPT-3](https://openai.com/api/) and Codex).\n\nUsers can contribute to this repository by submitting interesting prompt examples to the [Discussions](https://github.com/microsoft/PromptCraft-Robotics/discussions) section of this repository. A prompt can be submitted within different robotics categories such as [Manipulation](https://github.com/microsoft/PromptCraft-Robotics/discussions/categories/llm-manipulation), [Home Robotics](https://github.com/microsoft/PromptCraft-Robotics/discussions/categories/llm-home-robots), [Physical Reasoning](https://github.com/microsoft/PromptCraft-Robotics/discussions/categories/llm-physical-reasoning), among many others.\nOnce submitted, the prompt will be reviewed by the community (upvote your favorites!) and added to the repository by a team of admins if it is deemed interesting and useful.\nWe encourage users to submit prompts that are interesting, fun, or useful. We also encourage users to submit prompts that are not necessarily \"correct\" or \"optimal\" but are interesting nonetheless.\n\nWe encourage prompt submissions formatted as markdown, so that they can be easily transferred to the main repository. Please specify which LLM you used, and if possible provide other visuals of the model in action such as videos and pictures.\n\n## Paper, videos and citations\n\nBlog post: \u003ca href=\"https://aka.ms/ChatGPT-Robotics\" target=\"_blank\"\u003eaka.ms/ChatGPT-Robotics\u003c/a\u003e\n\nPaper: \u003ca href=\"https://www.microsoft.com/en-us/research/uploads/prod/2023/02/ChatGPT___Robotics.pdf\" target=\"_blank\"\u003eChatGPT for Robotics: Design Principles and Model Abilities\n\nVideo: \u003ca href=\"https://youtu.be/NYd0QcZcS6Q\" target=\"_blank\"\u003ehttps://youtu.be/NYd0QcZcS6Q\u003c/a\u003e\n\nIf you use this repository in your research, please cite the following paper:\n\n```\n@techreport{vemprala2023chatgpt,\nauthor = {Vemprala, Sai and Bonatti, Rogerio and Bucker, Arthur and Kapoor, Ashish},\ntitle = {ChatGPT for Robotics: Design Principles and Model Abilities},\ninstitution = {Microsoft},\nyear = {2023},\nmonth = {February},\nurl = {https://www.microsoft.com/en-us/research/publication/chatgpt-for-robotics-design-principles-and-model-abilities/},\nnumber = {MSR-TR-2023-8},\n}\n```\n\n## ChatGPT Prompting Guides \u0026 Examples\n\nThe list below contains links to the different robotics categories and their corresponding prompt examples. We welcome contributions to this repository to add more robotics categories and examples. Please submit prompt examples to the [Discussions](https://github.com/microsoft/PromptCraft-Robotics/discussions) page, or submit a pull request with your category and examples.\n\n* Embodied agent \n  * [ChatGPT - Habitat, closed loop object navigation 1](examples/embodied_agents/visual_language_navigation_1.md)\n  * [ChatGPT - Habitat, closed loop object navigation 2](examples/embodied_agents/visual_language_navigation_2.md)\n  * [ChatGPT - AirSim, object navigation using RGBD](examples/embodied_agents/airsim_objectnavigation.md)\n* Aerial robotics\n  * [ChatGPT - Real robot: Tello deployment](examples/aerial_robotics/tello_example.md) | [Video Link](https://youtu.be/i5wZJFb4dyA)\n  * [ChatGPT - AirSim turbine Inspection](examples/aerial_robotics/airsim_turbine_inspection.md) | [Video Link](https://youtu.be/38lA3U2J43w)\n  * [ChatGPT - AirSim solar panel Inspection](examples/aerial_robotics/airsim_solarpanel_inspection.md)\n  * [ChatGPT - AirSim obstacle avoidance](examples/aerial_robotics/airsim_obstacleavoidance.md) | [Video Link](https://youtu.be/Vn6NapLlHPE)\n* Manipulation\n  * [ChatGPT - Real robot: Picking, stacking, and building the MSFT logo](examples/manipulation/pick_stack_msft_logo.md) | [Video Link](https://youtu.be/wLOChUtdqoA)\n  * [ChatGPT - Manipulation tasks](examples/manipulation/manipulation_zeroshot.md)\n* Spatial-temporal reasoning\n  * [ChatGPT - Visual servoing with basketball](examples/spatial_temporal_reasoning/visual_servoing_basketball.md)\n\n\n## ChatGPT + Robotics Simulator\n\nWe provice a sample [AirSim](https://github.com/microsoft/AirSim) environment for users to test their ChatGPT prompts. The environment is a binary containing a sample inspection environment with assets such as wind turbines, electric towers, solar panels etc. The environment comes with a drone and interfaces with ChatGPT such that users can easily send commands in natural language. [[Simulator Link]](chatgpt_airsim/README.md)\n\nWe welcome contributions to this repository to add more robotics simulators and environments. Please submit a pull request with your simulator and environment.\n\n## Related resources\n\nBeyond the prompt examples here, we leave useful and related links to the use of large language models below:\n\n* [Read about the OpenAI APIs](https://openai.com/api/)\n* [Azure OpenAI service](https://azure.microsoft.com/en-us/products/cognitive-services/openai-service)\n* [OPT language model](https://huggingface.co/docs/transformers/model_doc/opt)\n\n## Contributing\n\nThis project welcomes contributions and suggestions.  Most contributions require you to agree to a\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\nthe rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.\n\nWhen you submit a pull request, a CLA bot will automatically determine whether you need to provide\na CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\nprovided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n\n## Trademarks\n\nThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft \ntrademarks or logos is subject to and must follow \n[Microsoft's Trademark \u0026 Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).\nUse of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.\nAny use of third-party trademarks or logos are subject to those third-party's policies.\n","funding_links":[],"categories":["Python","Uncategorized","Documentation and examples"],"sub_categories":["Uncategorized","Information about prompt engineering"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2FPromptCraft-Robotics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrosoft%2FPromptCraft-Robotics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2FPromptCraft-Robotics/lists"}