{"id":17938843,"url":"https://github.com/RobotecAI/rai","last_synced_at":"2025-03-24T10:31:43.630Z","repository":{"id":253134989,"uuid":"810385004","full_name":"RobotecAI/rai","owner":"RobotecAI","description":"RAI is an agentic framework for robotics, utilizing Langchain and ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more.","archived":false,"fork":false,"pushed_at":"2025-03-21T14:53:22.000Z","size":48792,"stargazers_count":252,"open_issues_count":67,"forks_count":29,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-21T14:59:20.055Z","etag":null,"topics":["ai","ai-agents-framework","embodied-agent","embodied-agents","embodied-ai","embodied-artificial-intelligence","generative-ai","llm","multimodal","o3de","robotec","robotics","ros2","vlm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RobotecAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-04T15:34:14.000Z","updated_at":"2025-03-20T16:58:39.000Z","dependencies_parsed_at":"2024-09-05T13:04:58.262Z","dependency_job_id":"e98431c2-5f20-4004-8433-681b2f6a60bc","html_url":"https://github.com/RobotecAI/rai","commit_stats":null,"previous_names":["robotecai/rai-private"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobotecAI%2Frai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobotecAI%2Frai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobotecAI%2Frai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobotecAI%2Frai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RobotecAI","download_url":"https://codeload.github.com/RobotecAI/rai/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245252299,"owners_count":20585007,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-agents-framework","embodied-agent","embodied-agents","embodied-ai","embodied-artificial-intelligence","generative-ai","llm","multimodal","o3de","robotec","robotics","ros2","vlm"],"created_at":"2024-10-29T00:06:23.760Z","updated_at":"2025-03-24T10:31:43.620Z","avatar_url":"https://github.com/RobotecAI.png","language":"Python","funding_links":[],"categories":["🤖 Robotics \u0026 Embodied AI","Community-Ready Frameworks","Frameworks","Embodied AI"],"sub_categories":["🟩 Development Tools 🛠️"],"readme":"# RAI\n\n\u003e [!IMPORTANT]  \n\u003e **RAI is meant for R\u0026D. Make sure to understand its limitations.**\n\nRAI is a flexible AI agent framework to develop and deploy Embodied AI features for your robots.\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n![rai-image](./docs/imgs/RAI_simple_diagram_medium.png)\n\n---\n\n[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n![GitHub Release](https://img.shields.io/github/v/release/RobotecAI/rai)\n![Contributors](https://img.shields.io/github/contributors/robotecai/rai)\n\n![Static Badge](https://img.shields.io/badge/Ubuntu-24.04-orange)\n![Static Badge](https://img.shields.io/badge/Ubuntu-22.04-orange)\n![Static Badge](https://img.shields.io/badge/Python-3.12-blue)\n![Static Badge](https://img.shields.io/badge/Python-3.10-blue)\n![Static Badge](https://img.shields.io/badge/ROS2-jazzy-blue)\n![Static Badge](https://img.shields.io/badge/ROS2-humble-blue)\n\n[![](https://dcbadge.limes.pink/api/server/https://discord.gg/3PGHgTaJSB)](https://discord.gg/3PGHgTaJSB)\n\n\u003c/div\u003e\n\n---\n\n## Overview\n\nThe RAI framework aims to:\n\n- Supply a general multi-agent system, bringing Gen AI features to your robots.\n- Add human interactivity, flexibility in problem-solving, and out-of-box AI features to existing robot stacks.\n- Provide first-class support for multi-modalities, enabling interaction with various data types.\n\n## Limitations\n\n- Limitations of LLMs and VLMs in use apply: poor spatial reasoning, hallucinations, jailbreaks, latencies, costs, ...\n- Resource use (memory, CPU) is not addressed yet.​\n- Requires connectivity and / or an edge platform.​\n\n## Table of Contents\n\n- [Features](#features)\n- [Setup](#setup)\n- [Usage examples (demos)](#simulation-demos)\n- [Debugging Assistant](#debugging-assistant)\n- [Developer resources](#developer-resources)\n\n## Features\n\n- [x] Voice interaction (both ways).\n- [x] Customizable robot identity, including constitution (ethical code) and documentation (understanding own capabilities).\n- [x] Accessing camera (\"What do you see?\"), utilizing VLMs.\n- [x] Summarizing own state through ROS logs.\n- [x] ROS 2 action calling and other interfaces. The Agent can dynamically list interfaces, check their message type, and publish.\n- [x] Integration with LangChain to abstract vendors and access convenient AI tools.\n- [x] Tasks in natural language to nav2 goals.\n- [x] [NoMaD](https://general-navigation-models.github.io/nomad/) integration.\n- [x] Tracing.\n- [x] Grounded SAM 2 integration.\n- [x] Improved Human-Robot Interaction with voice and text.\n- [x] Additional tooling such as GroundingDino.\n- [x] Support for at least 3 different AI vendors.\n- [x] Debugging assistant for ROS 2.\n- [ ] SDK for RAI developers.\n- [ ] UI for configuration to select features and tools relevant for your deployment.\n\n## Setup\n\nBefore going further, make sure you have ROS 2 (Jazzy or Humble) installed and sourced on your system.\n\n### 1. Setting up the workspace:\n\n#### 1.1 Install poetry\n\nRAI uses [Poetry](https://python-poetry.org/) for python packaging and dependency management. Install poetry (1.8+) with the following line:\n\n```bash\ncurl -sSL https://install.python-poetry.org | python3 -\n```\n\nAlternatively, you can opt to do so by following the [official docs](https://python-poetry.org/docs/#installation).\n\n#### 1.2 Clone the repository:\n\n```bash\ngit clone https://github.com/RobotecAI/rai.git\ncd rai\n```\n\n#### 1.3 Create poetry virtual environment and install dependencies:\n\n```bash\npoetry install\nrosdep install --from-paths src --ignore-src -r -y\n```\n\n\u003e [!TIP]  \n\u003e If you want to use features such as Grounded SAM 2 or NoMaD install additional dependencies:\n\u003e\n\u003e ```bash\n\u003e poetry install --with openset,nomad\n\u003e ```\n\n#### 1.4 Configure RAI\n\nRun the configuration tool to set up your vendor and other settings:\n\n```bash\npoetry run streamlit run src/rai/rai/utils/configurator.py\n```\n\n\u003e [!TIP]  \n\u003e If the web browser does not open automatically, open the URL displayed in the terminal manually.\n\n### 2. Build the project:\n\n#### 2.1 Build RAI workspace\n\n```bash\ncolcon build --symlink-install\n```\n\n#### 2.2 Activate a virtual environment:\n\n```bash\nsource ./setup_shell.sh\n```\n\n### 3. Setting up vendors\n\nRAI is vendor-agnostic. Use the configuration in [config.toml](./config.toml) to set up your vendor of choice for RAI modules.\nVendor choices for RAI and our recommendations are summarized in [Vendors Overview](docs/vendors_overview.md).\n\n\u003e We strongly recommend you to use of best-performing AI models to get the most out of RAI!\n\nPick your local solution or service provider and follow one of these guides:\n\n- **[Ollama](https://ollama.com/download)**\n- **[OpenAI](https://platform.openai.com/docs/quickstart)**\n- **[AWS Bedrock](https://console.aws.amazon.com/bedrock/home?#/overview)**\n\n## What's next?\n\nOnce you know your way around RAI, try the following challenges, with the aid the [developer guide](docs/developer_guide.md):\n\n- Run RAI on your own robot and talk to it, asking questions about what is in its documentation (and others!).\n- Implement additional tools and use them in your interaction.\n- Try a complex, multi-step task for your robot, such as going to several points to perform observations!\n\n## Debugging Assistant\n\nUse the [debugging assistant](./docs/debugging_assistant.md) to inspect ROS 2 network state and troubleshoot issues.\n\n### Simulation demos\n\nTry RAI yourself with these demos:\n| Application | Robot | Description | Docs Link |\n| ------------------------------------------ | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------- |\n| Mission and obstacle reasoning in orchards | Autonomous tractor | In a beautiful scene of a virtual orchard, RAI goes beyond obstacle detection to analyze best course of action for a given unexpected situation. | [link](docs/demos/agriculture.md) |\n| Manipulation tasks with natural language | Robot Arm (Franka Panda) | Complete flexible manipulation tasks thanks to RAI and Grounded SAM 2 | [link](docs/demos/manipulation.md) |\n| Autonomous mobile robot demo | Husarion ROSbot XL | Demonstrate RAI's interaction with an autonomous mobile robot platform for navigation and control | [link](docs/demos/rosbot_xl.md) |\n| Turtlebot demo | Turtlebot | Showcase RAI's capabilities with the popular Turtlebot platform | [link](docs/demos/turtlebot.md) |\n| Speech-to-speech interaction with autonomous taxi | Simulated car | Demonstrate RAI's speech-to-speech interaction capabilities for specifying destinations to an autonomous taxi in awsim with autoware environment | [link](docs/demos/taxi.md) |\n\n## Community\n\n### Embodied AI Community Group\n\nRAI is one of the main projects in focus of the [Embodied AI Community Group](https://github.com/ros-wg-embodied-ai). If you would like to join the next meeting, look for it in the [ROS Community Calendar](https://calendar.google.com/calendar/u/0/embed?src=c_3fc5c4d6ece9d80d49f136c1dcd54d7f44e1acefdbe87228c92ff268e85e2ea0@group.calendar.google.com\u0026ctz=Etc/UTC).\n\n### Publicity\n\n- A talk about [RAI at ROSCon 2024](https://vimeo.com/1026029511).\n\n### RAI Q\u0026A\n\nPlease take a look at [Q\u0026A](https://github.com/RobotecAI/rai/discussions/categories/q-a).\n\n### Developer Resources\n\nSee our [Developer Guide](docs/developer_guide.md) for a deeper dive into RAI, including instructions on creating a configuration specifically for your robot.\n\n### Contributing\n\nYou are welcome to contribute to RAI! Please see our [Contribution Guide](CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRobotecAI%2Frai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRobotecAI%2Frai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRobotecAI%2Frai/lists"}