Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/TEN-framework/TEN-Agent
TEN Agent is the world’s first real-time multimodal agent integrated with the OpenAI Realtime API, RTC, and features weather checks, web search, vision, and RAG capabilities.
https://github.com/TEN-framework/TEN-Agent
agent ai asr cpp gemini golang gpt-4 gpt-4o llm low-latency multimodal nextjs14 openai python rag real-time realtime tts vision voice-assistant
Last synced: 15 days ago
JSON representation
TEN Agent is the world’s first real-time multimodal agent integrated with the OpenAI Realtime API, RTC, and features weather checks, web search, vision, and RAG capabilities.
- Host: GitHub
- URL: https://github.com/TEN-framework/TEN-Agent
- Owner: TEN-framework
- License: apache-2.0
- Created: 2024-06-19T14:26:15.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2024-10-19T09:40:37.000Z (18 days ago)
- Last Synced: 2024-10-21T02:32:06.140Z (16 days ago)
- Topics: agent, ai, asr, cpp, gemini, golang, gpt-4, gpt-4o, llm, low-latency, multimodal, nextjs14, openai, python, rag, real-time, realtime, tts, vision, voice-assistant
- Language: Python
- Homepage: https://theTen.ai
- Size: 29 MB
- Stars: 1,025
- Watchers: 20
- Forks: 125
- Open Issues: 34
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: docs/code-of-conduct/code-of-conduct.md
Awesome Lists containing this project
- ai-game-devtools - TEN Agent - time multimodal agent integrated with the OpenAI Realtime API, RTC, and features weather checks, web search, vision, and RAG capabilities. | | | Agent | (<span id="game">Game (Agent)</span> / <span id="tool">Tool (AI LLM)</span>)
README
[](https://twitter.com/intent/follow?screen_name=TenFramework)
[](https://github.com/TEN-framework/ten-agent/discussions/)
[](https://github.com/TEN-framework/ten-agent/graphs/commit-activity)
[](https://github.com/TEN-framework/ten-agent/issues)
[](https://github.com/TEN-framework/ten-agent/pulls)
[](https://github.com/TEN-framework/ten-agent/blob/main/LICENSE)
[](https://discord.gg/VnPftUzAMJ)
[](https://GitHub.com/TEN-framework/ten-agent/watchers/?WT.mc_id=academic-105485-koreyst)
[](https://GitHub.com/TEN-framework/ten-agent/network/?WT.mc_id=academic-105485-koreyst)
[](https://GitHub.com/TEN-framework/ten-agent/stargazers/?WT.mc_id=academic-105485-koreyst)
[Getting Started](https://doc.theten.ai/ten-agent/getting_started)
•
[Create Extensions](https://doc.theten.ai/ten-agent/create_a_hello_world_extension)
•
[TEN Framework Repository](https://github.com/TEN-framework/ten_framework)
🎉 TEN Agent with OpenAI Realtime API and RTC
OpenAI Realtime API provides ultra-low latency, and RTC offers AI noise suppression. TEN Agent has both. Feel free to try it out at [agent.theten.ai](https://agent.theten.ai).
TEN Agent Features
Aside from being world's the first agent to integrate the OpenAI Realtime API and RTC, TEN Agent also offers the following features:
1. **High-Performance Real-Time Multimodal Interactions**:
Offers high-performance, low-latency solutions for complex audio-visual AI applications.
2. **Multi-Language and Multi-Platform Support** :
Supports extension development in C++, Go, Python, etc. Runs on Windows, Mac, Linux, and mobile devices.
3. **Edge-Cloud Integration**:
Flexibly combines edge and cloud-deployed extensions, balancing privacy, cost, and performance.
4. **Flexibility Beyond Model Limitations**:
Easily build complex AI applications through simple drag-and-drop programming, integrating audio-visual tools, databases, RAG, and more.
5. **Real-Time Agent State Management**:
Manages and adjusts agent behavior in real-time for dynamic responsiveness.
Stay Tuned
Before we get started, be sure to star our repository and get instant notifications for all new releases!
How to build TEN Agent locally
### Prerequisites
#### Keys
- Agora [ App ID ](https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project) and [ App Certificate ](https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project)(certificate only required if enabled in the Agora Console)
- Azure [SST(ASR)](https://azure.microsoft.com/en-us/products/ai-services/speech-to-text) and [TTS](https://azure.microsoft.com/en-us/products/ai-services/text-to-speech) API keys
- [OpenAI](https://openai.com/index/openai-api/) API key
#### Installation
- [Docker](https://www.docker.com/) / [Docker Compose](https://docs.docker.com/compose/)
- [Node.js(LTS) v18](https://nodejs.org/en)
#### Minimum system requirements
- CPU >= 2 Core
- RAM >= 4 GB
#### Docker setting on Apple Silicon
You will need to uncheck "Use Rosetta for x86_64/amd64 emulation on Apple Silicon" option for Docker if you are on Apple Silicon, otherwise the server is not going to work.
### Next step
#### 1. Modify config files
In the root of the project, use `cp` command to create `.env` from the example.
It will be used to store information for `docker compose` later.
```bash
cp ./.env.example ./.env
```
#### 2. Setup API keys
Open the `.env` file and fill in the `keys` and `regions`.
```bash
# Agora App ID
# Agora App Certificate(only required if enabled in the Agora Console)
AGORA_APP_ID=
AGORA_APP_CERTIFICATE=
# Azure STT key and region
AZURE_STT_KEY=
AZURE_STT_REGION=
# Azure TTS key and region
AZURE_TTS_KEY=
AZURE_TTS_REGION=
# OpenAI API key
OPENAI_API_KEY=
```
#### 3. Start agent development containers
In the same directory, run the `docker compose up` command to compose containers:
```bash
docker compose up
```
#### 4. Enter container and build agent
Open up a separate terminal window, enter the container and build the agent:
```bash
docker exec -it ten_agent_dev bash
make build
```
#### 5. Start the server
Once the build is done, `make run-server` on port `8080`:
```bash
make run-server
```
### Finish and verify
#### TEN Agent
Open up [localhost:3000]( http://localhost:3000 ) in browser to play the TEN Agent.
#### TEN Agent with OpenAI Realtime API
Open up [localhost:3002]( http://localhost:3002 ) in browser to play the TEN Agent with OpenAI Realtime API.
#### TEN Graph Designer
Open up another tab go to [localhost:3001]( http://localhost:3001 ), and use Graph Designer to create, connect and edit extensions on canvas.
Join Community
- [Discord](https://discord.gg/VnPftUzAMJ): Ideal for sharing your applications and engaging with the community.
- [GitHub Discussion](https://github.com/TEN-framework/ten-agent/discussions): Perfect for providing feedback and asking questions.
- [GitHub Issues](https://github.com/TEN-framework/ten-agent/issues): Best for reporting bugs and proposing new features. Refer to our [contribution guidelines](./docs/code-of-conduct/contributing.md) for more details.
- [X (formerly Twitter)](https://img.shields.io/twitter/follow/TenFramework?logo=X&color=%20%23f5f5f5): Great for sharing your agents and interacting with the community.
Code Contributors
[](https://github.com/TEN-framework/ten-agent/graphs/contributors)
Contribution Guidelines
Contributions are welcome! Please read the [contribution guidelines](./docs/code-of-conduct/contributing.md) first.
License
This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.