{"id":49857149,"url":"https://github.com/thc1006/reachy-mini-agent","last_synced_at":"2026-05-14T20:33:51.395Z","repository":{"id":352793749,"uuid":"1211736826","full_name":"thc1006/reachy-mini-agent","owner":"thc1006","description":"Real-time voice + vision AI agent for Reachy Mini robot. Local LLM (Ollama), Whisper STT, Kokoro/Edge TTS, WebRTC. Runs fully on your own GPU.","archived":false,"fork":false,"pushed_at":"2026-05-09T15:44:11.000Z","size":424,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-05-09T17:38:13.213Z","etag":null,"topics":["ai-agent","edge-tts","face-tracking","kokoro","llm","local-ai","multimodal","ollama","qwen","reachy-mini","real-time","robot","self-hosted","stt","tts","vision","voice-agent","voice-ai","webrtc","whisper"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thc1006.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-15T17:39:51.000Z","updated_at":"2026-05-09T15:44:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/thc1006/reachy-mini-agent","commit_stats":null,"previous_names":["thc1006/reachy-mini-agent"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/thc1006/reachy-mini-agent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thc1006%2Freachy-mini-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thc1006%2Freachy-mini-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thc1006%2Freachy-mini-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thc1006%2Freachy-mini-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thc1006","download_url":"https://codeload.github.com/thc1006/reachy-mini-agent/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thc1006%2Freachy-mini-agent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33042258,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T13:14:54.681Z","status":"online","status_checked_at":"2026-05-14T02:00:06.663Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agent","edge-tts","face-tracking","kokoro","llm","local-ai","multimodal","ollama","qwen","reachy-mini","real-time","robot","self-hosted","stt","tts","vision","voice-agent","voice-ai","webrtc","whisper"],"created_at":"2026-05-14T20:33:50.592Z","updated_at":"2026-05-14T20:33:51.390Z","avatar_url":"https://github.com/thc1006.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# reachy-mini-agent\n\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)\n[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](pyproject.toml)\n[![CI](https://github.com/thc1006/reachy-mini-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/thc1006/reachy-mini-agent/actions/workflows/ci.yml)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)\n[![Code of Conduct](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg)](CODE_OF_CONDUCT.md)\n\n**Real-time voice + vision AI agent for the [Reachy Mini](https://github.com/pollen-robotics/reachy-mini) robot.**\nRuns fully on your own GPU: local LLM (Ollama / Qwen3), Whisper STT, Kokoro or Microsoft Edge TTS, MediaPipe hand gestures, YuNet face tracking, WebRTC for audio/video — optionally bridged over Tailscale so the brain can live on a beefy server while the robot stays in the living room.\n\n\u003e **Status**: working prototype. Conversation loop, face tracking, gesture recognition, and multimodal scene understanding (vision-language model every 10 s) are all online. See [ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full pipeline.\n\n---\n\n## What it does\n\n1. **Sees you.** YuNet ONNX face detector drives head pose (`set_target` @ 50 Hz). MediaPipe hand landmarker reacts to finger counts.\n2. **Listens to you.** Robot's USB mic → WebRTC audio → local `faster-whisper` (large-v3-turbo, int8_float16) on GPU.\n3. **Thinks.** Local Ollama (Qwen3-8B by default) produces JSON with `{\"speech\": \"...\", \"actions\": [...]}`. Every 10 s, Qwen2.5-VL:7B describes the camera view and the description is injected (sandboxed) into the system prompt, so the robot can naturally reference what it sees.\n4. **Speaks.** Microsoft Ana (edge-tts, cloud) or Kokoro `af_heart` (local ONNX). Responses are cached to disk with LRU eviction — repeated phrases replay in ~20 ms.\n5. **Moves.** Head tracking + a library of expressive actions (nod, shake, greet, happy, think…).\n\n---\n\n## Quickstart\n\n```bash\ngit clone https://github.com/thc1006/reachy-mini-agent.git\ncd reachy-mini-agent\n\n# 1. Python env (uv or pip — pick one)\nuv venv \u0026\u0026 uv pip install -e . -e \".[servers,kokoro]\"\n\n# 2. Fetch perception models (~8 MB)\ncurl -L -o face_detection_yunet.onnx \\\n  https://github.com/opencv/opencv_zoo/raw/main/models/face_detection_yunet/face_detection_yunet_2023mar.onnx\ncurl -L -o hand_landmarker.task \\\n  https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task\n\n# 3. (Optional) Fetch Kokoro TTS models if you want local voice\ncurl -L -o kokoro-v1.0.onnx   https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.0/kokoro-v1.0.onnx\ncurl -L -o voices-v1.0.bin    https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.0/voices-v1.0.bin\n\n# 4. Configure\ncp .env.example .env\n#   → edit REACHY_HOST, pick TTS_ENGINE, etc.\n\n# 5. Run the three services (in three terminals, or via systemd units in systemd/)\npython src/whisper_server.py    # port 8881\npython src/kokoro_server.py     # port 8880 (skip if TTS_ENGINE=edge)\nscripts/run_robot.sh            # the main brain\n```\n\nHardware, network layout, and systemd setup — see [docs/HARDWARE.md](docs/HARDWARE.md) and [docs/SETUP.md](docs/SETUP.md).\n\n---\n\n## Architecture at a glance\n\n```\n┌──────────────────┐          ┌───────────────────────────────┐\n│   Reachy Mini    │          │   Brain host (any Linux box   │\n│   (Pollen CM4)   │          │   with a CUDA GPU ≥ 8 GB)     │\n│                  │          │                               │\n│  ┌────────────┐  │          │  ┌─────────────────────────┐  │\n│  │  daemon    │◄─┼──WebRTC──┼──┤ src/robot_brain.py      │  │\n│  │  1.6.3     │  │  + WS    │  │ ├ YuNet face detect     │  │\n│  │            │  │          │  │ ├ MediaPipe hands       │  │\n│  │  USB mic   │  │          │  │ ├ State machine         │  │\n│  │  USB spkr  │  │          │  │ └ TTS / STT / LLM glue  │  │\n│  │  camera    │  │          │  └─────────────────────────┘  │\n│  │  motors    │  │          │  ┌─ local services ────────┐  │\n│  └────────────┘  │          │  │ Ollama :11434           │  │\n└──────────────────┘          │  │   qwen3:8b (LLM)        │  │\n                              │  │   qwen2.5vl:7b (VLM)    │  │\n                              │  │ whisper_server :8881    │  │\n                              │  │ kokoro_server  :8880    │  │\n                              │  └─────────────────────────┘  │\n                              └───────────────────────────────┘\n```\n\nUnder the hood:\n\n- **P2P audio/video** via WebRTC (Opus + H264) negotiated through Pollen's Rust signaling server built into the daemon.\n- **Remote control** works over any IP reachable by the brain host — LAN, VPN (Tailscale), or localhost.\n- **Vision-language prompt injection** is sandboxed: camera descriptions are fenced in an untrusted block so the LLM doesn't obey text seen through the lens.\n\n---\n\n## Configuration\n\nEvery runtime knob is an environment variable. Copy `.env.example` → `.env` and edit:\n\n| Var | Default | Purpose |\n|---|---|---|\n| `REACHY_HOST` | `reachy-mini.local` | Daemon IP / mDNS name |\n| `LLM_MODE` | `ollama` | `ollama` / `litellm` / `claude-sdk` / `claude-cli` |\n| `OLLAMA_MODEL` | `qwen3:8b` | Any chat model on your Ollama instance |\n| `VISION_MODEL` | `qwen2.5vl:7b` | VLM for scene description |\n| `VISION_INTERVAL` | `10` | Seconds between frame captures |\n| `TTS_ENGINE` | `edge` | `edge` (Microsoft Ana, cloud) / `kokoro` (local GPU) |\n| `KOKORO_VOICE` | `af_heart` | `af_heart` / `af_nicole` / `af_sky` / `af_bella` / … |\n| `TTS_PEAK` | `0.95` | Peak-normalize each utterance to this level |\n| `TTS_GAIN` | `1.0` | Final gain multiplier (\u003e1 will clip) |\n| `TTS_CACHE_MAX_MB` | `50` | LRU cache size limit for edge-tts WAVs |\n\n---\n\n## Hardware we tested on\n\n- **Robot**: Reachy Mini (Pollen Robotics, CM4 variant) running daemon 1.6.3\n- **Brain**: Linux server with 2× RTX 3090 (48 GB total VRAM), no NVLink, Ubuntu 24.04\n- **Network**: home LAN 1 Gbit + Tailscale mesh for remote brain\n\nAlso known to run on a laptop with a single RTX 3050 (4 GB) using `tiny` Whisper + smaller LLM. See [docs/HARDWARE.md](docs/HARDWARE.md) for minimums and GPU allocation guidance.\n\n---\n\n## Why this repo exists\n\nPollen's official [`reachy_mini_conversation_app`](https://huggingface.co/spaces/pollen-robotics/reachy_mini_conversation_app) is great, but it's tied to OpenAI Realtime API and leaves your audio/video on someone else's servers. This project is:\n\n- **Fully self-hostable.** No OpenAI / Anthropic calls required (though supported as fallbacks).\n- **Designed for fast local GPUs.** Sub-second turn latency on a 3090.\n- **Transparent.** Every piece of state is a file or env var; no hidden middleware.\n\n---\n\n## Roadmap / known issues\n\n- [ ] USB audio pipeline from daemon to speaker occasionally silent after long uptime (workaround: `sudo systemctl restart reachy-mini-daemon`)\n- [ ] Voice cloning of Ana via XTTS-v2 to bring the cute Microsoft voice fully offline\n- [ ] Multi-peer WebRTC (daemon currently gives the media stream to one client at a time)\n- [ ] Windows / macOS brain-host support (currently Linux only due to GStreamer plugin paths)\n\nContributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).\n\n---\n\n## License\n\nApache 2.0 — see [LICENSE](LICENSE). Third-party notices in [NOTICE](NOTICE).\n\n---\n\n## Credits\n\n- **[Pollen Robotics](https://www.pollen-robotics.com/)** for making Reachy Mini and open-sourcing the daemon, SDK, and signaling stack.\n- **[Kokoro-ONNX](https://github.com/thewh1teagle/kokoro-onnx)** for a delightful, tiny local TTS.\n- **[faster-whisper](https://github.com/SYSTRAN/faster-whisper)** for the best open STT latency on a single GPU.\n- **[Ollama](https://ollama.com)** + **Qwen** teams for making local LLM inference frictionless.\n\nContributions, issues, and PRs welcome.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthc1006%2Freachy-mini-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthc1006%2Freachy-mini-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthc1006%2Freachy-mini-agent/lists"}