{"id":35849914,"url":"https://github.com/steffenpharai/zip","last_synced_at":"2026-01-08T07:04:46.443Z","repository":{"id":331372914,"uuid":"1123447376","full_name":"steffenpharai/Zip","owner":"steffenpharai","description":"ZIP Robot - AI-powered robot with voice control and autonomous capabilities. ELEGOO Smart Robot Car + Arduino + Next.js HUD + OpenAI.","archived":false,"fork":false,"pushed_at":"2026-01-05T09:49:43.000Z","size":267942,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-01-07T03:52:42.651Z","etag":null,"topics":["ai-assistant","elegoo-robot-car","hud","jarvis","langchain","nextjs","openai","robot-control","typescript","voice-assistant"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/steffenpharai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-26T22:29:00.000Z","updated_at":"2026-01-05T01:44:04.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/steffenpharai/Zip","commit_stats":null,"previous_names":["steffenpharai/zip"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/steffenpharai/Zip","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steffenpharai%2FZip","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steffenpharai%2FZip/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steffenpharai%2FZip/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steffenpharai%2FZip/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/steffenpharai","download_url":"https://codeload.github.com/steffenpharai/Zip/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steffenpharai%2FZip/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28242464,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2026-01-08T02:00:06.591Z","response_time":241,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-assistant","elegoo-robot-car","hud","jarvis","langchain","nextjs","openai","robot-control","typescript","voice-assistant"],"created_at":"2026-01-08T07:04:18.755Z","updated_at":"2026-01-08T07:04:46.427Z","avatar_url":"https://github.com/steffenpharai.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ZIP Robot\r\n\r\nZIP is an AI-powered robot built on the ELEGOO Smart Robot Car V4.0 platform, featuring voice control, computer vision, and autonomous capabilities through a Jarvis-style HUD interface.\r\n\r\n## Overview\r\n\r\nZIP combines custom Arduino firmware, a WebSocket bridge server, and a Next.js HUD interface to create an intelligent robot that can be controlled through voice commands, text chat, or autonomous AI orchestration.\r\n\r\n```\r\n┌─────────────────────────────────────────────────────────────────┐\r\n│                         ZIP HUD                                  │\r\n│              (Next.js + OpenAI Realtime Voice)                  │\r\n│                    http://localhost:3000                         │\r\n└─────────────────────────┬───────────────────────────────────────┘\r\n                          │ WebSocket\r\n┌─────────────────────────▼───────────────────────────────────────┐\r\n│                      Robot Bridge                                │\r\n│                  ws://localhost:8765/robot                       │\r\n│              (WebSocket ↔ Serial Translation)                    │\r\n└─────────────────────────┬───────────────────────────────────────┘\r\n                          │ Serial (115200 baud)\r\n┌─────────────────────────▼───────────────────────────────────────┐\r\n│                     ZIP Robot Firmware                           │\r\n│              (Arduino UNO + ELEGOO Shield)                       │\r\n│         Motors • Sensors • Servos • IMU • Camera                 │\r\n└─────────────────────────────────────────────────────────────────┘\r\n```\r\n\r\n## Features\r\n\r\n### Robot Hardware\r\n- **ELEGOO Smart Robot Car V4.0** base platform\r\n- **Arduino UNO R3** with SmartCar-Shield-v1.1\r\n- **TB6612FNG** dual H-bridge motor driver\r\n- **MPU6050** IMU for orientation sensing\r\n- **Ultrasonic sensor** for distance measurement\r\n- **Line tracking sensors** for navigation\r\n- **Servo-mounted sensor head** for scanning\r\n- **ESP32-CAM** module for robot vision (in development)\r\n\r\n### HUD Interface\r\n- **🎤 Realtime Voice Interface**: Low-latency voice control via OpenAI Realtime WebRTC with barge-in support\r\n- **🤖 AI Orchestration**: Intelligent request routing with research and workflow sub-graphs\r\n- **💬 Multi-Modal Interaction**: Voice, text chat, and vision (webcam analysis) support\r\n- **🧠 Memory System**: User-controlled pinned memory with natural language commands\r\n- **📊 Real-time Telemetry**: Motor states, sensor readings, battery voltage\r\n- **🎮 Motion Control**: Velocity/turn rate joystick and presets\r\n- **📡 Sensor Display**: Live ultrasonic, line sensor, and IMU data\r\n- **🔌 Serial Console**: Direct firmware communication\r\n\r\n### Additional Capabilities\r\n- **📚 Document Intelligence**: PDF ingestion, vector search, and Q\u0026A with citations\r\n- **🌐 Web Research**: Automated research pipeline with source validation and citations\r\n- **📝 Notes \u0026 Timers**: Full CRUD operations for notes and server-side timer reminders\r\n- **🖨️ 3D Printer Control**: Moonraker/Klipper printer integration (11 tools)\r\n- **🔒 Security**: Permission-based tool access, input validation, and audit logging\r\n- **📊 Observability**: Comprehensive tracing, audit logs, and request tracking\r\n\r\n## Documentation\r\n\r\n| Document | Description |\r\n|----------|-------------|\r\n| [Robot Firmware](robot/firmware/zip_robot_uno/README.md) | Arduino firmware documentation |\r\n| [Robot Bridge](robot/bridge/zip-robot-bridge/README.md) | WebSocket bridge server |\r\n| [Motion Control](robot/ELEGOO_MOTION_CONTROL.md) | Motion system details |\r\n| [ESP32-CAM](robot/firmware/zip_esp32_cam/README.md) | Camera module firmware |\r\n| [Docker Guide](docs/docker/README.md) | Docker deployment documentation |\r\n| [Windows Serial Port](docs/docker/WINDOWS_SERIAL_PORT.md) | Serial port setup on Windows |\r\n| [Agent Guide](AGENT_GUIDE.md) | Quick reference for AI agents |\r\n| [Agent Onboarding](docs/agents/README.md) | Getting started for agents |\r\n| [Architecture Guide](docs/agents/architecture.md) | System architecture details |\r\n| [Development Workflow](docs/agents/development-workflow.md) | How to work on tasks |\r\n\r\n## Quick Start\r\n\r\n### 1. Flash the Robot Firmware\r\n\r\n```bash\r\ncd robot/firmware/zip_robot_uno\r\npio run -t upload\r\n```\r\n\r\n### 2. Start the Bridge Server\r\n\r\n```bash\r\ncd robot/bridge/zip-robot-bridge\r\nnpm install\r\nnpm run dev\r\n```\r\n\r\n### 3. Start the HUD\r\n\r\n```bash\r\nnpm install\r\ncp example-env .env\r\n# Add your OPENAI_API_KEY to .env\r\nnpm run dev\r\n```\r\n\r\n### 4. Open the HUD\r\n\r\nNavigate to [http://localhost:3000](http://localhost:3000)\r\n\r\n## Local Development\r\n\r\nZIP can be run as multiple independent services locally. This setup is useful for development and debugging.\r\n\r\n### Prerequisites\r\n\r\n- Node.js 18+\r\n- PlatformIO (for firmware)\r\n- npm or yarn\r\n- `.env` file configured (see [Environment Variables](#environment-variables))\r\n- Serial port available or `LOOPBACK_MODE=true` for testing\r\n\r\n### Service Architecture\r\n\r\n1. **zip-app** (Next.js HUD application)\r\n   - Port: 3000\r\n   - Health: `GET /api/health`\r\n   - Hot reloading in development\r\n\r\n2. **robot-bridge** (Robot communication bridge)\r\n   - WebSocket: 8765\r\n   - HTTP: 8766\r\n   - Health: `GET /health`\r\n   - Serial port access for robot communication\r\n\r\n3. **zip-robot-firmware** (Arduino firmware)\r\n   - Serial: 115200 baud\r\n   - Protocol: ELEGOO-style JSON\r\n\r\n### Running Services\r\n\r\n**Terminal 1 - ZIP HUD:**\r\n```bash\r\nnpm run dev:local\r\n```\r\n\r\n**Terminal 2 - Robot Bridge:**\r\n```bash\r\nnpm run dev:bridge\r\n```\r\n\r\n**Firmware - Upload once:**\r\n```bash\r\ncd robot/firmware/zip_robot_uno\r\npio run -t upload\r\n```\r\n\r\n### Available Scripts\r\n\r\n#### ZIP HUD (Root Directory)\r\n\r\n| Script | Description |\r\n|--------|-------------|\r\n| `npm run dev:local` | Start ZIP HUD in development mode |\r\n| `npm run dev:bridge` | Start robot bridge in development mode |\r\n| `npm run build:local` | Build ZIP HUD for production |\r\n| `npm run build:bridge` | Build robot bridge for production |\r\n| `npm run test:local` | Run full test suite |\r\n| `npm run test:health` | Health check both services |\r\n| `npm run test:integration` | Integration tests between services |\r\n| `npm run typecheck` | TypeScript type checking |\r\n| `npm run lint` | Run ESLint |\r\n| `npm run test:e2e` | Run Playwright E2E tests |\r\n\r\n#### Robot Bridge (`robot/bridge/zip-robot-bridge/`)\r\n\r\n| Script | Description |\r\n|--------|-------------|\r\n| `npm run dev:local` | Development mode with hot reload |\r\n| `npm run build:local` | Production build |\r\n| `npm start` | Run production build |\r\n| `npm run test:health` | Health check endpoint test |\r\n| `npm run test:integration` | Integration tests |\r\n\r\n## Docker Deployment\r\n\r\nZIP can be run in Docker with hot reloading for development and optimized builds for production.\r\n\r\n### Quick Start\r\n\r\n**Development:**\r\n```bash\r\nmake dev\r\n# or\r\ndocker-compose up\r\n```\r\n\r\n**Production:**\r\n```bash\r\nmake prod:build\r\nmake prod:up\r\n```\r\n\r\n### Services\r\n\r\nThe Docker setup includes:\r\n\r\n1. **zip-app** (Next.js application)\r\n   - Port: 3000\r\n   - Health: `GET /api/health`\r\n   - Hot reloading in development\r\n\r\n2. **robot-bridge** (Robot communication bridge)\r\n   - WebSocket: 8765\r\n   - HTTP: 8766\r\n   - Health: `GET /health`\r\n   - Serial port access for robot communication\r\n\r\n### Serial Port Configuration\r\n\r\nFor robot bridge to access serial ports:\r\n\r\n**Linux:**\r\n```yaml\r\ndevices:\r\n  - /dev/ttyUSB0:/dev/ttyUSB0\r\n```\r\n\r\n**Windows/WSL2:**\r\n```yaml\r\ndevices:\r\n  - //./COM3:/dev/ttyUSB0\r\n```\r\n\r\n**Testing without hardware:**\r\nSet `LOOPBACK_MODE=true` in `.env` to enable loopback mode.\r\n\r\nFor detailed Docker documentation, see [docs/docker/README.md](docs/docker/README.md).\r\n\r\n## Architecture Overview\r\n\r\n### Event-Driven Architecture\r\n\r\nZIP uses a typed event bus system where all UI state changes flow through events:\r\n\r\n```\r\nUser Input → Event Bus → State Reducer → UI Updates\r\n                ↓\r\n         Tool Executor → OpenAI API → Tool Results → Panel Updates\r\n                ↓\r\n         Robot Bridge → Serial → Firmware → Motors/Sensors\r\n```\r\n\r\n### Core Systems\r\n\r\n| System | Location | Description |\r\n|--------|----------|-------------|\r\n| Event Bus | `lib/events/` | Typed event system for all UI state changes |\r\n| State Machine | `lib/state/hudStore.ts` | HUD reducer managing Zip states |\r\n| Tool Registry | `lib/tools/registry.ts` | Whitelisted tools with Zod schemas |\r\n| Tool Executor | `lib/tools/executor.ts` | Executor with permissions, audit, tracing |\r\n| AI Orchestration | `lib/orchestrators/brain.ts` | LangGraph-based orchestration system |\r\n| Robot Client | `lib/robot/` | WebSocket client for robot communication |\r\n| Memory | `lib/memory/` | SQLite-based pinned memory |\r\n| Observability | `lib/observability/` | Tracing and audit logging |\r\n\r\n### OpenAI Integration\r\n\r\nZIP uses a two-model approach:\r\n\r\n1. **Realtime Model** (`OPENAI_REALTIME_MODEL`): Low-latency voice interactions via WebRTC\r\n2. **Responses Model** (`OPENAI_RESPONSES_MODEL`): Stronger reasoning for planning and tool calling\r\n\r\n#### Realtime WebRTC\r\n\r\n- **Endpoint**: `/api/realtime/token` - Returns ephemeral token for WebRTC connection\r\n- **Bridge Endpoint**: `/api/realtime/bridge` - Bridges voice to AI orchestration\r\n- **Client Hook**: `hooks/useRealtime.ts` - Manages WebRTC connection\r\n- **State Mapping**: Realtime states → Zip states (LISTENING/THINKING/SPEAKING/TOOL_RUNNING)\r\n- **Barge-in**: User can interrupt Zip while speaking\r\n- **Fallback**: STT/TTS pipeline if Realtime unavailable\r\n\r\n#### Responses API\r\n\r\n- **Endpoint**: `/api/agent` - Main chat endpoint with tool calling\r\n- **AI Brain Orchestration**: All requests route through `lib/orchestrators/brain.ts`\r\n- **Intelligent Routing**: Automatically routes to research, workflow, or direct tool calling\r\n- **Multi-Step Loops**: Supports up to 10 iterations of tool calling\r\n- **Memory Integration**: Pinned memory automatically included in system prompt\r\n\r\n## Tool Registry\r\n\r\n### Permission Tiers\r\n\r\n| Tier | Description | Confirmation |\r\n|------|-------------|--------------|\r\n| READ | Safe read-only operations | None |\r\n| WRITE | Data modification | None |\r\n| ACT | Physical actions or external effects | Required |\r\n| ADMIN | Administrative operations | Reserved |\r\n\r\n### Robot Tools (7 tools)\r\n\r\n| Tool | Tier | Description |\r\n|------|------|-------------|\r\n| `get_robot_status` | READ | Connection status and bridge state |\r\n| `get_robot_diagnostics` | READ | Motor states, reset count, serial stats |\r\n| `get_robot_sensors` | READ | Ultrasonic, line sensors, battery voltage |\r\n| `robot_move` | ACT | Move with velocity and turn rate |\r\n| `robot_stop` | ACT | Emergency stop |\r\n| `robot_stream_start` | ACT | Start continuous motion streaming |\r\n| `robot_stream_stop` | ACT | Stop motion streaming |\r\n\r\n### 3D Printer Tools (11 tools)\r\n\r\n| Tool | Tier | Description |\r\n|------|------|-------------|\r\n| `get_printer_status` | READ | Comprehensive printer status |\r\n| `get_printer_temperature` | READ | Hotend and bed temperatures |\r\n| `get_print_progress` | READ | Current print job progress |\r\n| `list_printer_files` | READ | G-code files on printer |\r\n| `start_print` | ACT | Start printing a G-code file |\r\n| `pause_print` | ACT | Pause current print |\r\n| `resume_print` | ACT | Resume paused print |\r\n| `cancel_print` | ACT | Cancel current print |\r\n| `set_temperature` | ACT | Set hotend or bed temperature |\r\n| `home_axes` | ACT | Home specified axes |\r\n| `move_axis` | ACT | Move a specific axis |\r\n\r\n### System \u0026 Info Tools\r\n\r\n| Tool | Tier | Description |\r\n|------|------|-------------|\r\n| `get_system_stats` | READ | CPU, RAM, Disk usage |\r\n| `get_weather` | READ | Weather with forecast and air quality |\r\n| `get_uptime` | READ | System uptime and session stats |\r\n\r\n### Web \u0026 Research Tools\r\n\r\n| Tool | Tier | Description |\r\n|------|------|-------------|\r\n| `web_search` | READ | Search the web for current information |\r\n| `fetch_url` | READ | Fetch and extract content from URLs |\r\n| `summarize_sources` | READ | Summarize multiple sources with citations |\r\n\r\n### Document Tools\r\n\r\n| Tool | Tier | Description |\r\n|------|------|-------------|\r\n| `ingest_document` | WRITE | Ingest PDFs and text documents |\r\n| `doc_search` | READ | Vector search across documents |\r\n| `doc_answer` | READ | Answer questions from documents |\r\n\r\n### Notes \u0026 Timers\r\n\r\n| Tool | Tier | Description |\r\n|------|------|-------------|\r\n| `create_note` | WRITE | Create a new note |\r\n| `list_notes` | READ | List all notes |\r\n| `search_notes` | READ | Search notes by content |\r\n| `delete_note` | WRITE | Delete a note |\r\n| `create_timer` | ACT | Create a timer with reminder |\r\n| `cancel_timer` | ACT | Cancel a timer |\r\n\r\n### Vision \u0026 Camera\r\n\r\n| Tool | Tier | Description |\r\n|------|------|-------------|\r\n| `analyze_image` | READ | Analyze images using vision AI |\r\n| `set_camera_enabled` | ACT | Toggle camera state |\r\n| `open_url` | ACT | Open URL in browser (requires confirmation) |\r\n\r\n## Firmware Protocol\r\n\r\nThe firmware uses ELEGOO-style JSON protocol over serial (115200 baud):\r\n\r\n```json\r\n{\"N\":\u003ccommand\u003e,\"H\":\"\u003ctag\u003e\",\"D1\":\u003cval\u003e,\"D2\":\u003cval\u003e,\"T\":\u003cttl\u003e}\r\n```\r\n\r\n### Commands\r\n\r\n| N | Command | Parameters | Response | Description |\r\n|---|---------|------------|----------|-------------|\r\n| 0 | Hello | H=tag | `{\u003ctag\u003e_ok}` | Handshake/ping |\r\n| 21 | Read Ultrasonic | H=tag | `{\u003ctag\u003e_\u003ccm\u003e}` | Distance in cm |\r\n| 22 | Read Line Sensors | H=tag | `{\u003ctag\u003e_L_M_R}` | Line sensor values |\r\n| 23 | Read Battery | H=tag | `{\u003ctag\u003e_\u003cmV\u003e}` | Battery voltage |\r\n| 120 | Diagnostics | H=tag | Multi-line | Debug state dump |\r\n| 200 | Setpoint | D1=v, D2=w, T=ttl | (none) | Streaming motion |\r\n| 201 | Stop | H=tag | `{\u003ctag\u003e_ok}` | Immediate stop |\r\n| 210 | Macro Start | D1=id, H=tag | `{\u003ctag\u003e_ok}` | Start macro |\r\n| 211 | Macro Cancel | H=tag | `{\u003ctag\u003e_ok}` | Cancel macro |\r\n| 300 | Servo | D1=angle, H=tag | `{\u003ctag\u003e_ok}` | Set servo angle |\r\n| 999 | Direct Motor | D1=L, D2=R, H=tag | `{\u003ctag\u003e_ok}` | Raw PWM control |\r\n\r\nSee [robot/firmware/zip_robot_uno/README.md](robot/firmware/zip_robot_uno/README.md) for full protocol documentation.\r\n\r\n## API Endpoints\r\n\r\n### Agent \u0026 Chat\r\n- `POST /api/agent` - Main agent endpoint with tool calling\r\n\r\n### Realtime Voice\r\n- `GET /api/realtime/token` - Get ephemeral token for WebRTC\r\n- `POST /api/realtime/bridge` - Bridge voice to orchestration\r\n\r\n### Voice Fallback\r\n- `POST /api/voice/transcribe` - STT using Whisper\r\n- `POST /api/voice/speak` - TTS synthesis\r\n\r\n### Memory\r\n- `GET /api/memory/get` - Get memories\r\n- `POST /api/memory/add` - Add memory\r\n- `DELETE /api/memory/delete` - Delete memory\r\n\r\n### Notes\r\n- `POST /api/notes/create` - Create note\r\n- `GET /api/notes/list` - List notes\r\n- `POST /api/notes/search` - Search notes\r\n- `DELETE /api/notes/delete` - Delete note\r\n\r\n### Timers\r\n- `POST /api/timers/create` - Create timer\r\n- `DELETE /api/timers/cancel` - Cancel timer\r\n\r\n### Tools\r\n- `POST /api/tools/web_search` - Web search\r\n- `POST /api/tools/fetch_url` - Fetch URL content\r\n- `POST /api/tools/vision` - Analyze image\r\n- `POST /api/tools/docs/ingest` - Ingest document\r\n- `POST /api/tools/docs/search` - Search documents\r\n- `POST /api/tools/docs/answer` - Answer from documents\r\n\r\nAll endpoints support rate limiting and return JSON responses with proper error handling.\r\n\r\n## Environment Variables\r\n\r\n### Required\r\n\r\n| Variable | Description |\r\n|----------|-------------|\r\n| `OPENAI_API_KEY` | OpenAI API key for voice and AI features |\r\n\r\n### Robot Configuration\r\n\r\n| Variable | Default | Description |\r\n|----------|---------|-------------|\r\n| `NEXT_PUBLIC_ROBOT_BRIDGE_WS_URL` | `ws://localhost:8765/robot` | Bridge WebSocket URL |\r\n| `SERIAL_PORT` | auto-detect | Serial port (COM5, /dev/ttyUSB0) |\r\n| `SERIAL_BAUD` | `115200` | Serial baud rate |\r\n| `LOOPBACK_MODE` | `false` | Test mode without hardware |\r\n\r\n### OpenAI Models\r\n\r\n| Variable | Default | Description |\r\n|----------|---------|-------------|\r\n| `OPENAI_REALTIME_MODEL` | `gpt-4o-realtime-preview-2024-12-17` | Realtime voice model |\r\n| `OPENAI_RESPONSES_MODEL` | `gpt-4o` | Chat/reasoning model |\r\n| `OPENAI_VISION_MODEL` | `gpt-4o` | Vision API model |\r\n| `OPENAI_TTS_MODEL` | `gpt-4o-mini-tts-2025-12-15` | TTS model |\r\n| `OPENAI_STT_MODEL` | `whisper-1` | STT model |\r\n| `OPENAI_EMBEDDING_MODEL` | `text-embedding-3-small` | Embedding model |\r\n\r\n### HUD Configuration\r\n\r\n| Variable | Default | Description |\r\n|----------|---------|-------------|\r\n| `ZIP_REALTIME_ENABLED` | `true` | Enable Realtime WebRTC |\r\n| `ZIP_VOICE_FALLBACK_ENABLED` | `true` | Enable STT/TTS fallback |\r\n| `ZIP_UPDATE_INTERVAL_MS` | `2000` | Panel update interval |\r\n| `PRINTER_API_URL` | `http://169.254.178.90` | 3D printer API URL |\r\n\r\n## Project Structure\r\n\r\n```\r\nzip/\r\n├── robot/                      # Robot integration\r\n│   ├── firmware/\r\n│   │   ├── zip_robot_uno/      # Arduino UNO firmware\r\n│   │   │   ├── src/            # Source files\r\n│   │   │   ├── include/        # Headers\r\n│   │   │   ├── tools/          # Test utilities\r\n│   │   │   └── README.md       # Firmware documentation\r\n│   │   └── zip_esp32_cam/      # ESP32-CAM firmware (in development)\r\n│   ├── bridge/\r\n│   │   └── zip-robot-bridge/   # WebSocket-to-Serial bridge\r\n│   │       ├── src/            # Bridge source\r\n│   │       └── README.md       # Bridge documentation\r\n│   └── tools/                  # Testing and diagnostic utilities\r\n├── app/                        # Next.js HUD application\r\n│   ├── api/                    # API routes\r\n│   │   ├── agent/              # Main agent endpoint\r\n│   │   ├── realtime/           # Realtime endpoints\r\n│   │   ├── voice/              # Voice fallback endpoints\r\n│   │   ├── memory/             # Memory endpoints\r\n│   │   ├── notes/              # Notes endpoints\r\n│   │   ├── timers/             # Timer endpoints\r\n│   │   └── tools/              # Tool endpoints\r\n│   ├── (hud)/                  # Main HUD page\r\n│   └── robot/                  # Robot control page\r\n├── components/                 # React components\r\n│   ├── hud/                    # HUD-specific components\r\n│   │   ├── TopBar.tsx          # Top navigation bar\r\n│   │   ├── LeftRail.tsx        # Left panel rail\r\n│   │   ├── RightChat.tsx       # Chat interface\r\n│   │   ├── CenterCore.tsx      # Center display\r\n│   │   └── panels/             # Panel components\r\n│   └── robot/                  # Robot control components\r\n│       ├── ConnectionStatus.tsx\r\n│       ├── MotionControl.tsx\r\n│       ├── MotorGauges.tsx\r\n│       ├── SensorDisplay.tsx\r\n│       ├── SerialConsole.tsx\r\n│       └── ServoControl.tsx\r\n├── hooks/                      # React hooks\r\n│   ├── useChat.ts              # Chat hook\r\n│   ├── useRealtime.ts          # Realtime hook\r\n│   ├── useRobot.ts             # Robot bridge hook\r\n│   ├── usePanelUpdates.ts      # Panel update hook\r\n│   └── useTTS.ts               # TTS fallback hook\r\n├── lib/                        # Core libraries\r\n│   ├── robot/                  # Robot client\r\n│   │   ├── client.ts           # Browser WebSocket client\r\n│   │   ├── server-client.ts    # Server-side HTTP client\r\n│   │   ├── wifi-client.ts      # Direct WiFi client\r\n│   │   └── types.ts            # Robot type definitions\r\n│   ├── events/                 # Event bus system\r\n│   ├── orchestrators/          # AI orchestration\r\n│   │   ├── brain.ts            # Main orchestration graph\r\n│   │   ├── nodes/              # Orchestration nodes\r\n│   │   └── utils/              # Utilities\r\n│   ├── tools/                  # Tool registry and executor\r\n│   │   ├── registry.ts         # Tool registry\r\n│   │   ├── executor.ts         # Tool executor\r\n│   │   └── implementations/    # Tool implementations\r\n│   ├── memory/                 # Memory management\r\n│   ├── observability/          # Tracing and audit\r\n│   ├── openai/                 # OpenAI integration\r\n│   └── voice/                  # Voice system\r\n├── scripts/                    # Utility scripts\r\n├── docs/                       # Documentation\r\n│   ├── agents/                 # Agent guides\r\n│   └── docker/                 # Docker documentation\r\n└── data/                       # Runtime data (auto-created)\r\n    ├── audit.log               # Audit logs\r\n    ├── traces/                 # Trace files\r\n    ├── memory.db               # Memory database\r\n    ├── notes.db                # Notes database\r\n    └── docs.db                 # Documents database\r\n```\r\n\r\n## Hardware Setup\r\n\r\n### Components\r\n\r\n- ELEGOO Smart Robot Car V4.0 kit\r\n- Arduino UNO R3\r\n- SmartCar-Shield-v1.1 (TB6612FNG motor driver)\r\n- HC-SR04 ultrasonic sensor\r\n- SG90 servo for sensor head\r\n- MPU6050 IMU module\r\n- ESP32-CAM module (optional, for vision)\r\n\r\n### Wiring\r\n\r\nThe firmware is configured for the standard ELEGOO shield pinout:\r\n\r\n| Component | Pins |\r\n|-----------|------|\r\n| Left Motor | D5 (PWM), D8/D9 (DIR) |\r\n| Right Motor | D6 (PWM), D10/D11 (DIR) |\r\n| Motor Standby | D3 |\r\n| Ultrasonic | D13 (Trig), D12 (Echo) |\r\n| Servo | D10 |\r\n| Line Sensors | A0, A1, A2 |\r\n| Battery | A3 |\r\n| I2C (IMU) | A4 (SDA), A5 (SCL) |\r\n\r\nSee [robot/firmware/zip_robot_uno/README.md](robot/firmware/zip_robot_uno/README.md) for detailed pin mapping.\r\n\r\n## Testing\r\n\r\n### Firmware Tests\r\n\r\n```bash\r\ncd robot/firmware/zip_robot_uno/tools\r\n\r\n# Motor bringup test\r\nnode serial_motor_bringup.js COM5\r\n\r\n# Hardware smoke test\r\nnode hardware_smoke.js COM5\r\n\r\n# Motion-only test (safe 0.5ft radius)\r\nnode serial_motor_bringup.js COM5 --motion-only\r\n```\r\n\r\n### Bridge Tests\r\n\r\n```bash\r\ncd robot/bridge/zip-robot-bridge\r\n\r\n# Health check\r\nnpm run test:health\r\n\r\n# Integration tests\r\nnpm run test:integration\r\n```\r\n\r\n### HUD Tests\r\n\r\n```bash\r\n# E2E tests\r\nnpm run test:e2e\r\n\r\n# Type checking\r\nnpm run typecheck\r\n\r\n# Orchestration tests\r\nnpx tsx scripts/test-api-orchestration.ts\r\n\r\n# Brain integration tests\r\nnpx tsx scripts/test-brain-integration.ts\r\n\r\n# Voice fallback tests\r\nnpx tsx scripts/test-voice-fallback.ts\r\n\r\n# Eval harness (20 test prompts)\r\nnpx tsx scripts/eval-harness.ts\r\n```\r\n\r\n## Security\r\n\r\n### API Key Protection\r\n- All OpenAI API calls are server-side\r\n- Realtime token endpoint uses server-side WebSocket proxy\r\n\r\n### Tool Security\r\n- Tool execution strictly whitelisted via registry\r\n- Input/output validation using Zod schemas\r\n- Permission-based access control (READ/WRITE/ACT/ADMIN)\r\n- User confirmation required for ACT-tier tools\r\n- URL and file path sanitization\r\n\r\n### Rate Limiting\r\n- In-memory rate limiter for all endpoints\r\n- Default: 100 requests/minute per IP\r\n\r\n### Audit \u0026 Tracing\r\n- Every tool call logged to `./data/audit.log`\r\n- Request-scoped tracing in `./data/traces/`\r\n- Request IDs and step IDs for debugging\r\n\r\n## Development Status\r\n\r\n**Version**: 0.1.0\r\n\r\n### Robot - Completed\r\n- ✅ Arduino firmware with motion control\r\n- ✅ WebSocket bridge server\r\n- ✅ HUD with real-time telemetry\r\n- ✅ Voice control integration\r\n- ✅ AI tool orchestration for robot commands\r\n- ✅ Sensor reading and display\r\n- ✅ IMU integration (MPU6050)\r\n- ✅ Servo control\r\n- ✅ Drive safety layer (battery-aware, deadband, ramping)\r\n\r\n### Robot - In Development\r\n- 🔄 ESP32-CAM integration for robot vision\r\n- 🔄 Autonomous navigation modes\r\n- 🔄 Path planning and mapping\r\n\r\n### HUD - Completed\r\n- ✅ Voice and text interaction\r\n- ✅ 39 tools with permission-based access\r\n- ✅ AI orchestration with intelligent routing\r\n- ✅ 3D printer integration (Moonraker/Klipper)\r\n- ✅ Document intelligence with vector search\r\n- ✅ Web research with citations\r\n- ✅ Memory system\r\n- ✅ Audit logging and tracing\r\n- ✅ Docker deployment support\r\n\r\n## Technology Stack\r\n\r\n**Robot:**\r\n- PlatformIO + Arduino framework\r\n- ELEGOO Smart Robot Car V4.0\r\n- TB6612FNG motor driver\r\n- MPU6050 IMU\r\n- ESP32-CAM (in development)\r\n\r\n**Bridge:**\r\n- Node.js + TypeScript\r\n- WebSocket (ws)\r\n- SerialPort\r\n\r\n**HUD Frontend:**\r\n- Next.js 16, React 19, TypeScript\r\n- Tailwind CSS\r\n- Three.js + React Three Fiber\r\n- Framer Motion\r\n\r\n**HUD Backend:**\r\n- Next.js API Routes\r\n- OpenAI API (Realtime, Responses, Vision, TTS, STT, Embeddings)\r\n- SQLite (better-sqlite3)\r\n- Zod validation\r\n\r\n**AI \u0026 Orchestration:**\r\n- LangGraph + LangChain\r\n- OpenAI Embeddings\r\n- Semantic similarity search\r\n\r\n## Contributing\r\n\r\n### For AI Agents\r\n\r\nThis project is configured for AI agent collaboration:\r\n\r\n- **[Agent Guide](AGENT_GUIDE.md)** - Quick reference\r\n- **[GitHub Copilot Instructions](.github/copilot-instructions.md)** - Copilot guide\r\n- **Issue Templates** - `.github/ISSUE_TEMPLATE/`\r\n\r\n### For Developers\r\n\r\n1. Fork the repository\r\n2. Create a feature branch\r\n3. Follow patterns in `.cursorrules`\r\n4. Run `npm run typecheck` and `npm run lint`\r\n5. Submit a pull request\r\n\r\n## License\r\n\r\nMIT\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsteffenpharai%2Fzip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsteffenpharai%2Fzip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsteffenpharai%2Fzip/lists"}