{"id":47303278,"url":"https://github.com/sucheet2000/aria","last_synced_at":"2026-04-01T17:08:21.585Z","repository":{"id":344981050,"uuid":"1183912443","full_name":"sucheet2000/aria","owner":"sucheet2000","description":"ARIA: Adaptive Realtime Intelligence Avatar – real-time vision, emotion detection, voice conversation, and 3D avatar interface powered by Claude","archived":false,"fork":false,"pushed_at":"2026-03-27T01:49:14.000Z","size":350,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"integration","last_synced_at":"2026-03-27T13:58:07.130Z","etag":null,"topics":["ai","anthropic","computer-vision","go","mediapipe","nextjs","voice-interface","whisper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sucheet2000.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-03-17T04:13:12.000Z","updated_at":"2026-03-27T01:49:18.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/sucheet2000/aria","commit_stats":null,"previous_names":["sucheet2000/aria"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/sucheet2000/aria","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sucheet2000%2Faria","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sucheet2000%2Faria/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sucheet2000%2Faria/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sucheet2000%2Faria/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sucheet2000","download_url":"https://codeload.github.com/sucheet2000/aria/tar.gz/refs/heads/integration","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sucheet2000%2Faria/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31290538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","anthropic","computer-vision","go","mediapipe","nextjs","voice-interface","whisper"],"created_at":"2026-03-17T06:23:17.675Z","updated_at":"2026-04-01T17:08:21.578Z","avatar_url":"https://github.com/sucheet2000.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ARIA — Adaptive Realtime Intelligence Avatar\n\nARIA is a multimodal AI companion that perceives you through your camera and microphone, reasons about your emotional and cognitive state using a neurosymbolic architecture, and maintains a persistent world model of who you are across sessions. It runs locally on Apple Silicon with no cloud vision or audio processing.\n\n## Architecture\n\nARIA is built on a three-language stack. A Go server coordinates all subsystems: it manages WebSocket connections to the browser, proxies cognition requests to a Python FastAPI service, and launches vision and audio workers as long-running subprocesses. Python handles all perception — MediaPipe Tasks for face and hand landmark extraction, faster-whisper for speech transcription, and a FastAPI service that builds neurosymbolic prompts, queries ChromaDB memory collections, and calls Claude. The Next.js frontend renders the live landmark overlay, conversation history, memory panel, and thinking indicator over WebSocket.\n\n```\n  Browser (Next.js :3000)\n       |\n       | WebSocket  ws://localhost:8080/ws\n       | HTTP POST  /api/cognition\n       | HTTP POST  /api/tts\n       |\n  Go Server (:8080)\n       |-- subprocess --\u003e Python Vision Worker\n       |                    MediaPipe Tasks FaceLandmarker (478 points)\n       |                    Emotion classifier (7 classes, landmark geometry)\n       |                    Head pose via solvePnP (pitch / yaw / roll)\n       |                    Hand landmarks (21 points per hand)\n       |                    15 fps JSON to stdout\n       |\n       |-- subprocess --\u003e Python Audio Worker\n       |                    sounddevice microphone capture at 16kHz\n       |                    webrtcvad voice activity detection\n       |                    DeepFilterNet noise isolation (--denoise flag)\n       |                    faster-whisper speech-to-text (base, int8)\n       |                    JSON to stdout on utterance end\n       |\n       |-- HTTP proxy --\u003e Python FastAPI (:8000)\n                            Neurosymbolic prompt builder\n                            Conflict detection (speech vs visual sentiment)\n                            Claude claude-haiku-4-5 structured response\n                            ChromaDB layered memory (profile / episodic / working)\n```\n\n## Neurosymbolic reasoning\n\nSystem 1 is the fast, parallel, pattern-based perception layer: MediaPipe extracts 478 face landmarks, head pose angles, and hand geometry at 15 fps, while faster-whisper transcribes speech in real time. These streams produce continuous signals — emotion class, attention estimate, head orientation, and utterance text — that are forwarded to the cognition layer as structured JSON on every cycle.\n\nSystem 2 is the symbolic inference layer built around Claude. On each cognition turn, the FastAPI service assembles a structured prompt from the current perceptual state, the last 10 symbolic inferences held in Go working memory, and relevant long-term facts retrieved from ChromaDB. Claude returns a structured response containing a symbolic inference, a world model triple, and a natural language reply. The triple is then written back to the appropriate ChromaDB collection, updating ARIA's persistent model of the user.\n\nConflict detection: if the absolute difference between speech sentiment and visual sentiment exceeds 0.4, ARIA responds to the visual truth via open invitation rather than validating the speech surface or directly confronting the incongruence. This prevents ARIA from reinforcing a stated emotion that contradicts observable affect.\n\nStructured response schema:\n\n```json\n{\n  \"symbolic_inference\": \"user is in focused debugging state\",\n  \"world_model_update\": {\n    \"triple\": { \"subject\": \"...\", \"predicate\": \"...\", \"object\": \"...\" },\n    \"confidence\": 0.85,\n    \"source\": \"explicit_statement\"\n  },\n  \"natural_language_response\": \"spoken response here\"\n}\n```\n\n## Memory system\n\nARIA uses three ChromaDB collections with different retention policies:\n\n| Collection    | Retention   | Contents |\n|---------------|-------------|----------|\n| aria_profile  | permanent   | explicit user facts |\n| aria_episodic | 30-day TTL  | behavioral and visual inferences |\n| aria_working  | session     | cleared on shutdown |\n\nWorking memory is implemented as a Go circular buffer holding the last 10 symbolic inferences. Its contents are serialized and injected into every Claude prompt as short-term context, giving ARIA continuity within a session without requiring a database lookup on every turn.\n\n## Versions\n\n| Version | Status   | Description |\n|---------|----------|-------------|\n| v0.1.0  | released | Monorepo scaffold |\n| v0.2.0  | released | Go WebSocket server, MediaPipe vision pipeline, Next.js frontend |\n| v0.3.0  | released | Emotion detection, Claude cognition API |\n| v0.4.1  | released | Voice input, faster-whisper, webrtcvad, streaming TTS |\n| v0.5.0  | released | Neurosymbolic reasoning, world model triples, working memory |\n| v0.6.0  | released | ChromaDB persistent memory, DeepFilterNet noise isolation, message UI |\n| v0.7.0  | planned  | VRM avatar, three-vrm, lip sync, emotion blendshapes |\n| v1.0.0  | planned  | Gesture recognition, wake word detection, full MVP |\n\n## Tech stack\n\n| Layer     | Technology |\n|-----------|-----------|\n| Server    | Go 1.25, chi, gorilla/websocket, zerolog |\n| Vision    | Python 3.13, MediaPipe Tasks 0.10.32, OpenCV |\n| Audio     | Python 3.13, faster-whisper, webrtcvad, sounddevice, DeepFilterNet |\n| AI        | Anthropic Claude claude-haiku-4-5 |\n| Memory    | ChromaDB 1.5.5 (profile, episodic, working collections) |\n| Frontend  | Next.js 14, TypeScript, Tailwind CSS, Zustand |\n| TTS       | ElevenLabs (macOS say fallback) |\n\n## Prerequisites\n\n- macOS Apple Silicon (M1/M2/M3) or Linux\n- Go 1.25+\n- Python 3.11+ ARM64 native (see setup below)\n- Node.js 20+\n- Anthropic API key\n\n## Setup\n\n### 1. Clone\n\n    git clone https://github.com/sucheet2000/aria.git\n    cd aria\n    cp backend/.env.example backend/.env\n    # Add ANTHROPIC_API_KEY to backend/.env\n\n### 2. Python environment (Apple Silicon)\n\n    curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh\n    bash Miniconda3-latest-MacOSX-arm64.sh -b -p ~/miniconda-arm64\n    source ~/miniconda-arm64/bin/activate\n    pip install mediapipe opencv-python torch faster-whisper webrtcvad \\\n                sounddevice structlog pydantic pydantic-settings \\\n                anthropic fastapi uvicorn chromadb deepfilternet\n\n    # Set in backend/.env:\n    # PYTHON_BIN=/Users/YOUR_USERNAME/miniconda-arm64/bin/python3\n\n### 3. Go dependencies\n\n    cd backend \u0026\u0026 go mod download\n\n### 4. Frontend\n\n    cd frontend \u0026\u0026 npm install\n\n### 5. Run (three terminals required)\n\n    # Terminal 1 — FastAPI cognition and memory service\n    source ~/miniconda-arm64/bin/activate\n    cd backend\n    PYTHONPATH=$(pwd) python3 -m uvicorn app.main:app --port 8000\n\n    # Terminal 2 — Go server (auto-starts vision and audio workers)\n    cd backend\n    go run cmd/server/main.go\n\n    # Terminal 3 — Frontend\n    cd frontend\n    npm run dev\n\n    Open http://localhost:3000 and allow camera and microphone access.\n\n## Development commands\n\n| Command | Description |\n|---------|-------------|\n| go run cmd/server/main.go | Start Go server with Python workers |\n| uvicorn app.main:app --port 8000 | Start FastAPI cognition service |\n| npm run dev | Start Next.js frontend |\n| pytest tests/ -v | Run Python test suite |\n| go test ./... | Run Go test suite |\n| python scripts/audio_test.py | Test microphone and transcription |\n| python scripts/audio_test.py --denoise | Test with noise isolation |\n| curl localhost:8000/api/memory/profile | Inspect stored profile facts |\n\n## Environment variables\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| ANTHROPIC_API_KEY | Anthropic API key | required |\n| PYTHON_BIN | Path to ARM64 Python | python3 |\n| WHISPER_MODEL | Whisper model size | base |\n| AUDIO_ENABLED | Enable audio worker | true |\n| ELEVENLABS_API_KEY | ElevenLabs TTS key | optional |\n| USE_OLLAMA | Use Ollama instead of Claude | false |\n\nSee backend/.env.example for the complete list.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsucheet2000%2Faria","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsucheet2000%2Faria","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsucheet2000%2Faria/lists"}