{"id":30818442,"url":"https://github.com/sinarck/hand-wave","last_synced_at":"2026-05-01T12:33:12.515Z","repository":{"id":285306854,"uuid":"957665071","full_name":"sinarck/hand-wave","owner":"sinarck","description":"Detecting sign language in real time with Meta AI glasses (through WhatsApp video call).","archived":false,"fork":false,"pushed_at":"2025-10-19T02:45:15.000Z","size":52731,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-19T15:53:47.790Z","etag":null,"topics":["ai","augmented-reality","expo","meta","mobile-app","python","sign-language","wearable"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sinarck.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-30T22:35:05.000Z","updated_at":"2025-10-19T02:45:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"cf755c1f-4073-40b4-bcb8-d329a29658f7","html_url":"https://github.com/sinarck/hand-wave","commit_stats":null,"previous_names":["sinarck/sign-language-detection","sinarck/hand-wave"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sinarck/hand-wave","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinarck%2Fhand-wave","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinarck%2Fhand-wave/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinarck%2Fhand-wave/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinarck%2Fhand-wave/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sinarck","download_url":"https://codeload.github.com/sinarck/hand-wave/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinarck%2Fhand-wave/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32497812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","augmented-reality","expo","meta","mobile-app","python","sign-language","wearable"],"created_at":"2025-09-06T09:52:37.604Z","updated_at":"2026-05-01T12:33:12.505Z","avatar_url":"https://github.com/sinarck.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hand Wave with Meta AI Glasses\n\nReal-time ASL (American Sign Language) recognition from Meta AI glasses, streamed via WhatsApp Desktop (or macOS continuity camera) to a web app. Uses MediaPipe Tasks (browser) to extract landmarks and a TFLite model (Python FastAPI) to translate landmarks to text.\n\n## Demo\n\n\u003cdiv align=\"center\"\u003e\n\u003ca href=\"https://youtu.be/ZAxBf1XG9lU\"\u003e\n\u003cimg src=\"https://img.youtube.com/vi/ZAxBf1XG9lU/0.jpg\" alt=\"Demo: Hand Wave with Meta AI Glasses\" width=\"720\" /\u003e\n\u003c/a\u003e\n\u003cbr /\u003e\n\u003cem\u003eReal-time stream from glasses with hand landmark detection and ASL translation.\u003c/em\u003e\n\u003c/div\u003e\n\n## Status\n\n- Prototype; not production-ready\n- WhatsApp transport (lower latency than Instagram Live)\n- Hand landmark detection and translation working end-to-end\n- Audio readout to glasses in progress\n\n### Model status and attribution\n- We are actively training a better ASL model tailored to our pipeline and dataset.\n- For demo purposes, we temporarily use models inspired by and structured after CMU ETC Questure’s open-source work. Please see their repository for details and credit: [Questure Tracker (Apache-2.0)](https://github.com/ZWeiweiY/Questure_Tracker/tree/main).\n- If you use our demo configuration, attribute Questure appropriately and respect their license.\n\n## Quick Start\n\n**Prerequisites:**\n- Meta AI glasses\n- iPhone (Sender) with WhatsApp (Account A)\n- Laptop (Receiver) with WhatsApp Desktop (Account B)\n- Two WhatsApp accounts\n- Bun (monorepo uses Bun workspaces)\n\n**Install and run:**\n\n```bash\nbun install\nbun run dev\n```\n\nThis starts:\n- Web app: `https://localhost:3001`\n- tRPC server: `https://localhost:3000`\n- Inference service: `http://localhost:8000`\n\n## Use with WhatsApp\n\n1. On the phone (Account A), pair the glasses and open WhatsApp\n2. On the laptop (Account B), open WhatsApp Desktop\n3. Start a WhatsApp video call between Account A and Account B\n4. On the glasses, double-tap the picture button to switch the call's camera to the glasses\n5. Open `https://localhost:3001` and click \"Share Screen\"; select the WhatsApp call window\n6. The browser runs on-device hand detection and sends landmarks to the inference service for translation\n7. Audio readback to the glasses is WIP\n\n## How It Works\n\n**Video \u0026 Detection Path:**\n- Video: Glasses → Phone → WhatsApp → Laptop → Browser screen share\n- Detection: Browser MediaPipe Tasks: Face, Pose, Hand landmarkers (WASM) combined in-app\n- Translation: Landmarks (not raw frames) → FastAPI Inference Service → TFLite models (static and movement)\n- Output: Text in browser; TTS back to glasses (WIP)\n\n**Privacy:**\n- No raw video frames are uploaded\n- Only hand landmark coordinates are sent to the inference service\n\n## Architecture\n\n```\nMeta AI Glasses\n    ↓ video stream\nWhatsApp Call\n    ↓ screen share\nBrowser (MediaPipe Face/Pose/Hand)\n    ↓ landmarks [frames, keypoints, (x,y,z)]\nFastAPI Inference (port 8000)\n    ↓ TFLite Models (static_model.tflite, movement_model.tflite)\n    ↓ prediction\nBrowser UI (tRPC optional)\n    ↓ (future) TTS\nMeta Glasses audio\n```\n\n## Tech Stack\n\n**Web (apps/web):**\n- Next.js 15, React 19, TypeScript\n- Tailwind CSS 4, shadcn/ui\n- MediaPipe Tasks Vision (Face, Pose, Hand landmarkers; combined holistically)\n- Zustand, TanStack Query 5\n\n**Backend (apps/server):**\n- tRPC 11 over Hono (Bun runtime)\n- Optional in this flow; browser can call FastAPI directly\n\n**Inference (apps/inference):**\n- FastAPI (Python)\n- TensorFlow Lite\n- Static and movement models (TFLite). While we train our own, demo may use models aligned with Questure’s structure. See Attribution above.\n\n**Monorepo:**\n- Turbo + Bun workspaces\n\n## Project Structure\n\n```\napps/\n  web/        # Next.js app (UI, screen share, hand detection)\n  server/     # Bun + Hono + tRPC API\n  inference/  # FastAPI + TFLite inference service\n```\n\n## Scripts\n\nRun from repository root:\n\n```bash\nbun run dev          # Start all services (web HTTPS:3001, server, inference)\nbun run build        # Build all apps\nbun run check-types  # Typecheck all apps\nbun run dev:web      # Web only\nbun run dev:server   # Server only\n```\n\nFor inference service only:\n```bash\ncd apps/inference\nuv run uvicorn src.main:app --reload --port 8000\n```\n\n## Environment\n\nCreate a `.env` file at the repository root:\n\n```env\nNEXT_PUBLIC_SERVER_URL=https://localhost:3000\nPORT=3000\nCORS_ORIGIN=https://localhost:3001\n```\n\n**Web (`apps/web`):**\n- `NEXT_PUBLIC_SERVER_URL` – tRPC server URL\n\n**Server (`apps/server`):**\n- `PORT` – Server port (default: 3000)\n- `CORS_ORIGIN` – Allowed CORS origin\n\n## Models\n\nPlace the following in `apps/inference/models/`:\n- `static_model.tflite` – single-frame classifier\n- `movement_model.tflite` – sequence classifier\n- `label.csv` – label mapping\n\nWe are training new models. For demos, you may mirror the file layout informed by Questure’s repo. Credit: [Questure Tracker (Apache-2.0)](https://github.com/ZWeiweiY/Questure_Tracker/tree/main).\n\n## Troubleshooting\n\n**\"Share Screen\" doesn't list WhatsApp window:**\n- Ensure the call is active and the window is visible\n- macOS: Grant Screen Recording permission to your browser (System Settings → Privacy \u0026 Security → Screen Recording)\n- Browser must use HTTPS; `getDisplayMedia` requires a secure context\n\n**Model not found:**\n```\nWarning: No model found at .../models/static_model.tflite\n```\nPlace `static_model.tflite`, `movement_model.tflite`, and `label.csv` in `apps/inference/models/`\n\n**TensorFlow errors:**\n```bash\ncd apps/inference\nuv add tensorflow\n```\n\n**Wrong input shape:**\nThe model expects exactly 130 landmarks per frame (21 left hand + 21 right hand + face + pose landmarks). Check MediaPipe configuration.\n\n## API Documentation\n\n**Inference Service:**\n- Swagger UI: http://localhost:8000/docs\n- ReDoc: http://localhost:8000/redoc\n\n**Endpoints:**\n- `GET /health` - Health check\n- `POST /predict` - Predict ASL text from landmarks\n\n## Limitations\n\n- Hackathon-grade prototype, not production-ready\n- In-glasses audio readout not yet implemented\n- No authentication or multi-user coordination\n- No recording controls or session management\n\n## Roadmap\n\n- Complete text-to-speech back to the glasses\n- Improve model accuracy and add gesture classification\n- Add authentication and session management\n- Recording controls and replay functionality\n- Multi-user support\n\n## License\n\nSee LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsinarck%2Fhand-wave","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsinarck%2Fhand-wave","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsinarck%2Fhand-wave/lists"}