{"id":25360207,"url":"https://github.com/owengretzinger/aintern","last_synced_at":"2025-04-09T02:46:51.578Z","repository":{"id":274146119,"uuid":"921997411","full_name":"owengretzinger/aintern","owner":"owengretzinger","description":"3D AI video meeting assistant that responds to questions in real-time and takes notes","archived":false,"fork":false,"pushed_at":"2025-01-30T09:30:30.000Z","size":12079,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-14T21:33:04.069Z","etag":null,"topics":["express","google-meet","hackathon","meetings","openai","qhacks","react","recall-ai","supabase","websockets"],"latest_commit_sha":null,"homepage":"https://youtu.be/C3fAWMhHPIY","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/owengretzinger.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-25T03:42:11.000Z","updated_at":"2025-02-11T13:12:47.000Z","dependencies_parsed_at":"2025-01-25T08:34:03.838Z","dependency_job_id":null,"html_url":"https://github.com/owengretzinger/aintern","commit_stats":null,"previous_names":["owengretzinger/qhacks"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owengretzinger%2Faintern","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owengretzinger%2Faintern/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owengretzinger%2Faintern/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owengretzinger%2Faintern/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/owengretzinger","download_url":"https://codeload.github.com/owengretzinger/aintern/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247968241,"owners_count":21025797,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["express","google-meet","hackathon","meetings","openai","qhacks","react","recall-ai","supabase","websockets"],"created_at":"2025-02-14T21:28:31.613Z","updated_at":"2025-04-09T02:46:51.556Z","avatar_url":"https://github.com/owengretzinger.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# (A)Intern\n\nA 3D AI meeting assistant that joins your video calls, responds to questions in real-time, and takes meeting notes automatically.\n\nhttps://github.com/user-attachments/assets/e6afa2e4-1626-4273-97b0-04f05b5f4d0c\n\n## Overview\n\n(A)Intern features a 3D virtual AI assistant named Iris that can:\n\n\u003cdetails\u003e\n\u003csummary\u003eJoin meetings on platforms like Google Meet by simply providing a URL\u003c/summary\u003e\n\n![Enter URL](https://github.com/user-attachments/assets/b32e5e28-5a92-465c-b4f5-8e3de35239e6)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eAnswer questions in real-time with natural voice responses and lifelike expressions\u003c/summary\u003e\n\n![In Meeting](https://github.com/user-attachments/assets/c33e46a5-2e69-4d7d-be45-7f330ee2c62a)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eAccess and reference uploaded company documents for contextual responses\u003c/summary\u003e\n\n![image](https://github.com/user-attachments/assets/61af77c3-0eec-46a8-8a6e-e3c37aaaa59a)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTake automated meeting notes and generate summaries\u003c/summary\u003e\n\n![Transcript and Summary](https://github.com/user-attachments/assets/21d8f06f-1fc1-40ef-aad4-90c305af949a)\n\n\u003c/details\u003e\n\n## Project Structure\n\nThe project consists of four main components:\n\n- `/avatar` - 3D avatar rendering and animation system\n- `/backend` - Express server handling AI, meeting management, and data processing\n- `/dashboard` - React-based user interface for managing meetings and documents\n- `/video-stream` - Video streaming service for meeting integration\n\n## Technical Architecture\n\n### Avatar System\n\n- Ready Player Me for 3D modeling\n- Mixamo for natural animations\n- ElevenLabs for voice synthesis\n- Rhubarb Lip Sync for lip synchronization\n- Three.js for 3D rendering\n\n### Meeting Integration\n\n- Recall.ai API for meeting bot capabilities\n- Custom video streaming solution for Three.js compatibility\n- Real-time transcription and response system\n- OpenAI for natural language processing\n\n### Core Technologies\n\n- React/Vite frontends\n- Express.js backend\n- Supabase database\n- WebSocket communication\n\n## Technical Implementation\n\nThe system works through the following flow:\n\n1. User submits a meeting URL through the dashboard\n2. Meeting bot joins the call via Recall.ai\n3. Questions are transcribed in real-time\n4. Backend processes queries with relevant document context\n5. AI generates responses with voice synthesis and lip-sync\n6. Avatar stream is piped back into the meeting\n\n### Latency Handling\n\nThe current system has approximately 10 seconds of latency due to our unique technical solution:\n\n- Recall's headless browser can't directly render Three.js\n- Solution: Convert avatar to video stream first\n- Trade-off: Higher latency for better visual experience\n\n![Flow Diagram](https://github.com/user-attachments/assets/d4c49d0a-0bc0-48bf-8da4-7904f82d9f1f)\n\n## Development\n\n### Prerequisites\n\n- Node.js\n- pnpm\n- ngrok (for local development with Recall.ai)\n\n### Local Development with ngrok\n\nWhen testing the meeting integration locally, you'll need ngrok to expose your local server to the internet. This is required because Recall.ai needs to communicate with publicly accessible endpoints.\n\n1. Install ngrok: https://ngrok.com/download\n2. Start ngrok on port 3001 (or your backend port):\n   ```bash\n   ngrok http 3001\n   ```\n3. Update your environment variables to use the ngrok URL:\n   - In `backend/.env`, set `PUBLIC_URL` to your ngrok URL\n   - In `video-stream/.env`, update the WebSocket URL accordingly\n\n### Setup\n\n1. Clone the repository\n2. Install dependencies in each directory:\n   ```bash\n   cd avatar \u0026\u0026 pnpm install\n   cd ../backend \u0026\u0026 pnpm install\n   cd ../dashboard \u0026\u0026 pnpm install\n   cd ../video-stream \u0026\u0026 pnpm install\n   ```\n3. Configure environment variables (see `.env.example` in each directory)\n4. Start the development servers\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fowengretzinger%2Faintern","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fowengretzinger%2Faintern","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fowengretzinger%2Faintern/lists"}