https://github.com/nagarajpandith/aido
3D multilingual chatbot with interactive gestures, contextual conversations, persistent chat history, and personalized medical and mental health support.
https://github.com/nagarajpandith/aido
chatbot cloud-run googlecloud mixamo nextjs readyplayerme shadcn-ui t3-stack tailwindcss threejs translate trpc
Last synced: 1 day ago
JSON representation
3D multilingual chatbot with interactive gestures, contextual conversations, persistent chat history, and personalized medical and mental health support.
- Host: GitHub
- URL: https://github.com/nagarajpandith/aido
- Owner: nagarajpandith
- Created: 2023-08-28T15:47:58.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2023-08-31T10:21:51.000Z (over 1 year ago)
- Last Synced: 2023-09-01T00:34:46.250Z (over 1 year ago)
- Topics: chatbot, cloud-run, googlecloud, mixamo, nextjs, readyplayerme, shadcn-ui, t3-stack, tailwindcss, threejs, translate, trpc
- Language: TypeScript
- Homepage: https://aido-ar2ndw3szq-uc.a.run.app/
- Size: 21.9 MB
- Stars: 4
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Aido
🏆 Awarded the 'Most Innovative Approach' at [NivHack](https://niveussolutions.com/hackathon) 2023 hosted at Niveus Solutions Pvt. Ltd., Mangalore.
- [Live URL](https://aido-ar2ndw3szq-uc.a.run.app/)
- VertexAi API Microservice was built with Flask -- [Repo](https://github.com/NidheeshaT/vertexai-api)
- Both of the repos were dockerized & deployed on [Google Cloud Run](https://cloud.google.com/run)
- [Final Presentation Pitch Deck](https://www.canva.com/design/DAFstFPq9mU/VrI2Bq1hMvutvrdiTBLdRw/view?utm_content=DAFstFPq9mU&utm_campaign=designshare&utm_medium=link&utm_source=publishsharelink)
> **NOTE**
> The deployments might be down right now as the Cloud access was given only during the hackathon.
## Features
- Virtually Interactive 3D Character with speech & hand gestures
- Multilingual Chat Sessions in different languages
- Follow-up Questions based on previous conversation
- Persistent chat sessions considering chat history of user
- Personalised medical query & mental health support
- Multilingual Voice Chat Support
## Architecture
## Tech Stack
- Google PaLM2 based on model chat-bison using Google vertex AI
- Deployed both client & server on Google Cloud Run
- Database on Cloud SQL (PostgreSQL)
- Google OAuth
- Deepgram STT & ElevenLabs TTS
- CI/CD using Google Cloud source repositories & Cloud build
- Google Translate
- T3 Stack: Next.js, tRPC, Prisma, Typescript
- Flask Backend API
## Screenshots
## Challenges Faced
- No finetuning available for chat-bison model
- It was our first time using & configuring GCP & Gitlab
- No Direct support of CI/CD of Gitlab on Cloud Run
- Didn't get access to Google TTS & STT due to Quota Limit
- The TTS generation & lipsyncing had a lot of latency
## Future Scope
- MedPaLM integration for better accuracy.
- Using 3D model for other usecases like kids' trainer, lifestyle expert etc.
- Break the sentences according to fullstops & punctuations before processing further (to avoid latency)
- Can extend the concept to Metaverse domain
## Team Members - t3-tribe
- Nagaraj Pandith
- Swasthik Shetty
- Nidheesha T