https://github.com/scf4/llmvoicepoc
Proof of concept, see README.md
https://github.com/scf4/llmvoicepoc
Last synced: 3 months ago
JSON representation
Proof of concept, see README.md
- Host: GitHub
- URL: https://github.com/scf4/llmvoicepoc
- Owner: scf4
- Created: 2024-08-13T07:42:25.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-12-18T04:39:04.000Z (over 1 year ago)
- Last Synced: 2025-01-16T21:40:41.650Z (over 1 year ago)
- Language: Python
- Homepage: https://scottinallca.ps
- Size: 28.3 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Whisper/LLM Voice Chat
A quick/messy proof of concept as an attempt at a low-latency, natural-feeling (and interruptible) voice-to-voice chat using LLMs.
## How to run
1. Clone the repo `https://github.com/scf4/shapevoice.git`
2. Install required packages `pip install -r requirements.txt`
3. Rename `.env.example` to `.env` and add your [Cartesia](https://cartesia.ai) and [Groq](https://groq.com) API keys.
4. Run `main.py`
5. **Wear headphones** (see below for limitations)
> **Note:** You will likely need to adjust some of the constants such as silence threshold etc.
## Recommended next steps
- [ ] Improve voice activity detection (VAD). Currently, it's very sensitive to background noise which Whisper infers as phrases such as "Thank you" or "Thanks for watching", etc.
- [ ] Echo cancellation to prevent the assistant from hearing itself and responding.
- [ ] Stream LLM output to TTS
- [ ] When the user interrupts, replace the assistant's response in the message history with a cut-off version at the same point
- [ ] Don't queue up multiple user messages/responses at once. Replace with messages concatenated into a single message.
- [ ] Remove noise from the audio input, adjust thresholds based on user automatically etc.
- [ ] General refactor/cleanup