https://github.com/lloydzhou/llm-realtime

https://github.com/lloydzhou/llm-realtime

Last synced: 27 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/lloydzhou/llm-realtime
• Owner: lloydzhou
• License: mit
• Created: 2024-11-11T11:06:25.000Z (about 2 months ago)
• Default Branch: master
• Last Pushed: 2024-11-11T11:56:28.000Z (about 2 months ago)
• Last Synced: 2024-11-11T12:19:31.943Z (about 2 months ago)
• Language: Python
• Size: 5.86 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# llm-realtime

## start server

```

pip install tornado nanoid litellm 

export OPENAI_API_KEY=xxxx

python main.py

```

## using Azure-Samples/aoai-realtime-audio-sdk

```

Azure-Samples/aoai-realtime-audio-sdk/tree/main

Azure-Samples/aoai-realtime-audio-sdk/tree/main

clientRef.current = new RTClient(

  {key: ''},

  {model: 'gpt-4o-mini', endpoint: new URL('ws://localhost:8888/realtime')},

)

```

## TODO

#### 语音输入

1. 通过端点检测[fsmn-vad](https://github.com/lovemefan/fsmn-vad)拿到开始时间和结束时间

2. 使用两个时间调用语音识别(whisper/whisper.cpp/SenseVoice/PaddleSpeech)结合时间拿到文本

3. 拿到文本之后调用LLM API

#### 语音输出

1. 传统tts/CosyVoice/...