{"id":50541677,"url":"https://github.com/hanlinlibham/ablevoice","last_synced_at":"2026-06-03T20:30:39.939Z","repository":{"id":360828615,"uuid":"1251298619","full_name":"hanlinlibham/ablevoice","owner":"hanlinlibham","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-28T02:09:48.000Z","size":1381,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-28T04:09:22.871Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hanlinlibham.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-27T12:48:18.000Z","updated_at":"2026-05-28T02:09:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hanlinlibham/ablevoice","commit_stats":null,"previous_names":["hanlinlibham/ablevoice"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/hanlinlibham/ablevoice","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanlinlibham%2Fablevoice","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanlinlibham%2Fablevoice/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanlinlibham%2Fablevoice/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanlinlibham%2Fablevoice/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hanlinlibham","download_url":"https://codeload.github.com/hanlinlibham/ablevoice/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanlinlibham%2Fablevoice/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33878990,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-03T02:00:06.370Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-03T20:30:39.006Z","updated_at":"2026-06-03T20:30:39.920Z","avatar_url":"https://github.com/hanlinlibham.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# able-asr\n\n\u003e 本地 ASR + LLM + TTS 语音管线 + LangGraph polish agent + crash-safe\n\u003e draft 持久化。Apple Silicon Mac 上的 thin 桌面 / 浏览器 voice client。\n\u003e\n\u003e 仍是 ablework 生态的 probe(本地目录:`voice-asr-test/`),但已经\n\u003e 从一次性 demo 长成了一个独立可运行项目 — 拆了模块、配置化、有测试\n\u003e 套件、有 polish agent、有 crash recovery。\n\n## 是什么\n\n一个端到端中文语音对话回路。**按住说话 → 实时转写显示 → AI 流式生成 →\n逐句 TTS 顺序播放**。三个 stage(ASR / LLM / TTS)各支持 3 类 provider 自\n由组合,**全本地**(隐私 + 离线)或 **全云端**(质量 + 低本地资源)都行。\n\n跑在 Apple Silicon Mac(MLX 加速本地模型)+ 可选的 DashScope / ablework\n云端 backend。\n\n## 架构\n\n```\n                       ┌─────────────────────────────┐\n                       │  Client (browser UI / TUI)  │\n                       └─────────────────────────────┘\n                                ↓ /api/ws (WebSocket)\n                                ↓   binary: int16 PCM 16kHz mono (AudioWorklet)\n                                ↓   json:   start/stop/interrupt/tts/reset\n                       ┌─────────────────────────────┐\n                       │   server.py  (FastAPI)      │\n                       │   - ASR / LLM / TTS dispatch│\n                       │   - chat pipeline (LLM →    │\n                       │       sentence split → TTS) │\n                       │   - SQLite transcripts      │\n                       └─────────────────────────────┘\n            ┌───────────────────┼───────────────────┐\n            ↓                   ↓                   ↓\n        ┌───────┐           ┌───────┐           ┌───────┐\n        │  ASR  │           │  LLM  │           │  TTS  │\n        ├───────┤           ├───────┤           ├───────┤\n   mlx  │ Qwen3 │       mlx │ Qwen3 │       mlx │ Qwen3 │\n        │ -ASR  │           │ -4B   │           │ -TTS  │\n        │ MLX   │           │ MLX   │           │ Custom│\n        │       │           │       │           │ Voice │\n   云端 │ para- │     cloud │ qwen  │     cloud │ qwen3-│\n        │former │           │3.7-   │           │tts-   │\n        │realt- │           │max    │           │instr- │\n        │ime-v2 │           │       │           │uct-fl-│\n        │ (WS)  │           │       │           │ash    │\n        │       │      able │ ag-   │           │ (50+  │\n        │       │      work │ ent   │           │voices)│\n        └───────┘           └───────┘           └───────┘\n```\n\n## Quick start\n\n### 准备\n\n```bash\n# Python venv + 依赖(首次)\npython3 -m venv .venv\n.venv/bin/pip install -U pip\n.venv/bin/pip install fastapi 'uvicorn[standard]' httpx websockets \\\n  mlx-qwen3-asr mlx-audio mlx-lm \\\n  textual sounddevice numpy\n\n# 浏览器 UI(可选)\ncd demo-ui \u0026\u0026 npm install \u0026\u0026 cd ..\n```\n\n### 启动方式(四选一)\n\n```bash\n# A. 终端 TUI(textual)— SSH / headless 场景 / 快速测试\n./start-tui.sh\n\n# B. 浏览器 UI(Vite :5173 + server :8501)— 调样式时方便\n./start.sh         # 然后开 http://127.0.0.1:5173\n\n# C. 桌面 app(Tauri,推荐日常用)— 系统托盘 + 全局快捷键 ⌘⇧Space\n./start.sh \u0026                                          # 起 server\ncd demo-ui \u0026\u0026 npm run tauri:dev                       # 起 Tauri 窗口\n# 生产打包:cd demo-ui \u0026\u0026 npm run tauri:build → src-tauri/target/release/\n\n# D. 只起 server(自己写 client)\n.venv/bin/uvicorn server:app --host 127.0.0.1 --port 8501\n```\n\n启动脚本自动加载 `.env.local`(放 cloud provider 的 API key)。\n\n### 第一次跑\n\n模型按需 lazy download(首次 chat 触发):\n- ASR Qwen3-ASR-1.7B: ~5GB\n- TTS Qwen3-TTS-CustomVoice-bf16: ~4GB\n- LLM Qwen3-4B-Instruct-2507-4bit: ~2.5GB\n\n冷启动总下载 ~12GB(只一次,之后在 `~/.cache/huggingface/`)。\n\nmacOS 第一次按录音键会弹麦克风权限,允许即可。\n\n## 配置系统(`.env.local` + `voice/config.py`)\n\n**所有 secret 走 `.env.local`**(gitignored)。新人配环境抄一份模板:\n\n```bash\ncp .env.example .env.local\n# 编辑 .env.local 填:\n#   DASHSCOPE_API_KEY=sk-...    (云 ASR/LLM/TTS/Polish 共用)\n#   ABLEWORK_TOKEN=eyJ...        (ablework 后端 JWT)\n```\n\n**单 source of truth**:`voice/config.py` 的 `settings` 单例 — 43 个 env 全在这一处定义、解析、validate。typo 的 int/float 会在 startup 抛 RuntimeError;空 secret 会 warn 一句。\n\n**结构**:\n\n```\nsettings\n├── dashscope    # api_key + base_url + chat_model + asr_model + tts_model + polish_model (一个 vendor account 服务所有 cloud stage)\n├── ablework     # url + token + verify_ssl\n├── ollama       # url + model\n├── asr          # provider + mlx_model + stream_chunk_sec\n├── llm          # provider + mlx_model + system_prompt\n├── tts          # provider + mlx_model + voice + 12 个调参\n├── polish       # enabled + provider + use_polished_for_chat\n├── sentence     # 句切 4 个常量\n├── storage      # db_path / audio_dir / keep_audio\n└── warmup\n```\n\n**运行时查实际生效配置**:\n\n```bash\ncurl -s http://127.0.0.1:8501/config | jq\n# secret 被脱敏成 \"sk-4…(len=35)\" / \"eyJh…(len=3023)\" — 能验配但不泄露\n```\n\n**SYSTEM_PROMPT 支持文件**(多行中文 prompt 用 env 转义麻烦):\n\n```bash\n# .env.local\nSYSTEM_PROMPT_FILE=prompts/system.md\n```\n\n完整 env 参考见 [`.env.example`](.env.example)。\n\n## Provider 配置\n\n通过 env 切换。默认配置见 server.py 顶部,常用组合:\n\n```bash\n# 全本地(最快,~2s 首音,纯离线,隐私)\nASR_PROVIDER=mlx LLM_PROVIDER=mlx TTS_PROVIDER=mlx ./start-tui.sh\n\n# 当前默认(本地 ASR + ablework agent + 本地 TTS serena)\n./start-tui.sh\n\n# 全云端(最高质量,可选 Maia 等 ~50 voice)\nASR_PROVIDER=dashscope LLM_PROVIDER=dashscope TTS_PROVIDER=dashscope \\\n  MLX_TTS_VOICE=Maia ./start-tui.sh\n```\n\n| Env | 默认 | 可选值 |\n|---|---|---|\n| `ASR_PROVIDER` | `mlx` | `mlx` / `dashscope` |\n| `MLX_QWEN_MODEL` | `Qwen/Qwen3-ASR-1.7B` | `Qwen/Qwen3-ASR-0.6B`(更快) |\n| `LLM_PROVIDER` | `ablework` | `mlx` / `dashscope` / `ablework` / `ollama` |\n| `MLX_LLM_MODEL` | `mlx-community/Qwen3-4B-Instruct-2507-4bit` | 任何 mlx-lm 兼容模型 |\n| `DASHSCOPE_MODEL` | `qwen3.7-max` | `qwen-max` / `qwen-plus` 等 |\n| `TTS_PROVIDER` | `mlx` | `mlx` / `dashscope` |\n| `MLX_TTS_VOICE` | `serena` | 本地 9 个 / 云端 ~50 个 |\n| `MLX_TTS_INSTRUCT` | (新闻播报指令) | 任何自然语言描述,空串关闭 |\n| `DASHSCOPE_API_KEY` | (.env.local 读) | sk-xxx |\n| `ABLEWORK_TOKEN` | (.env.local 读) | ablework JWT |\n| `VOICE_INPUT_DEVICE` | (PortAudio default) | sounddevice 设备索引 |\n| `VOICE_INPUT_NAME` | — | substring 匹配设备名,蓝牙重连后仍有效 |\n| `KEEP_AUDIO` | `0` | `1` 保留上传录音到 `recordings/` |\n\n完整 env 列表在 `start-tui.sh` 顶部注释。\n\n## 实测延迟(M3 Max,warm)\n\n| 配置 | 首音(松手→听到字)| Total |\n|---|---|---|\n| 全本地小模型(0.6B/4B-MLX/0.6B-Lite) | **2.1s** | ~4.3s |\n| 本地 ASR + 云 LLM + 本地 TTS | ~4.5s | ~5.5s |\n| 全云端 + ablework agent | ~8.5s | ~16s |\n\nablework agent 5-6s 是 backend 端 RAG/tool pipeline 固有延迟,跟 voice 客户端无关。\n\n## 主要特性\n\n- **流式 ASR partial 显示**:用户还在说时实时显示识别中文字(本地用\n  `mlx_qwen3_asr.Session.init_streaming/feed_audio/finish_streaming`,云端\n  桥接 `paraformer-realtime-v2` WebSocket)\n- **句级 streaming TTS**:LLM 一边吐 token,server 按句号切分,每句独立\n  合成 + 推给客户端按序播放;首句 ~2s 可到\n- **真打断**:用户开始新录音 / 按 i 键 → server `task.cancel`,清音频队\n  列,partial assistant text **不写入** history(没听完的不该记)\n- **AudioWorklet 16kHz PCM 上行**:替代 MediaRecorder webm/opus,消除\n  ffmpeg decode 延迟,WebSocket binary frame 直传\n- **Voice 稳定化**:\n  - per-chunk RMS normalize(段间响度均一)\n  - silence trim(段间真空 178ms → 40ms)\n  - 客户端持久 OutputStream + ring buffer(消除 sd.play 重启 gap)\n  - `MLX_TTS_INSTRUCT` 平稳播报指令 + temperature 0.5 + fixed seed(同段\n    文字每次输出一致)\n- **Markdown 自动剥**:`**bold**` / `*italic*` / `` `code` `` / 链接 /\n  headers / **markdown 表格(pipes 转逗号,separator 行删)** 都剥,TTS\n  不会念出 `**` 或 `|`\n- **TUI 动效**:assistant 等首 token / TTS 合成 / mic 录音都有 braille\n  spinner 转动,自动滚到底\n- **SQLite 持久化所有转写**(空也存,便于诊断 mic 静默)\n- **Crash-safe draft 录音**:每段 recording 实时落盘 (`recordings/draft-\u003cid\u003e.pcm`)\n  + ASR partial 实时入库,WS 异常断开 / server crash 后通过\n  `GET /drafts` + `POST /drafts/\u003cid\u003e/recover` 还原成 transcript(长录音\n  1+ 分钟时关键)\n\n## 文件布局\n\n```\n~/voice-asr-test/\n├── server.py                    # FastAPI entry — lifespan + warmup + 挂路由\n├── tui.py                       # textual TUI 客户端\n├── start.sh / start-tui.sh      # 一键起 server + UI/TUI\n├── .env.local                   # API keys(gitignored)\n├── transcripts.db               # SQLite,自动创建\n├── voice/                       # 核心包 — 拆出来的模块\n│   ├── config.py                #   所有 env → typed Settings 单例 + validation\n│   ├── runtime.py               #   单线程 MLX executor + 共享 httpx\n│   ├── db.py                    #   SQLite CRUD\n│   ├── audio.py                 #   WAV pack / 句切 / markdown strip\n│   ├── chat.py                  #   chat pipeline (LLM → 句切 → TTS)\n│   ├── ws.py                    #   /ws WebSocket handler(WsSession 类)\n│   ├── http.py                  #   /health /transcribe /tts /history /chat\n│   └── providers/{base,asr,llm,tts}.py  # Protocol + 各 provider class\n├── scripts/                     # 辅助脚本\n│   ├── mic-check.py             #   独立 mic 诊断\n│   └── smoke_voice_loop.py      #   端到端 smoke(打 HTTP + WS)\n├── tests/                       # pytest 套件\n│   ├── unit/                    #   纯函数,offline,\u003c 1s\n│   ├── integration/             #   真 server + 真模型(``-m integration``)\n│   ├── fixtures/                #   test_zh*.wav / test_zh2*.aiff 测试音频\n│   ├── conftest.py              #   live_server session fixture\n│   └── README.md                #   测试规约 + 啥时候写什么\n├── demo-ui/                     # Vite + React + Tailwind UI (+ Tauri 桌面壳)\n│   ├── src/App.tsx              #   单文件 UI + useVoiceWS hook\n│   ├── public/pcm-worklet.js    #   AudioWorklet processor\n│   └── src-tauri/               #   Rust Tauri wrapper (托盘 + 全局快捷键)\n│       ├── Cargo.toml\n│       ├── src/lib.rs           #   tray icon + ⌘⇧Space global shortcut\n│       └── tauri.conf.json      #   window / bundle 配置\n├── recordings/                  # opt-in audio store(KEEP_AUDIO=1 才存)\n├── docs/done/daily_record/      # 工作日记(gitignored)\n├── ADR-001-...md                # 架构决策\n└── CLAUDE.md                    # Claude Code 协作规范 + 项目状态\n```\n\n## 测试\n\n```bash\npytest                    # unit 套件(秒级,offline)\npytest tests/integration  # 全链路(起真 server,~40s,需 .env.local cred)\n```\n\n约定 + 啥时候加测试:见 [`tests/README.md`](tests/README.md)。\n\n## 范围 / 不做的事\n\n跟 ablework 主仓的关系见 [`CLAUDE.md`](CLAUDE.md)。简短:\n\n- **不**写 OS-level hotkey / 全局通知(那是真桌面客户端,这是 browser/TUI demo)\n- **不**做用户认证(localhost only,只你自己用)\n- **不**做模型微调 / 词表扩展\n- **不**进 git(本目录全 gitignored,CLAUDE.md / ADR / 代码逻辑后续按需 port 到 ablework 主仓)\n\n## 进一步\n\n- 项目状态 + 协作规范 + 启动重启 + 常见陷阱:[`CLAUDE.md`](CLAUDE.md)\n- 架构决策:[`ADR-001-voice-client-as-thin-edge.md`](ADR-001-voice-client-as-thin-edge.md)\n- 历次工作日记:[`docs/done/daily_record/`](docs/done/daily_record/)\n- API 健康/状态:`curl http://127.0.0.1:8501/health | jq`\n\n## 已知短板\n\n- **本地 TTS 跨 chunk「语气不衔接」**:open-source mlx-audio 不暴露 KV\n  cache 跨 generate 复用,只能 stream=True 单次 generate(代价首音 +1s),\n  或上云端 qwen3-tts-instruct-flash(有 `instructions` 自然语言风格控制)\n- **ablework agent 首音 ~8s**:backend 端 RAG/tool pipeline 固有延迟,\n  voice 客户端没法在 server 端压短\n- **长表格 TTS 仍逐项念**:strip 掉 `|` 后变成 \"name, value, name,\n  value...\",真\"概括\"需要 LLM 二次重写\n- **VoiceDesign 变体没装**:1.7B-VoiceDesign-bf16 可以用自然语言描述生\n  成新 voice,3.84GB 下载没做\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhanlinlibham%2Fablevoice","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhanlinlibham%2Fablevoice","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhanlinlibham%2Fablevoice/lists"}