{"id":50493446,"url":"https://github.com/GitJuhb/voice-typing-linux","last_synced_at":"2026-06-18T22:00:45.994Z","repository":{"id":301835042,"uuid":"1010446985","full_name":"GitJuhb/voice-typing-linux","owner":"GitJuhb","description":"Fast, accurate voice typing for Linux — IBus input method engine with streaming STT, Whisper refinement, and CUDA acceleration","archived":false,"fork":false,"pushed_at":"2026-02-10T20:41:32.000Z","size":125,"stargazers_count":9,"open_issues_count":2,"forks_count":4,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-10T23:38:53.691Z","etag":null,"topics":["accessibility","faster-whisper","linux","nixos","speech-to-text","voice-typing","wayland","whisper"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GitJuhb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-29T04:57:05.000Z","updated_at":"2026-02-10T20:41:35.000Z","dependencies_parsed_at":"2025-06-29T05:38:49.939Z","dependency_job_id":null,"html_url":"https://github.com/GitJuhb/voice-typing-linux","commit_stats":null,"previous_names":["gitjuhb/voice-typing-linux"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/GitJuhb/voice-typing-linux","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitJuhb%2Fvoice-typing-linux","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitJuhb%2Fvoice-typing-linux/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitJuhb%2Fvoice-typing-linux/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitJuhb%2Fvoice-typing-linux/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GitJuhb","download_url":"https://codeload.github.com/GitJuhb/voice-typing-linux/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitJuhb%2Fvoice-typing-linux/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34508867,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-18T02:00:06.871Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accessibility","faster-whisper","linux","nixos","speech-to-text","voice-typing","wayland","whisper"],"created_at":"2026-06-02T05:00:36.702Z","updated_at":"2026-06-18T22:00:45.985Z","avatar_url":"https://github.com/GitJuhb.png","language":"Python","funding_links":[],"categories":["By Platform"],"sub_categories":["Linux"],"readme":"# Voice Typing for Linux\n\nFast, accurate voice typing for Linux with IBus atomic text insertion, streaming STT, and optional post-commit correction. The default streaming path is now NVIDIA Riva ASR NIM with `nemotron-asr-streaming-nim`, while local Parakeet CTC remains available as the zero-service fallback. Works on Wayland and X11 — in terminals, browsers, and every app.\n\n## Features\n\n- **IBus input method engine** — Atomic text insertion via `commit_text`. No key injection lag, no garbled output in terminals. One unified path for every app.\n- **Streaming-first STT** — NVIDIA Riva ASR NIM `nemotron-asr-streaming-nim` is now the default streaming backend. Local buffered `parakeet-ctc-0.6b` remains available as the zero-service fallback, older Parakeet CTC NIM profiles remain selectable, Moonshine native remains selectable, Parakeet TDT remains available as an optional post-commit correction model, and zipformer remains available as a sherpa fallback.\n- **Immediate IBus commit** — Endpoint text commits immediately on IBus. Optional post-commit correction can replace the last utterance afterward when enabled.\n- **GPU acceleration** — TF32 Tensor Cores, cudnn benchmark mode, pinned memory transfers, model warm-up.\n- **Pre-recording buffer** — 600ms circular buffer captures speech before VAD triggers. Never miss the first word.\n- **Voice commands** — Window management, text editing, app launching, web search. Automatic dictation vs command disambiguation.\n- **Audio visualizer** — GTK4 spectrum analyzer overlay, auto-shows on speech, auto-hides on silence.\n- **Push-to-talk** — Hold or toggle modes with configurable hotkey.\n- **NixOS-ready** — Full Nix shell with all dependencies, NixOS service module included.\n\n## Quick Start\n\n```bash\ngit clone https://github.com/GitJuhb/voice-typing-linux.git\ncd voice-typing-linux\n\n# NixOS (recommended)\nnix-shell\n./voice --streaming --device cuda\n\n# Other distros\npython3 -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -r requirements.txt\npython enhanced-voice-typing.py --streaming --device cuda\n```\n\n## Architecture\n\nTwo processes communicate via Unix socket:\n\n```\n                        ┌─────────────────────────────────────────────┐\n                        │         enhanced-voice-typing.py            │\n                        │                                             │\n  Microphone ──▶ PyAudio ──▶ WebRTC VAD ──▶ Pre-Buffer (600ms)       │\n                        │         │                                   │\n                        │         ▼                                   │\n                        │   ┌──────────┐     ┌───────────────────┐    │\n                        │   │ Moonshine│     │ sherpa-onnx       │    │\n                        │   │ native   │────▶│ parakeet optional │    │\n                        │   │ (stream) │     │ (GPU/CPU)         │    │\n                        │   └────┬─────┘     └────────┬──────────┘    │\n                        │        │ partials           │ final text    │\n                        └────────┼────────────────────┼───────────────┘\n                                 │                    │\n                            Unix socket           Unix socket\n                           preedit:text           commit:text\n                                 │                    │\n                        ┌────────┴────────────────────┴───────────────┐\n                        │           ibus_voice_engine.py              │\n                        │                                             │\n                        │  IBus.Engine ──▶ update_preedit_text()      │\n                        │              ──▶ commit_text()              │\n                        │                                             │\n                        │  Keyboard passthrough (do_process_key_event │\n                        │  returns False — normal typing unaffected)  │\n                        └─────────────────────────────────────────────┘\n                                          │\n                                          ▼\n                                    Focused App\n                              (Ghostty, Firefox, etc.)\n```\n\n**Pass 1 (streaming):** The default streaming path is NVIDIA Riva ASR NIM with `nemotron-asr-streaming-nim`. Local buffered `parakeet-ctc-0.6b` remains the fallback that works without any external service. The older Parakeet CTC NIM profiles remain available as compatibility baselines. Moonshine native remains available for local true-online partials, and zipformer remains available as the sherpa true-online fallback.\n\n**Pass 2 (optional post-commit correction):** When enabled, endpoint audio can go to Parakeet TDT after the streaming text is already committed. If the correction is accepted, the last utterance is replaced in place.\n\n**Fallback:** If the IBus engine isn't running, falls back to direct uinput key injection via python-evdev (sub-millisecond), then ydotool, then xdotool.\n\n## IBus Setup\n\nThe IBus engine gives you atomic text insertion in every app — terminals, browsers, editors.\n\n### 1. Install the component\n\n```bash\nmkdir -p ~/.local/share/ibus/component\ncp voice-typing-ibus.xml ~/.local/share/ibus/component/\n```\n\nEdit the `\u003cexec\u003e` path in the XML to point to your checkout's `ibus-engine-voice-typing` script.\n\n### 2. Restart IBus and add the engine\n\n```bash\nibus restart\n# Add \"Voice Typing\" input source in GNOME Settings → Keyboard → Input Sources\n# Or via CLI:\nibus engine voice-typing\n```\n\n### 3. Run both processes\n\n```bash\n# Terminal 1: IBus engine\npython ibus_voice_engine.py\n\n# Terminal 2: Voice typing\n./voice --streaming --device cuda\n```\n\nWhen the IBus engine is running, voice typing auto-detects it and routes all text through IBus. When it's not running, key injection is used as fallback.\n\n## Usage\n\n```bash\n# Default batch mode (speak → pause → text appears)\n./voice\n\n# Streaming mode (words appear as you speak)\n./voice --streaming\n./voice --streaming --streaming-model parakeet-ctc-0.6b\n./voice --streaming --post-commit-correction --device cuda\n./voice --streaming --post-commit-correction --correction-model large-v3-turbo\n\n# Streaming model selection\n./voice --streaming --streaming-model parakeet-ctc-0.6b\nVOICE_NIM_URL=http://127.0.0.1:9000 ./voice --streaming --streaming-model nemotron-asr-streaming-nim\nVOICE_NIM_URL=http://127.0.0.1:9000 ./voice --streaming --streaming-model parakeet-ctc-0.6b-nim\nVOICE_NIM_URL=http://127.0.0.1:9000 ./voice --streaming --streaming-model parakeet-ctc-1.1b-nim\n./voice --streaming --streaming-model zipformer-en-20M\n./voice --streaming --streaming-model moonshine-tiny-streaming-en\n./voice --streaming --streaming-model moonshine-small-streaming-en\n./voice --streaming --streaming-model moonshine-medium-streaming-en\n\n# Batch Parakeet\n./voice --model parakeet-tdt-0.6b-v2 --device cuda\n\n# Audio visualizer overlay\n./voice --viz --viz-position top-right\n\n# Voice commands\n./voice --commands\n./voice --commands --command-arm --command-arm-seconds 10\n\n# Push-to-talk\n./voice --ptt --ptt-hotkey f9 --ptt-mode hold\n\n# Custom hotkey, language, model\n./voice --hotkey f11 --language es --model medium\n\n# Noise controls\n./voice --calibrate-seconds 1.0 --noise-gate --agc\n\n# List audio devices\n./voice --list-devices\n./voice --input-device \"Jabra Evolve2 30\"\n```\n\n### Pause/Resume\n\n- **X11/XWayland:** Press F12 (pynput handles it directly)\n- **Wayland:** Bind F12 in your compositor to `./voice-toggle`, or: `echo toggle | nc -U /run/user/$UID/voice-typing-$UID.sock`\n\n### Models\n\nFirst run downloads models automatically to `~/.cache/`.\n\n| Model | Size | Backend | Use Case |\n|-------|------|---------|----------|\n| tiny / base / small / medium / large-v3-turbo | 39 MB to 1.5 GB | faster-whisper | Multilingual batch / fallback |\n| parakeet-tdt-0.6b-v2 | ~300 MB | sherpa-onnx | Default English batch/post-commit correction |\n\nStreaming models:\n- `parakeet-ctc-0.6b` (default, buffered local streaming)\n- `nemotron-asr-streaming-nim` (default and recommended NVIDIA Riva ASR NIM realtime backend)\n- `parakeet-ctc-0.6b` (buffered local fallback)\n- `parakeet-ctc-0.6b-nim` (NVIDIA Riva ASR NIM realtime websocket backend)\n- `parakeet-ctc-1.1b-nim` (older Parakeet CTC NIM baseline on large cards)\n- `moonshine-medium-streaming-en` (native streaming)\n- `moonshine-small-streaming-en` (smaller native streaming)\n- `moonshine-tiny-streaming-en` (smallest native streaming)\n- `zipformer-en` (sherpa true-online fallback)\n- `zipformer-en-20M` (small sherpa true-online fallback)\n\n### NVIDIA Nemotron NIM\n\nBest GPU streaming setup on a large NVIDIA card:\n\n```bash\nexport NGC_API_KEY=\u003cyour-ngc-key\u003e\ndocker login nvcr.io\n\ndocker run -it --rm --name=nemotron-asr-streaming \\\n  --runtime=nvidia \\\n  --gpus '\"device=0\"' \\\n  --shm-size=8GB \\\n  -e NGC_API_KEY \\\n  -e NIM_HTTP_API_PORT=9000 \\\n  -e NIM_GRPC_API_PORT=50051 \\\n  -e NIM_TAGS_SELECTOR=mode=str \\\n  -p 9000:9000 \\\n  -p 50051:50051 \\\n  nvcr.io/nim/nvidia/nemotron-asr-streaming:latest\n\ncurl http://localhost:9000/v1/health/ready\nVOICE_NIM_URL=http://127.0.0.1:9000 ./voice --streaming --streaming-model nemotron-asr-streaming-nim --device cuda\n```\n\nOlder Parakeet CTC NIM profiles remain supported when you explicitly select `parakeet-ctc-0.6b-nim` or `parakeet-ctc-1.1b-nim`.\n\nThis backend talks to NIM over the official realtime websocket API:\n- `POST /v1/realtime/transcription_sessions`\n- `WS /v1/realtime?intent=transcription`\n\n## Voice Commands\n\nEnable with `--commands`. Spoken text is analyzed for command patterns — high-confidence matches execute as commands, everything else is typed as dictation.\n\n| Voice | Action |\n|-------|--------|\n| \"switch window\" | Alt+Tab |\n| \"close window\" | Alt+F4 |\n| \"select all\" / \"copy\" / \"paste\" | Ctrl+A / Ctrl+C / Ctrl+V |\n| \"undo\" / \"redo\" | Ctrl+Z / Ctrl+Shift+Z |\n| \"new line\" / \"new paragraph\" | Enter / Double Enter |\n| \"scratch that\" | Delete last transcription |\n| \"open [app]\" | Launch application |\n| \"search for [query]\" | Web search |\n| \"type [text]\" | Force dictation mode |\n\nPunctuation: \"period\", \"comma\", \"question mark\", \"exclamation mark\", etc. — inserted with smart spacing.\n\nCustom commands via `~/.config/voice-typing/commands.yaml`.\n\n## Configuration\n\nConfig file: `~/.config/voice-typing/config.yaml`\n\n```yaml\nmodel: parakeet-tdt-0.6b-v2\ndevice: cuda\nstreaming: true\nstreaming_model: nemotron-asr-streaming-nim\npost_commit_correction: false\ncorrection_model: parakeet-tdt-0.6b-v2\ncommands: true\nnoise_gate: true\nadaptive_vad: true\n```\n\nEnvironment overrides (prefix `VOICE_`): `VOICE_MODEL`, `VOICE_DEVICE`, `VOICE_HOTKEY`, `VOICE_STREAMING`, `VOICE_STREAMING_MODEL`, `VOICE_POST_COMMIT_CORRECTION`, `VOICE_CORRECTION_MODEL`, `VOICE_COMMANDS`, `VOICE_NOISE_GATE`, `VOICE_PTT`, `VOICE_LOG_FILE`, `VOICE_ADAPTIVE_VAD`, `VOICE_NIM_URL`, `VOICE_NIM_API_KEY`. Legacy `VOICE_REFINEMENT*` env vars are still accepted.\n\n## Project Structure\n\n```\nvoice-typing-linux/\n├── voice                      # Launcher script\n├── voice-toggle               # Wayland pause/resume helper\n├── enhanced-voice-typing.py   # Main STT pipeline, IBus client, streaming worker\n├── ibus_voice_engine.py       # IBus input method engine (separate process)\n├── ibus-engine-voice-typing   # IBus engine launcher script\n├── voice-typing-ibus.xml      # IBus component descriptor\n├── streaming_stt.py           # Streaming backends + offline model wrappers\n├── commands.py                # Voice command detection and execution\n├── audio_visualizer.py        # GTK4 spectrum analyzer overlay\n├── shell.nix                  # Nix environment (Python + system deps)\n├── ydotool-service.nix        # NixOS ydotool daemon module\n├── nix/voice-typing.nix       # NixOS service module\n├── systemd/                   # systemd user service template\n├── requirements.txt           # Python dependencies\n├── pyproject.toml             # Package metadata\n└── setup.py                   # Package setup\n```\n\n## Threading Model\n\nUp to 6 concurrent threads:\n\n1. **Audio callback** (PyAudio) — Non-blocking VAD + pre-buffer, queues recordings\n2. **Transcription worker** — Offline model inference, optional post-commit correction comparison\n3. **Streaming worker** — Parakeet CTC buffered streaming, Moonshine native, or zipformer partials with endpoint detection\n4. **Hotkey listener** (pynput) — Global F12 toggle\n5. **Socket listener** — Wayland fallback, accepts toggle/pause/resume\n6. **Visualizer** (GTK4) — FFT spectrum overlay at ~30fps\n\n## Troubleshooting\n\n### No audio input\n```bash\n# Check PipeWire sources\nwpctl status | grep -A5 Sources\nwpctl set-default \u003cdevice-id\u003e  # Set correct mic\n\n# Test recording\narecord -d 5 test.wav \u0026\u0026 aplay test.wav\n```\n\n### IBus engine not connecting\n```bash\n# Check if engine is registered\nibus list-engine | grep voice\n\n# Restart IBus\nibus restart\n\n# Verify socket exists\nls /run/user/$UID/voice-typing-ibus-$UID.sock\n```\n\n### Text not appearing (Wayland)\n```bash\n# Check if uinput is accessible (fallback mode)\nls -la /dev/uinput\nsudo usermod -aG input $USER  # Then logout/login\n```\n\n## Technical Details\n\n- **Speech Recognition:** sherpa-onnx Parakeet TDT (default) or Whisper via faster-whisper\n- **Streaming STT:** Parakeet CTC by default, with Moonshine native and zipformer available as alternatives\n- **Text Insertion:** IBus commit_text (primary), evdev uinput (fallback), ydotool/xdotool (legacy)\n- **Audio:** PyAudio + PortAudio, 16kHz mono, 20ms chunks\n- **VAD:** WebRTC Voice Activity Detection (aggressiveness 2)\n- **Pre-buffer:** 600ms (30 chunks), post-silence: 800ms (40 chunks)\n- **GPU:** TF32 Tensor Cores, cudnn benchmark, 90% VRAM allocation, pinned memory\n\n## License\n\nMIT License — see [LICENSE](LICENSE)\n\n## Acknowledgments\n\n- [OpenAI Whisper](https://github.com/openai/whisper) — speech recognition model\n- [faster-whisper](https://github.com/guillaumekln/faster-whisper) — CTranslate2 optimized inference\n- [Moonshine Voice](https://github.com/moonshine-ai/moonshine) — native streaming speech recognition\n- [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) — streaming and offline speech recognition\n- [IBus](https://github.com/ibus/ibus) — intelligent input bus for Linux\n- [RealtimeSTT](https://github.com/KoljaB/RealtimeSTT) — pre-buffer technique inspiration\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGitJuhb%2Fvoice-typing-linux","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FGitJuhb%2Fvoice-typing-linux","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGitJuhb%2Fvoice-typing-linux/lists"}