{"id":49740063,"url":"https://github.com/ekhodzitsky/polyvoice","last_synced_at":"2026-05-16T21:16:46.137Z","repository":{"id":355848207,"uuid":"1229884139","full_name":"ekhodzitsky/polyvoice","owner":"ekhodzitsky","description":"Speaker diarization for Rust — who spoke when, without Python. Silero VAD + WeSpeaker + AHC in a single Pipeline::run() call.","archived":false,"fork":false,"pushed_at":"2026-05-10T12:19:27.000Z","size":90023,"stargazers_count":0,"open_issues_count":8,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-05-10T18:02:41.775Z","etag":null,"topics":["audio","diarization","machine-learning","onnx","python-bindings","rust","speaker-diarization","speech","vad","voice"],"latest_commit_sha":null,"homepage":"https://docs.rs/polyvoice","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ekhodzitsky.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"docs/security/audit-2026-05-08.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-05-05T13:25:51.000Z","updated_at":"2026-05-10T12:19:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"18a0112c-8a4f-48ba-a241-66aecb86484a","html_url":"https://github.com/ekhodzitsky/polyvoice","commit_stats":null,"previous_names":["ekhodzitsky/polyvoice"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/ekhodzitsky/polyvoice","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekhodzitsky%2Fpolyvoice","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekhodzitsky%2Fpolyvoice/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekhodzitsky%2Fpolyvoice/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekhodzitsky%2Fpolyvoice/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ekhodzitsky","download_url":"https://codeload.github.com/ekhodzitsky/polyvoice/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekhodzitsky%2Fpolyvoice/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32908529,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-11T17:09:15.040Z","status":"ssl_error","status_checked_at":"2026-05-11T17:08:45.420Z","response_time":120,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audio","diarization","machine-learning","onnx","python-bindings","rust","speaker-diarization","speech","vad","voice"],"created_at":"2026-05-09T17:21:33.070Z","updated_at":"2026-05-11T19:01:07.788Z","avatar_url":"https://github.com/ekhodzitsky.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# polyvoice\n\n[![CI](https://github.com/ekhodzitsky/polyvoice/actions/workflows/ci.yml/badge.svg)](https://github.com/ekhodzitsky/polyvoice/actions/workflows/ci.yml)\n[![Crates.io](https://img.shields.io/crates/v/polyvoice)](https://crates.io/crates/polyvoice)\n[![PyPI](https://img.shields.io/pypi/v/polyvoice)](https://pypi.org/project/polyvoice)\n[![Docs.rs](https://docs.rs/polyvoice/badge.svg)](https://docs.rs/polyvoice)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n\n\u003e **Speaker diarization for Rust — who spoke when, without Python.**\n\u003e\n\u003e Silero VAD + WeSpeaker embeddings + AHC clustering in a single `Pipeline::run()` call.\n\n![CLI Demo](docs/assets/demo.gif)\n\n```\nInput:  14 seconds of two-speaker audio (16 kHz mono WAV)\nOutput: SPEAKER_00: 0.10s -  7.60s\n        SPEAKER_01: 8.10s - 14.10s\n```\n\n## Quick start\n\n### 1. Add the dependency\n\n```toml\n[dependencies]\npolyvoice = { version = \"0.5\", features = [\"onnx\"] }\n```\n\n### 2. Download models\n\n```bash\nbash scripts/download-models.sh\n# Downloads WeSpeaker ResNet34 (25 MB) and Silero VAD v5 (2.2 MB) to models/\n```\n\n### 3. Run the pipeline\n\n```rust,no_run\nuse polyvoice::{\n    Pipeline, DiarizationConfig, VadConfig,\n    FbankOnnxExtractor, SileroVad,\n};\nuse std::path::Path;\n\nfn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error\u003e\u003e {\n    // Load models\n    let extractor = FbankOnnxExtractor::new(\n        Path::new(\"models/wespeaker_resnet34.onnx\"),\n        256, // embedding dim\n        4,   // ONNX session pool size\n    )?;\n    let mut vad = SileroVad::new(Path::new(\"models/silero_vad.onnx\"), 512)?;\n\n    // Configure and run\n    let pipeline = Pipeline::new(\n        DiarizationConfig::default(),\n        VadConfig::default(),\n    );\n    let (samples, _sr) = polyvoice::wav::read_wav(Path::new(\"meeting.wav\"))?;\n    let result = pipeline.run(\u0026samples, \u0026extractor, \u0026mut vad)?;\n\n    for turn in \u0026result.turns {\n        println!(\"{}: {:.2}s - {:.2}s\", turn.speaker, turn.time.start, turn.time.end);\n    }\n    Ok(())\n}\n```\n\n## Python\n\n```bash\npip install polyvoice\n```\n\nOr build from source:\n\n```bash\ncd python\nmaturin develop --release\n```\n\n```python\nimport polyvoice\n\npipeline = polyvoice.Pipeline(\"models/\")\nturns = pipeline(\"meeting.wav\")\n\nfor turn in turns:\n    print(f\"{turn.speaker}: {turn.start:.1f}s - {turn.end:.1f}s\")\n```\n\n## CLI\n\n```bash\ncargo install polyvoice --features cli\n\npolyvoice download-models\npolyvoice diarize meeting.wav\npolyvoice diarize meeting.wav --format json\npolyvoice diarize meeting.wav --format rttm --max-speakers 4\n```\n\n## How it works\n\n```\nWAV / PCM audio (16 kHz mono)\n       |\n       v\n+-------------+     +------------------+     +---------+\n|  Silero VAD |----\u003e| WeSpeaker        |----\u003e|   AHC   |---\u003e Speaker turns\n|  (speech    |     | ResNet34         |     | cluster |\n|   regions)  |     | (256-d embed.)   |     |         |\n+-------------+     +------------------+     +---------+\n                     fbank + CMVN           cosine similarity\n                     lock-free pool         threshold merging\n```\n\n**VAD** detects speech regions, skipping silence. **WeSpeaker** extracts 256-dimensional speaker embeddings from log-mel filterbank features (80-bin, CMVN-normalized). **AHC** clusters embeddings by cosine similarity into speaker groups. The `Pipeline` wires it all together.\n\n## Comparison with pyannote\n\n| | polyvoice | pyannote |\n|---|---|---|\n| Language | Rust | Python |\n| Runtime | ONNX Runtime | PyTorch |\n| GIL-free | Yes | No |\n| Binary size | ~30 MB (with models) | ~2 GB (torch + models) |\n| Deploy | Single binary / C FFI | Python env + pip |\n| Concurrent sessions | Lock-free session pool | Thread-limited |\n| Streaming | `OnlineDiarizer` built-in | Third-party wrappers |\n\npyannote is the gold standard for accuracy. polyvoice trades some accuracy for deployment simplicity: no Python runtime, no GPU required, ~30 MB total.\n\n## Minimum Supported Rust Version (MSRV)\n\n1.85 (Rust 2024 edition).\n\n## Accuracy (DER benchmarks)\n\nEvaluated with 0.25s collar on standard diarization benchmarks:\n\n### VoxConverse (232 files, 43.5 hours — broadcast, meetings, interviews)\n\n| System | DER | Miss | FA | Confusion | Speed |\n|--------|-----|------|-----|-----------|-------|\n| **polyvoice** (AHC, t=0.45, me=2) | **~15%** | 3.9% | 3.2% | 7.9% | **10.6x RT (CPU)** |\n| pyannote 3.0 | ~11% | — | — | — | ~1x RT (GPU) |\n\n### AMI (16 meetings, 9 hours — meeting room recordings)\n\n| System | DER | Miss | FA | Confusion | Speed |\n|--------|-----|------|-----|-----------|-------|\n| **polyvoice** (AHC, t=0.45, me=2) | **~23%** | 15.4% | 3.5% | 4.1% | 7x RT (CPU) |\n| pyannote 3.0 | ~18% | — | — | — | ~1x RT (GPU) |\n| Simple i-vector + AHC | ~33% | — | — | — | — |\n\npolyvoice delivers **~80% of pyannote's accuracy at 10x the speed on CPU alone** — no GPU, no Python, ~30 MB total. The accuracy gap comes from neural end-to-end training and overlap-aware resegmentation, which polyvoice doesn't do yet.\n\n```bash\n# Reproduce benchmarks\nbash scripts/download-ami-test.sh\ncargo run --release --features cli --bin polyvoice-bench -- data/ami-test\n\nbash scripts/download-voxconverse-test.sh\ncargo run --release --features cli --bin polyvoice-bench -- data/voxconverse-test --threshold 0.4\n```\n\n## Features\n\n- **Pipeline API** — `Pipeline::run()` for one-call diarization with VAD + embeddings + clustering.\n- **Online \u0026 Offline** — `OnlineDiarizer` for real-time streaming, `OfflineDiarizer` for batch files.\n- **ONNX-powered** — WeSpeaker and ECAPA-TDNN extractors with 80-bin log-mel fbank + CMVN.\n- **Lock-free session pool** — `crossbeam-queue` backed pool for concurrent ONNX inference.\n- **Silero VAD** — integrated voice activity detection with stateful LSTM context.\n- **Overlap detection** — find regions where multiple speakers talk simultaneously.\n- **Word alignment** — assign speaker IDs to transcript words by timestamp.\n- **Python bindings** — `pip install polyvoice`, 3-line API via PyO3/maturin.\n- **CLI** — `polyvoice diarize meeting.wav` with text/json/rttm output.\n- **C FFI** — drop-in `.so`/`.dylib`/`.dll` for Go, Node.js, C++ callers.\n- **Safety verified** — Miri (memory), Loom (concurrency), cargo-fuzz (inputs), across Linux/macOS/Windows.\n\n## Configuration\n\n```rust\nuse polyvoice::{DiarizationConfig, VadConfig, SampleRate};\n\nlet config = DiarizationConfig {\n    threshold: 0.45,          // cosine similarity threshold\n    max_speakers: 64,         // hard speaker limit\n    window_secs: 1.5,         // analysis window\n    hop_secs: 0.75,           // sliding step\n    min_speech_secs: 0.25,    // discard shorter segments\n    max_gap_secs: 0.5,        // merge same-speaker gaps under 500 ms\n    min_turn_duration_secs: 1.0,  // filter turns shorter than 1s\n    min_embeddings_per_speaker: 2, // merge speakers with \u003c2 embeddings\n    sample_rate: SampleRate::new(16000).unwrap(),\n};\n\nlet vad_config = VadConfig {\n    frame_size: 512,          // Silero VAD chunk size (32 ms at 16 kHz)\n    threshold: 0.5,           // speech probability threshold\n    min_silence_ms: 300.0,    // minimum silence to split segments\n};\n```\n\n## Streaming (real-time)\n\n```rust,no_run\nuse polyvoice::{OnlineDiarizer, DiarizationConfig, DummyExtractor};\n\nlet config = DiarizationConfig::default();\nlet mut diarizer = OnlineDiarizer::new(config);\nlet extractor = DummyExtractor::new(256);\n\n// In your audio callback:\n# let chunk = vec![0.0f32; 4800];\nlet segments = diarizer.feed(\u0026chunk, \u0026extractor).unwrap();\nfor seg in segments {\n    println!(\"Speaker {:?} at {:.2}s\", seg.speaker, seg.time.start);\n}\n```\n\n## Verification\n\n| Check | Tool |\n|-------|------|\n| Unsafe memory safety | Miri (nightly CI) |\n| Concurrency correctness | Loom model-checking |\n| Input fuzzing | cargo-fuzz (4 targets) |\n| API stability | cargo-semver-checks |\n| Cross-platform | Ubuntu, macOS, Windows CI |\n| Dependency audit | cargo-audit |\n\n## Roadmap\n\n- [x] WeSpeaker + ECAPA-TDNN ONNX extractors\n- [x] Silero VAD integration\n- [x] Agglomerative hierarchical clustering (AHC)\n- [x] Pipeline API (VAD + embeddings + AHC)\n- [x] C FFI bindings\n- [x] Miri / Loom / fuzz verification\n- [x] Cross-platform CI\n- [x] Python bindings (PyO3 / maturin)\n- [x] CLI tool (`polyvoice diarize` / `download-models`)\n- [x] DER benchmarks on AMI (~23%) and VoxConverse (~15%), 0.25s collar\n- [x] Spectral clustering backend (experimental)\n- [x] Merge-small-speakers post-processing\n- [ ] PLDA scoring backend\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md).\n\n## Changelog\n\nSee [CHANGELOG.md](CHANGELOG.md).\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fekhodzitsky%2Fpolyvoice","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fekhodzitsky%2Fpolyvoice","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fekhodzitsky%2Fpolyvoice/lists"}