{"id":28225787,"url":"https://github.com/zhima-mochi/whisper-v3-server","last_synced_at":"2026-04-13T22:34:05.970Z","repository":{"id":290015404,"uuid":"866850860","full_name":"Zhima-Mochi/whisper-v3-server","owner":"Zhima-Mochi","description":"A robust backend server for audio processing, delivering high-accuracy transcription and speaker diarization. Powered by Whisper for speech-to-text and Pyannote for speaker segmentation, wrapped in a clean, maintainable architecture based on Domain-Driven Design (DDD) and Hexagonal Architecture.","archived":false,"fork":false,"pushed_at":"2025-05-06T15:41:47.000Z","size":1858,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-26T05:59:17.981Z","etag":null,"topics":["audio-processing","domain-driven-design","fastapi","ports-and-adapters-architecture","pyannote","speaker-diarization","speech-recognition","speech-to-text","whisper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Zhima-Mochi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-03T01:39:13.000Z","updated_at":"2025-12-25T23:38:45.000Z","dependencies_parsed_at":"2025-06-13T00:32:19.585Z","dependency_job_id":"521110fe-f6ad-4ce9-8f29-b43b2d24b8fc","html_url":"https://github.com/Zhima-Mochi/whisper-v3-server","commit_stats":null,"previous_names":["zhima-mochi/whisper-v3-server"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Zhima-Mochi/whisper-v3-server","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zhima-Mochi%2Fwhisper-v3-server","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zhima-Mochi%2Fwhisper-v3-server/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zhima-Mochi%2Fwhisper-v3-server/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zhima-Mochi%2Fwhisper-v3-server/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Zhima-Mochi","download_url":"https://codeload.github.com/Zhima-Mochi/whisper-v3-server/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zhima-Mochi%2Fwhisper-v3-server/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31774060,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-13T20:17:16.280Z","status":"ssl_error","status_checked_at":"2026-04-13T20:17:08.216Z","response_time":93,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audio-processing","domain-driven-design","fastapi","ports-and-adapters-architecture","pyannote","speaker-diarization","speech-recognition","speech-to-text","whisper"],"created_at":"2025-05-18T11:09:38.858Z","updated_at":"2026-04-13T22:34:05.953Z","avatar_url":"https://github.com/Zhima-Mochi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Whisper-v3 Server: Transcription \u0026 Diarization API\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\nA robust backend server for audio processing, delivering **high-accuracy transcription** and **speaker diarization**.  \nPowered by **Whisper** for speech-to-text and **Pyannote** for speaker segmentation, wrapped in a **clean, maintainable** architecture based on **Domain-Driven Design (DDD)** and **Hexagonal Architecture**.\n\n---\n\n## ✨ Key Features\n\n- **High-Accuracy Transcription:** Powered by OpenAI's Whisper models.\n- **Speaker Diarization:** Identify *who* spoke *when* using Pyannote models.\n- **Segmented Results:** Provides speaker-separated transcriptions with precise timestamps.\n- **Asynchronous Workflow:** Upload audio first, transcribe later using a `clip_id`.\n- **Clean Architecture:** Follows DDD and Hexagonal (Ports \u0026 Adapters) principles for scalability and maintainability.\n- **Configurable Models:** Easily switch between Whisper/Pyannote models via environment variables.\n\n---\n\n## 🏛️ Architecture Overview\n\nThis project implements a strict **Hexagonal Architecture** (Ports \u0026 Adapters) with **Domain-Driven Design**:\n\n| Layer | Responsibility | Key Components |\n|:-----|:---------------|:--------------|\n| **Domain** | Core business entities, interfaces (ports), and business rules | `AudioClip`, `SpeakerSegment`, `TranscriptionText`, `DiarizationPort`, `TranscriptionPort` |\n| **Application** | Orchestrates use cases by combining domain logic | `TranscribeAudioUseCase`, `StoreAudioUseCase` |\n| **Adapters** | Input/output adapters implementing domain ports | Input: `FastAPI routers`, Output: `ChunkedDiarizationService`, `WhisperTranscriptionService` |\n| **Infrastructure** | Technical implementations and DI container | `DIContainer`, repository implementations, model providers |\n\nKey architectural concepts implemented:\n\n- **Dependency Inversion:** All dependencies flow inward toward the domain\n- **Dependency Injection:** Services injected via FastAPI's dependency system\n- **Ports \u0026 Adapters:** Clean separation through interfaces (ports) and implementations (adapters)\n- **Single Responsibility:** Each component has exactly one reason to change\n\nThis structure enables:\n- ✅ **Testability:** Mock any external system through port interfaces\n- ✅ **Maintainability:** Change implementations without affecting business logic\n- ✅ **Flexibility:** Swap out infrastructure components with minimal impact\n\n---\n\n## 🚀 Getting Started\n\n### Prerequisites\n\n- Python 3.10+\n- [Poetry](https://python-poetry.org/) for dependency management\n- A Hugging Face account and API Token (required for Pyannote models)\n\n---\n\n### Installation \u0026 Setup\n\n1. **Clone the repository:**\n    ```bash\n    git clone https://github.com/Zhima-Mochi/whisper-v3-server.git\n    cd whisper-v3-server\n    ```\n\n2. **Configure environment variables:**\n    ```bash\n    cp .env.example .env\n    ```\n    Edit `.env` and add your Hugging Face token:\n    ```dotenv\n    HUGGINGFACE_AUTH_TOKEN=hf_YOUR_SECRET_TOKEN\n    ```\n\n3. **Install dependencies:**\n    ```bash\n    poetry install\n    ```\n\n4. **Run the application:**\n    ```bash\n    poetry run uvicorn app:app --reload --host 0.0.0.0 --port 8000\n    ```\n    ➔ API available at `http://localhost:8000`\n\n---\n\n### Running with Docker\n\n1. **Build the image:**\n    ```bash\n    docker build -t whisper-v3-server .\n    ```\n\n2. **Run the container:**\n    ```bash\n    docker run -p 8000:8000 \\\n        -e HUGGINGFACE_AUTH_TOKEN=your_token_here \\\n        -v $(pwd)/audio_data:/tmp/whisper_v3_server_storage \\\n        --name whisper-v3-server \\\n        whisper-v3-server\n    ```\n    ➔ API available at `http://localhost:8000`\n\n---\n\n## 📡 API Endpoints\n\nAll endpoints are under `/api`.\n\n### Audio Management\n\n| Method | Endpoint | Description |\n|:-------|:---------|:------------|\n| `POST` | `/api/audio` | Upload audio file and receive `clip_id` |\n| `GET` | `/api/audio/{clip_id}` | Get information about a stored audio clip |\n| `DELETE` | `/api/audio/{clip_id}` | Delete an audio clip and its transcription |\n\n### Transcription \u0026 Diarization\n\n| Method | Endpoint | Description |\n|:-------|:---------|:------------|\n| `POST` | `/api/transcribe?clip_id={clip_id}` | Process audio with transcription \u0026 diarization |\n| `POST` | `/api/transcribe/stream?clip_id={clip_id}` | Stream results as they're processed |\n| `GET` | `/api/transcription/{clip_id}` | Get stored transcription results |\n| `GET` | `/api/transcription/stream/{clip_id}` | Stream stored transcription results |\n| `DELETE` | `/api/transcription/{clip_id}` | Delete transcription for a clip |\n\n### Example Responses\n\n**Upload Audio**\n```json\n{\n  \"clip_id\": \"550e8400-e29b-41d4-a716-446655440000\",\n  \"message\": \"File uploaded successfully. Use this clip_id with the /api/transcribe endpoint.\"\n}\n```\n\n**Transcribe Audio**\n```json\n{\n  \"segments\": [\n    {\n      \"id\": \"a1b2c3d4-e5f6-7890-abcd-ef1234567890\",\n      \"audio_clip_id\": \"550e8400-e29b-41d4-a716-446655440000\",\n      \"start\": 0.0,\n      \"end\": 2.5,\n      \"speaker_label\": \"SPEAKER_01\",\n      \"text\": \"Hello, how are you today?\"\n    }\n    // Additional segments...\n  ]\n}\n```\n\n---\n\n## ⚙️ Configuration\n\nSet via `.env` or environment variables:\n\n| Variable | Description | Default | Required |\n|:---------|:-------------|:--------|:--------|\n| `HUGGINGFACE_AUTH_TOKEN` | Hugging Face token for Pyannote models | `None` | ✅ |\n| `PYANNOTE_MODEL` | Model path for speaker diarization | `pyannote/speaker-diarization` | |\n| `WHISPER_MODEL` | Model path for transcription | `openai/whisper-large-v3` | |\n| `AUDIO_STORAGE_PATH` | Path to store uploaded audio | `/tmp/whisper_v3_server_storage` | |\n| `TRANSCRIPTION_STORAGE_PATH` | Path to store transcription results | `/tmp/whisper_v3_server_storage/transcription_texts` | |\n| `APP_HOST` | Host to bind the API server | `0.0.0.0` | |\n| `APP_PORT` | Port to bind the API server | `8000` | |\n\n---\n\n## 🛠️ Technology Stack\n\n- **API Framework:** FastAPI\n- **Transcription:** OpenAI Whisper\n- **Speaker Diarization:** Pyannote Audio\n- **Dependency Management:** Poetry\n- **Containerization:** Docker\n\n---\n\n## 📜 License\n\nThis project is licensed under the [MIT License](https://opensource.org/licenses/MIT).\n\n## 📌 Todo\n\n| Done | Priority | Code  | Milestone                           | Purpose \u0026 Key Actions                                                                 |\n|------|----------|-------|--------------------------------------|----------------------------------------------------------------------------------------|\n| ✔    | **1**    | **C-1** | **Max out RTX 2060 single-GPU performance** | *Faster-Whisper small FP16 / int8_float16* → quantize first, then compare baseline; implement singleton model |\n| ⬜    | **2**    | **B-1** | **WebSocket Streaming MVP**         | Add `/ws/stream`: 500 ms Opus frame → Whisper → `send_json`; 10 s ping/heartbeat      |\n| ⬜    | **3**    | **F-1** | **Monitoring + Rate Limiting**      | Prometheus GPU/latency metrics, IP concurrency limit, timeout / 429 response          |\n| ⬜    | **4**    | **D-1** | **Silero-VAD pre-segmentation**     | Silence \u003e 600 ms → flush; 0.2 s overlap → save 20% GPU time                           |\n| ⬜    | **5**    | **B-2** | **HTTP/2 NDJSON Streaming**         | Change `/transcribe/stream` to `application/x-ndjson` + heartbeat lines              |\n| ⬜    | **6**    | **A-2** | **Optional Diarization**            | Add `diarize=true/false` query param; skip Pyannote if not needed                     |\n| ⬜    | **7**    | **C-2** | **GPU↔CPU Pipeline**                | Whisper on GPU → `asyncio.Queue` → Pyannote on CPU; GPU can proceed immediately       |\n| ⬜    | **8**    | **H-1~4** | **Dual-GPU management + Round-Robin** | Scan with NVML, create ModelPool per GPU, load-balanced GPU selection; support 2x 2060/3060 |\n| ⬜    | **9**    | **A-1** | **Single-step API**                 | Add `/upload+transcribe` endpoint with webhook callback; simplify client usage        |\n| ⬜    | **10**   | **H-5~6** | **Run Pyannote on GPU2 / parallel pipeline** | Load Pyannote on idle second GPU; true parallel speaker diarization + transcription   |\n| ⬜    | **11**   | **D-2** | **Incremental output algorithm**    | Only send \"new words\" to avoid flickering on frontend                                 |\n| ⬜    | **12**   | **E-1** | **Dual-model real-time + accuracy** | Use tiny model for 0.5s partial, small model for 30s final → overwrite result         |\n| ⬜    | **13**   | **H-7~8** | **Batch inference \u0026 config-driven pipeline** | Batch=4 under high concurrency; move thresholds to `.env`                             |\n| ⬜    | **14**   | **F-2** | **Opus-compressed streaming**       | Frontend sends `ogg/opus`, backend handles decoding                                   |\n| ⬜    | **15**   | **G-1~2** | **Disconnection recovery / resume \u0026 multiprocessing** | Support offset retransmit, `uvicorn --workers 2` + `CUDA_VISIBLE_DEVICES`            |\n| ⬜    | **16**   | **H-9~10** | **Monitoring dashboard + Horizon** | Grafana panels for concurrency / GPU heat; complete horizontal scaling                |","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhima-mochi%2Fwhisper-v3-server","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhima-mochi%2Fwhisper-v3-server","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhima-mochi%2Fwhisper-v3-server/lists"}