https://github.com/TheStageAI/TheWhisper
Optimized Whisper models for streaming and on-device use
https://github.com/TheStageAI/TheWhisper
apple-silicon coreml mlx nvidia-gpu on-device-ai real-time speech-recognition speech-to-text streaming transcription translation voice voice-ai
Last synced: about 2 months ago
JSON representation
Optimized Whisper models for streaming and on-device use
- Host: GitHub
- URL: https://github.com/TheStageAI/TheWhisper
- Owner: TheStageAI
- License: mit
- Created: 2025-10-19T18:07:02.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2026-04-23T14:01:28.000Z (about 2 months ago)
- Last Synced: 2026-04-23T15:13:41.521Z (about 2 months ago)
- Topics: apple-silicon, coreml, mlx, nvidia-gpu, on-device-ai, real-time, speech-recognition, speech-to-text, streaming, transcription, translation, voice, voice-ai
- Language: Python
- Homepage: https://thestage.ai
- Size: 2.53 MB
- Stars: 827
- Watchers: 11
- Forks: 55
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-side-quests - TheStageAI/TheWhisper - device use — faster, smaller, still accurate (Developer Tools / Python Tools)
- awesome-mlx - TheWhisper - device use (Audio & Speech)
README
# TheWhisper: High-Performance Speech-to-Text
[](LICENSE)
[](https://huggingface.co/TheStageAI/thewhisper-large-v3-turbo/)
[](#usage-deployment)
[](#usage-deployment)
## 🚀 Overview
This repository aims to share and develop the most efficient speech-to-text and text-to-speech inference solution -with a strong focus on self-hosting, cloud hosting, and on-device inference across multiple devices.
For the first release this repository provides **open-source transcription models** with **streaming inference support** and:
- Hugging Face open weights for whisper models with a flexible chunk size (original models have 30s)
- High-performance TheStage AI inference engines (NVIDIA GPU), 220 tok/s on L40s for whisper-large-v3 model
- CoreML engines for macOS / Apple Silicon with the lowest in the world power consumption for MacOS
- Local RestAPI with frontend examples using JS and Electron [see for details](electron_app/README.md)
- Electron demo app built by TheStage AI (Certified by Apple): [TheNotes for macOS](https://cdn.thestage.ai/production/cms_file_upload/1761746543-88b5430a-5897-4348-b031-8a1101352c72/The%20Notes.pkg)
- [Tutorial](https://app.thestage.ai/blog/Building-a-macOS-Note-Taker-app-on-Electron-with-TheWhisper?id=6) on building local note-taking app for macOS using Electron and TheWhisper
https://github.com/user-attachments/assets/f4d3fe7b-e2c5-42ff-a5d0-fef6afd11684
It is optimized for **low-latency**, **low power usage**, and **scalable** streaming transcription. Ideal for real-time captioning, live meetings, voice interfaces, and edge deployments.
## 📖 Table of Contents
- [✨ Features](#-features)
- [⚡ Quick Start](#-quick-start)
- [🛠️ Support Matrix](#%EF%B8%8F-support-matrix-and-system-requirements)
- [💡 Usage](#%EF%B8%8F-usage-and-deployment)
- [🖥️ Build On-Device Desktop Application for Apple](#-build-on-device-desktop-application-for-apple)
- [📊 Benchmarks](#-benchmarks)
- [🏢 Enterprise License Summary](#-enterprise-license-summary)
- [🧭 Development Status](#-development-status)
- [📝 Changelog](#-changelog-high-level)
- [🙌 Acknowledgements](#-acknowledgements)
---
## ✨ Features
- Open weights fine-tuned versions of Whisper models
- Fine-tuned models support inference with 10s, 15s, 20s and 30s
- CoreML engines for macOS and Apple Silicon, ~2W of power consumption, ~2GB RAM usage
- Optimized engines for NVIDIA GPUs through TheStage AI [ElasticModels](https://docs.thestage.ai/elastic_models/docs/source/index.html) (free for small orgs)
- Streaming implementation (NVIDIA + macOS)
- Benchmarks: latency, memory, power, and ASR accuracy (OpenASR)
- Simple Python API, examples and [tutorial](https://app.thestage.ai/blog/Building-a-macOS-Note-Taker-app-on-Electron-with-TheWhisper?id=6) of deployment for MacOS desktop app with Electron and ReactJS
.png)
For comprehensive performance and quality benchmarks see [benchmark/](benchmark/README.md).
---
## 📦 Quick start
### Clone the repository
```bash
git clone https://github.com/TheStageAI/TheWhisper.git
cd TheWhisper
```
### Install for Apple
```bash
pip install .[apple]
```
### Install for Nvidia
```bash
pip install .[nvidia]
```
### Install for Nvidia with TheStage AI optmized engines
```bash
pip install 'thestage-elastic-models[nvidia]==0.1.7' --index-url https://thestage.jfrog.io/artifactory/api/pypi/pypi-thestage-ai-production/simple --extra-index-url https://pypi.nvidia.com --extra-index-url https://pypi.org/simple
pip install .[nvidia]
pip install thestage
```
### Install for Jetson-Thor with TheStage AI optmized engines
Make sure you have `tensorrt==10.13.3.9` installed on your jetson and run:
```bash
pip install thestage-elastic-models[thor]==0.1.7 --extra-index-url https://thestage.jfrog.io/artifactory/api/pypi/pypi-thestage-ai-jetson-thor/simple -i https://pypi.jetson-ai-lab.io/sbsa/cu130/+simple/ --extra-index-url https://pypi.org
pip install .
pip install thestage
```
Then generate access token on [TheStage AI Platform](https://app.thestage.ai) in your profile and execute the following command:
```bash
thestage config set -t
```
-----
## 🏗️ Support Matrix and System Requirements
| **Feature** | **whisper-large-v3 (Nvidia)** | **whisper-large-v3 (Apple)** | **whisper-large-v3-turbo (Nvidia)** | **whisper-large-v3-turbo (Apple)** |
| --- | --- | --- | --- | --- |
| Streaming | ✅ | ✅ | ✅ | ✅ |
| Accelerated | ✅ | ✅ | ✅ | ✅ |
| Word Timestamps | ✅ | ✅ | ✅ | ✅ |
| Multilingual | ✅ | ✅ | ✅ | ✅ |
| 10s Chunk Mode | ✅ | ✅ | ✅ | ✅ |
| 15s Chunk Mode | ✅ | ✅ | ✅ | ✅ |
| 20s Chunk Mode | ✅ | ✅ | ✅ | ✅ |
| 30s Chunk Mode | ✅ | ✅ | ✅ | ✅ |
### Nvidia GPU Requirements
- **Supported GPUs:** RTX 4090, RTX 5090, L40s, H100, A100, Jetson-Thor
- **Operating System:** Ubuntu 20.04+
- **Minimum RAM:** 2.5 GB (5 GB recommended for large-v3 model)
- **CUDA Version:** 11.8 or higher
- **Driver Version:** 520.0 or higher
- **Python version**: 3.10-3.12
### Apple Silicon Requirements
- **Supported Chipsets:** M1, M1 Pro, M1 Max, M1 Ultra, M2, M2 Pro, M2 Max, M2 Ultra, M3, M3 Pro, M3 Max, M4, M4 Pro, M4 Max
- **Operating System:** macOS 15.0 (Ventura) or later, iOS 18.0 or later
- **Minimum RAM:** 2 GB (4 GB recommended for large-v3 model)
- **Python version**: 3.10-3.12
---
## ▶️ Usage and Deployment
### Apple Usage
```python
import torch
from thestage_speechkit.apple import ASRPipeline
model = ASRPipeline(
model='TheStageAI/thewhisper-large-v3-turbo',
# optimized model with ANNA
model_size='S',
chunk_length_s=10
)
# inference
result = model(
"path_to_your_audio.wav",
return_timestamps="word"
)
print(result["text"])
```
### Apple Usage with Streaming
```python
from thestage_speechkit.streaming import StreamingPipeline, MicStream, FileStream, StdoutStream
streaming_pipe = StreamingPipeline(
model='TheStageAI/thewhisper-large-v3-turbo',
# Optimized model by ANNA
model_size='S',
# Window length
chunk_length_s=10,
platform='apple',
language='en'
)
# set stride in miliseconds
mic_stream = MicStream(step_size_s=0.5)
output_stream = StdoutStream()
while True:
chunk = mic_stream.next_chunk()
if chunk is not None:
approved_text, assumption = streaming_pipe(chunk)
output_stream.write(approved_text, assumption)
else:
break
```
### Nvidia Usage (HuggingFace Transfomers)
```python
import torch
from thestage_speechkit.nvidia import ASRPipeline
model = ASRPipeline(
model='TheStageAI/thewhisper-large-v3-turbo',
# allowed: 10s, 15s, 20s, 30s
chunk_length_s=10,
# optimized TheStage AI engines
batch_size=32,
device='cuda'
)
# inference
result = model(
"path_to_your_audio.wav",
chunk_length_s=10,
generate_kwargs={'do_sample': False, 'use_cache': True}
)
print(result["text"])
```
### Nvidia Usage (TheStage AI engines)
```python
import torch
from thestage_speechkit.nvidia import ASRPipeline
model = ASRPipeline(
model='TheStageAI/thewhisper-large-v3-turbo',
# allowed: 10s, 15s, 20s, 30s
chunk_length_s=10,
# optimized TheStage AI engines
model_size='S',
batch_size=32,
device='cuda'
)
# inference
result = model(
"path_to_your_audio.wav",
chunk_length_s=10,
generate_kwargs={'do_sample': False, 'use_cache': True}
)
print(result["text"])
```
-----
## 💻 Build On-Device Desktop Application for Apple
You can build a macOS desktop app with real-time transcription. Find a simple ReactJS application here: **Link to React Frontend**
You can also download our app built using this backend here: [TheNotes for macOS](https://cdn.thestage.ai/production/cms_file_upload/1761693601-8ef0605f-a2e0-4bef-97c1-b61452e4f7dc/The%20Notes%20Package%20Oct%2028%202025.pkg)
-----
## 📊 Benchmarks
TheWhisper is a fine-tuned Whisper model that can process audio chunks of any size up to 30 seconds. Unlike the original Whisper models, it doesn't require padding audio with silence to reach 30 seconds. For quality benchmarks, we used the multilingual benchmarks [Open ASR Leaderboard](https://github.com/huggingface/open_asr_leaderboard#evaluate-a-model).
For comprehensive quality and performance benchmarks, including comparisons with other Whisper inference solutions, please refer to the [benchmark/](benchmark/README.md) directory.
---
## 🏢 Enterprise License Summary
To get commercial license for bigger number of GPUs to use TheStage AI optimized engines please contact us here: [Service request](https://app.thestage.ai/contact)
| Platform | Engine Type | Status | License |
|--------------------------|---------------------------|------------|-----------------------------------------|
| NVIDIA GPUs (CUDA) | Pytorch HF Transformers | ✅ Stable | Free |
| macOS / Apple Silicon | CoreML Engine + MLX | ✅ Stable | Free |
| NVIDIA GPUs (CUDA) | TheStage AI (Optimized) | ✅ Stable | Free ≤ 4 GPUs/year for small orgs |
----
## 🧭 Development Status
✅ OpenASR WER [benchmark](benchmark/README.md) for multiple chunk sizes
✅ Performance [benchmark](benchmark/README.md) for NVIDIA
✅ Support for L40S, H100, RTX 4090, RTX 5090
✅ Time-stamp support on Nvidia
✅ Nvidia Jetson support
☐ Streaming containers for Nvidia
☐ Ready-to-go containers for inference on Nvidia GPUs with OpenAI compatible API
☐ Speaker diarization and speaker identification
----
## 🙌 Acknowledgements
- **Silero VAD**: Used for voice activity detection in `thestage_speechkit/vad.py`. See [@snakers4](https://github.com/snakers4/silero-vad).
- **OpenAI Whisper**: Original Whisper model and pretrained checkpoints. See [@openai](https://github.com/openai/whisper).
- **Hugging Face Transformers**: Model, tokenizer, and inference utilities. See [@transformers](https://github.com/huggingface/transformers).
- **MLX community**: MLX Whisper implementation for Apple Silicon. See [@mlx-explore](https://github.com/ml-explore/mlx-examples/tree/main/whisper).