{"id":34899017,"url":"https://github.com/istupakov/onnx-asr","last_synced_at":"2025-12-26T08:02:56.982Z","repository":{"id":289175818,"uuid":"963929810","full_name":"istupakov/onnx-asr","owner":"istupakov","description":"Automatic Speech Recognition in Python using ONNX models","archived":false,"fork":false,"pushed_at":"2025-12-21T17:58:06.000Z","size":680,"stargazers_count":184,"open_issues_count":11,"forks_count":13,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-12-23T05:04:50.993Z","etag":null,"topics":["asr","conformer","kaldi","onnx","python","speech-recognition","speech-to-text","stt","whisper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/istupakov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-04-10T12:34:34.000Z","updated_at":"2025-12-23T03:47:04.000Z","dependencies_parsed_at":"2025-07-02T15:31:36.048Z","dependency_job_id":null,"html_url":"https://github.com/istupakov/onnx-asr","commit_stats":null,"previous_names":["istupakov/onnx-asr"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/istupakov/onnx-asr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/istupakov%2Fonnx-asr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/istupakov%2Fonnx-asr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/istupakov%2Fonnx-asr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/istupakov%2Fonnx-asr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/istupakov","download_url":"https://codeload.github.com/istupakov/onnx-asr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/istupakov%2Fonnx-asr/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28049582,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-26T02:00:06.189Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asr","conformer","kaldi","onnx","python","speech-recognition","speech-to-text","stt","whisper"],"created_at":"2025-12-26T08:02:43.073Z","updated_at":"2025-12-26T08:02:56.972Z","avatar_url":"https://github.com/istupakov.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ONNX ASR\n\n[![PyPI - Version](https://img.shields.io/pypi/v/onnx-asr)](https://pypi.org/project/onnx-asr)\n[![PyPI - Downloads](https://img.shields.io/pypi/dm/onnx-asr)](https://pypi.org/project/onnx-asr)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/onnx-asr)](https://pypi.org/project/onnx-asr)\n[![PyPI - Types](https://img.shields.io/pypi/types/onnx-asr)](https://pypi.org/project/onnx-asr)\n[![GitHub - License](https://img.shields.io/github/license/istupakov/onnx-asr)](https://github.com/istupakov/onnx-asr/blob/main/LICENSE)\n[![GitHub - CI](https://github.com/istupakov/onnx-asr/actions/workflows/python-package.yml/badge.svg)](https://github.com/istupakov/onnx-asr/actions/workflows/python-package.yml)\n[![GitHub - Release Date](https://img.shields.io/github/release-date/istupakov/onnx-asr)](https://github.com/istupakov/onnx-asr/releases/latest)\n\n[![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-xl-dark.svg)](https://istupakov-onnx-asr.hf.space/)\n\n**onnx-asr** is a Python package for Automatic Speech Recognition using ONNX models. It's a simple and fast pure Python package with minimal dependencies (PyTorch, Transformers, and FFmpeg are not required):\n\n[![numpy](https://img.shields.io/badge/numpy-required-blue?logo=numpy)](https://pypi.org/project/numpy/)\n[![onnxruntime](https://img.shields.io/badge/onnxruntime-required-blue?logo=onnx)](https://pypi.org/project/onnxruntime/)\n[![huggingface-hub](https://img.shields.io/badge/huggingface--hub-optional-blue?logo=huggingface)](https://pypi.org/project/huggingface-hub/)\n\n\u003e [!TIP]\n\u003e Supports **Parakeet v2 (En) / v3 (Multilingual)**, **Canary v2 (Multilingual)** and **GigaAM v2/v3 (Ru)** models!\n\nThe **onnx-asr** package supports many modern ASR [models](#supported-models-architectures) and the following features:\n * Works on a wide range of devices, from small IoT devices to servers with powerful GPUs ([benchmarks](#benchmarks))\n * Runs on Windows, Linux, and macOS on x86 or Arm CPUs and can use CUDA, TensorRT, CoreML, ROCm, and DirectML\n * NumPy versions from 1.21.6 to 2.4+ and Python versions from 3.10 to 3.14 are supported\n * Loading models from Hugging Face or local folders (including quantized versions)\n * Accepts WAV files or NumPy arrays (built-in support for reading and resampling files)\n * Custom models (if their architecture is supported)\n * Batch processing\n * (experimental) Longform recognition using VAD (Voice Activity Detection)\n * (experimental) Returns token timestamps and logprobs\n * Simple CLI\n * Online demo in [HF Spaces](https://istupakov-onnx-asr.hf.space/)\n\n## Supported models architectures\n\nThe package supports the following modern ASR model architectures ([comparison](#comparison-with-original-implementations) with original implementations):\n* Nvidia NeMo Conformer/FastConformer/Parakeet/Canary (with CTC, RNN-T, TDT and Transformer decoders)\n* Kaldi Icefall Zipformer (with stateless RNN-T decoder) including Alpha Cephei Vosk 0.52+\n* Sber GigaAM v2/v3 (with CTC and RNN-T decoders, including E2E versions)\n* T-Tech T-one (with CTC decoder, no streaming support yet)\n* OpenAI Whisper\n\nWhen saving these models in ONNX format, usually only the encoder and decoder are saved. To run them, the corresponding preprocessor and decoding must be implemented. Therefore, the package contains these implementations for all supported models:\n* Log-mel spectrogram preprocessors\n* Greedy search decoding\n\n## Installation\n\nThe package can be installed from [PyPI](https://pypi.org/project/onnx-asr/):\n\n1. With CPU `onnxruntime` and `huggingface-hub`:\n```shell\npip install onnx-asr[cpu,hub]\n```\n\n2. With `onnxruntime` for NVIDIA GPUs and `huggingface-hub`:\n```shell\npip install onnx-asr[gpu,hub]\n```\n\n\u003e [!IMPORTANT]\n\u003e First, you need to install the [required](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements) version of CUDA / TensorRT.\n\nYou can alse install `onnxruntime` dependencies and TensorRT via Pip:\n```shell\npip install onnxruntime-gpu[cuda,cudnn] tensorrt-cu12-libs\n```\n\n3. Without `onnxruntime` and `huggingface-hub` (if you already have some version of `onnxruntime` installed and prefer to download the models yourself):\n```shell\npip install onnx-asr\n```\n\nTo install latest version `onnx-asr` from **GitHub**, use `pip` (or `uv pip`):\n```shell\npip install git+https://github.com/istupakov/onnx-asr\n```\n\n## Usage examples\n\n### Load ONNX model from Hugging Face\n\nLoad ONNX model from Hugging Face and recognize wav file:\n```py\nimport onnx_asr\nmodel = onnx_asr.load_model(\"nemo-parakeet-tdt-0.6b-v3\")\nprint(model.recognize(\"test.wav\"))\n```\n\n\u003e [!IMPORTANT]\n\u003e Supported wav file formats: PCM_U8, PCM_16, PCM_24 and PCM_32 formats. For other formats, you either need to convert them first, or use a library that can read them into a numpy array.\n\n#### Supported model names:\n* `gigaam-v2-ctc` for Sber GigaAM v2 CTC ([origin](https://github.com/salute-developers/GigaAM), [onnx](https://huggingface.co/istupakov/gigaam-v2-onnx))\n* `gigaam-v2-rnnt` for Sber GigaAM v2 RNN-T ([origin](https://github.com/salute-developers/GigaAM), [onnx](https://huggingface.co/istupakov/gigaam-v2-onnx))\n* `gigaam-v3-ctc` for Sber GigaAM v3 CTC ([origin](https://github.com/salute-developers/GigaAM), [onnx](https://huggingface.co/istupakov/gigaam-v3-onnx))\n* `gigaam-v3-rnnt` for Sber GigaAM v3 RNN-T ([origin](https://github.com/salute-developers/GigaAM), [onnx](https://huggingface.co/istupakov/gigaam-v3-onnx))\n* `gigaam-v3-e2e-ctc` for Sber GigaAM v3 E2E CTC ([origin](https://github.com/salute-developers/GigaAM), [onnx](https://huggingface.co/istupakov/gigaam-v3-onnx))\n* `gigaam-v3-e2e-rnnt` for Sber GigaAM v3 E2E RNN-T ([origin](https://github.com/salute-developers/GigaAM), [onnx](https://huggingface.co/istupakov/gigaam-v3-onnx))\n* `nemo-fastconformer-ru-ctc` for Nvidia FastConformer-Hybrid Large (ru) with CTC decoder ([origin](https://huggingface.co/nvidia/stt_ru_fastconformer_hybrid_large_pc), [onnx](https://huggingface.co/istupakov/stt_ru_fastconformer_hybrid_large_pc_onnx))\n* `nemo-fastconformer-ru-rnnt` for Nvidia FastConformer-Hybrid Large (ru) with RNN-T decoder ([origin](https://huggingface.co/nvidia/stt_ru_fastconformer_hybrid_large_pc), [onnx](https://huggingface.co/istupakov/stt_ru_fastconformer_hybrid_large_pc_onnx))\n* `nemo-parakeet-ctc-0.6b` for Nvidia Parakeet CTC 0.6B (en) ([origin](https://huggingface.co/nvidia/parakeet-ctc-0.6b), [onnx](https://huggingface.co/istupakov/parakeet-ctc-0.6b-onnx))\n* `nemo-parakeet-rnnt-0.6b` for Nvidia Parakeet RNNT 0.6B (en) ([origin](https://huggingface.co/nvidia/parakeet-rnnt-0.6b), [onnx](https://huggingface.co/istupakov/parakeet-rnnt-0.6b-onnx))\n* `nemo-parakeet-tdt-0.6b-v2` for Nvidia Parakeet TDT 0.6B V2 (en) ([origin](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2), [onnx](https://huggingface.co/istupakov/parakeet-tdt-0.6b-v2-onnx))\n* `nemo-parakeet-tdt-0.6b-v3` for Nvidia Parakeet TDT 0.6B V3 (multilingual) ([origin](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3), [onnx](https://huggingface.co/istupakov/parakeet-tdt-0.6b-v3-onnx))\n* `nemo-canary-1b-v2` for Nvidia Canary 1B V2 (multilingual) ([origin](https://huggingface.co/nvidia/canary-1b-v2), [onnx](https://huggingface.co/istupakov/canary-1b-v2-onnx))\n* `whisper-base` for OpenAI Whisper Base exported with onnxruntime ([origin](https://huggingface.co/openai/whisper-base), [onnx](https://huggingface.co/istupakov/whisper-base-onnx))\n* `alphacep/vosk-model-ru` for Alpha Cephei Vosk 0.54-ru ([origin](https://huggingface.co/alphacep/vosk-model-ru))\n* `alphacep/vosk-model-small-ru` for Alpha Cephei Vosk 0.52-small-ru ([origin](https://huggingface.co/alphacep/vosk-model-small-ru))\n* `t-tech/t-one` for T-Tech T-one ([origin](https://huggingface.co/t-tech/T-one))\n* `onnx-community/whisper-tiny`, `onnx-community/whisper-base`, `onnx-community/whisper-small`, `onnx-community/whisper-large-v3-turbo`, etc. for OpenAI Whisper exported with Hugging Face optimum ([onnx-community](https://huggingface.co/onnx-community?search_models=whisper))\n\n\u003e [!IMPORTANT]\n\u003e Some long-ago converted `onnx-community` models have a broken `fp16` precision version.\n\nExample with `soundfile`:\n```py\nimport onnx_asr\nimport soundfile as sf\n\nmodel = onnx_asr.load_model(\"nemo-parakeet-tdt-0.6b-v3\")\n\nwaveform, sample_rate = sf.read(\"test.wav\", dtype=\"float32\")\nmodel.recognize(waveform, sample_rate=sample_rate)\n```\n\nBatch processing is also supported:\n```py\nimport onnx_asr\nmodel = onnx_asr.load_model(\"nemo-parakeet-tdt-0.6b-v3\")\nprint(model.recognize([\"test1.wav\", \"test2.wav\", \"test3.wav\", \"test4.wav\"]))\n```\n\nMost models have a quantized versions:\n```py\nimport onnx_asr\nmodel = onnx_asr.load_model(\"nemo-parakeet-tdt-0.6b-v3\", quantization=\"int8\")\nprint(model.recognize(\"test.wav\"))\n```\n\nReturn tokens, timestamps and logprobs:\n```py\nimport onnx_asr\nmodel = onnx_asr.load_model(\"nemo-parakeet-tdt-0.6b-v3\").with_timestamps()\nprint(model.recognize(\"test1.wav\"))\n```\n\n### TensorRT\n\nRunning ONNX model on the TensorRT provider and fp16 precision:\n```py\nimport onnx_asr\nimport tensorrt_libs\n\nproviders = [\n    (\n        \"TensorrtExecutionProvider\",\n        {\n            \"trt_max_workspace_size\": 6 * 1024**3,\n            \"trt_fp16_enable\": True,\n        },\n    )\n]\nmodel = onnx_asr.load_model(\"nemo-parakeet-tdt-0.6b-v3\", providers=providers)\nprint(model.recognize(\"test.wav\"))\n```\n\n### VAD\n\nLoad VAD ONNX model from Hugging Face and recognize wav file:\n```py\nimport onnx_asr\nvad = onnx_asr.load_vad(\"silero\")\nmodel = onnx_asr.load_model(\"nemo-parakeet-tdt-0.6b-v3\").with_vad(vad)\nfor res in model.recognize(\"test.wav\"):\n    print(res)\n```\n\n\u003e [!NOTE]\n\u003e You will most likely need to adjust VAD parameters to get the correct results.\n\n#### Supported VAD names:\n* `silero` for Silero VAD ([origin](https://github.com/snakers4/silero-vad), [onnx](https://huggingface.co/onnx-community/silero-vad))\n\n### CLI\n\nPackage has simple CLI interface\n```shell\nonnx-asr nemo-parakeet-tdt-0.6b-v3 test.wav\n```\n\nFor full usage parameters, see help:\n```shell\nonnx-asr -h\n```\n\n### Gradio\n\nCreate simple web interface with Gradio:\n```py\nimport onnx_asr\nimport gradio as gr\n\nmodel = onnx_asr.load_model(\"nemo-parakeet-tdt-0.6b-v3\")\n\ndef recognize(audio):\n    if not audio:\n        return None\n\n    sample_rate, waveform = audio\n    waveform = waveform / 2**15\n    if waveform.ndim == 2:\n        waveform = waveform.mean(axis=1)\n    return model.recognize(waveform, sample_rate=sample_rate)\n\ndemo = gr.Interface(fn=recognize, inputs=\"audio\", outputs=\"text\")\ndemo.launch()\n```\n\n### Load ONNX model from local directory\n\nLoad ONNX model from local directory and recognize wav file:\n```py\nimport onnx_asr\nmodel = onnx_asr.load_model(\"nemo-parakeet-tdt-0.6b-v3\", \"models/parakeet-v3\")\nprint(model.recognize(\"test.wav\"))\n```\n\n\u003e [!NOTE]\n\u003e If the directory does not exist, it will be created and the model will be loaded into it.\n\n### Load a custom ONNX model from Hugging Face\n\nLoad the Canary 180M Flash model from Hugging Face [repo](https://huggingface.co/istupakov/canary-180m-flash-onnx) and recognize the wav file:\n```py\nimport onnx_asr\nmodel = onnx_asr.load_model(\"istupakov/canary-180m-flash-onnx\")\nprint(model.recognize(\"test.wav\"))\n```\n\n#### Supported model types:\n* All models from [supported model names](#supported-model-names)\n* `nemo-conformer-ctc` for NeMo Conformer/FastConformer/Parakeet with CTC decoder\n* `nemo-conformer-rnnt` for NeMo Conformer/FastConformer/Parakeet with RNN-T decoder\n* `nemo-conformer-tdt` for NeMo Conformer/FastConformer/Parakeet with TDT decoder\n* `nemo-conformer-aed` for NeMo Canary with Transformer decoder\n* `kaldi-rnnt` or `vosk` for Kaldi Icefall Zipformer with stateless RNN-T decoder\n* `whisper-ort` for Whisper (exported with [onnxruntime](#openai-whisper-with-onnxruntime-export))\n* `whisper` for Whisper (exported with [optimum](#openai-whisper-with-optimum-export))\n\n## Comparison with original implementations\n\nPackages with original implementations:\n* `gigaam` for GigaAM models ([github](https://github.com/salute-developers/GigaAM))\n* `nemo-toolkit` for NeMo models ([github](https://github.com/nvidia/nemo))\n* `openai-whisper` for Whisper models ([github](https://github.com/openai/whisper))\n* `sherpa-onnx` for Vosk models ([github](https://github.com/k2-fsa/sherpa-onnx), [docs](https://k2-fsa.github.io/sherpa/onnx/index.html))\n* `T-one` for T-Tech T-one model ([github](https://github.com/voicekit-team/T-one))\n\nHardware:\n1. CPU tests were run on a laptop with an Intel i7-7700HQ processor.\n2. GPU tests were run in Google Colab on Nvidia T4.\n\nTests of Russian ASR models were performed on a *test* subset of the [Russian LibriSpeech](https://huggingface.co/datasets/istupakov/russian_librispeech) dataset.\n\n| Model                     | Package / decoding   | CER    | WER    | RTFx (CPU) | RTFx (GPU)   |\n|---------------------------|----------------------|--------|--------|------------|--------------|\n|       GigaAM v2 CTC       |        default       | 1.06%  | 5.23%  |        7.2 | 44.2         |\n|       GigaAM v2 CTC       |       onnx-asr       | 1.06%  | 5.23%  |       11.6 | 197.0        |\n|      GigaAM v2 RNN-T      |        default       | 1.10%  | 5.22%  |        5.5 | 23.3         |\n|      GigaAM v2 RNN-T      |       onnx-asr       | 1.10%  | 5.22%  |       10.7 | 84.1         |\n|       GigaAM v3 CTC       |        default       | 0.98%  | 4.72%  |       12.2 | 73.3         |\n|       GigaAM v3 CTC       |       onnx-asr       | 0.98%  | 4.72%  |       14.5 | 223.1        |\n|      GigaAM v3 RNN-T      |        default       | 0.93%  | 4.39%  |        8.2 | 41.6         |\n|      GigaAM v3 RNN-T      |       onnx-asr       | 0.93%  | 4.39%  |       13.3 | 92.1         |\n|     GigaAM v3 E2E CTC     |        default       | 1.50%  | 7.10%  |        N/A | 178.0        |\n|     GigaAM v3 E2E CTC     |       onnx-asr       | 1.56%  | 7.80%  |        N/A | 222.8        |\n|    GigaAM v3 E2E RNN-T    |        default       | 1.61%  | 6.94%  |        N/A | 47.6         |\n|    GigaAM v3 E2E RNN-T    |       onnx-asr       | 1.67%  | 7.60%  |        N/A | 98.5         |\n|  Nemo FastConformer CTC   |        default       | 3.11%  | 13.12% |       29.1 | 143.0        |\n|  Nemo FastConformer CTC   |       onnx-asr       | 3.13%  | 13.10% |       45.8 | 484.7        |\n| Nemo FastConformer RNN-T  |        default       | 2.63%  | 11.62% |       17.4 | 111.6        |\n| Nemo FastConformer RNN-T  |       onnx-asr       | 2.62%  | 11.57% |       27.2 | 119.4        |\n| Nemo Parakeet TDT 0.6B V3 |        default       | 2.34%  | 10.95% |        5.6 | 75.4         |\n| Nemo Parakeet TDT 0.6B V3 |       onnx-asr       | 2.38%  | 10.95% |        9.7 | 97.3         |\n|     Nemo Canary 1B V2     |        default       | 4.89%  | 20.00% |        N/A | 14.0         |\n|     Nemo Canary 1B V2     |       onnx-asr       | 5.00%  | 20.03% |        N/A | 18.6         |\n|       T-Tech T-one        |        default       | 1.28%  | 6.56%  |       11.9 | N/A          |\n|       T-Tech T-one        |       onnx-asr       | 1.28%  | 6.57%  |       11.7 | 40.6         |\n|      Vosk 0.52 small      |     greedy_search    | 3.64%  | 14.53% |       48.2 | 71.4         |\n|      Vosk 0.52 small      | modified_beam_search | 3.50%  | 14.25% |       29.0 | 24.7         |\n|      Vosk 0.52 small      |       onnx-asr       | 3.64%  | 14.53% |       45.5 | 115.0        |\n|         Vosk 0.54         |     greedy_search    | 2.21%  | 9.89%  |       34.8 | 64.2         |\n|         Vosk 0.54         | modified_beam_search | 2.21%  | 9.85%  |       23.9 | 24           |\n|         Vosk 0.54         |       onnx-asr       | 2.21%  | 9.89%  |       33.6 | 97.6         |\n|       Whisper base        |        default       | 10.61% | 38.89% |        5.4 | 17.3         |\n|       Whisper base        |       onnx-asr*      | 10.64% | 38.33% |        6.6 | 58.0         |\n|  Whisper large-v3-turbo   |        default       | 2.96%  | 10.27% |        N/A | 13.6         |\n|  Whisper large-v3-turbo   |       onnx-asr**     | 2.63%  | 10.13% |        N/A | 19.5         |\n\nTests of English ASR models were performed on a *test* subset of the [Voxpopuli](https://huggingface.co/datasets/facebook/voxpopuli) dataset.\n\n| Model                     | Package / decoding   | CER    | WER    | RTFx (CPU) | RTFx (GPU)   |\n|---------------------------|----------------------|--------|--------|------------|--------------|\n|  Nemo Parakeet CTC 0.6B   |        default       | 4.09%  | 7.20%  | 8.3        | 107.7        |\n|  Nemo Parakeet CTC 0.6B   |       onnx-asr       | 4.10%  | 7.22%  | 11.5       | 154.7        |\n| Nemo Parakeet RNN-T 0.6B  |        default       | 3.64%  | 6.32%  | 6.7        | 85.0         |\n| Nemo Parakeet RNN-T 0.6B  |       onnx-asr       | 3.64%  | 6.33%  | 8.7        | 69.7         |\n| Nemo Parakeet TDT 0.6B V2 |        default       | 3.88%  | 6.52%  | 6.5        | 87.6         |\n| Nemo Parakeet TDT 0.6B V2 |       onnx-asr       | 3.87%  | 6.52%  | 10.5       | 116.7        |\n| Nemo Parakeet TDT 0.6B V3 |        default       | 3.97%  | 6.76%  | 6.1        | 90.0         |\n| Nemo Parakeet TDT 0.6B V3 |       onnx-asr       | 3.97%  | 6.75%  | 9.5        | 106.2        |\n|     Nemo Canary 1B V2     |        default       | 4.62%  | 7.42%  | N/A        | 17.5         |\n|     Nemo Canary 1B V2     |       onnx-asr       | 4.67%  | 7.47%  | N/A        | 22.1         |\n|       Whisper base        |        default       | 7.81%  | 13.24% | 8.4        | 27.7         |\n|       Whisper base        |       onnx-asr*      | 7.52%  | 12.76% | 9.2        | 92.2         |\n|  Whisper large-v3-turbo   |        default       | 6.85%  | 11.16% | N/A        | 20.4         |\n|  Whisper large-v3-turbo   |       onnx-asr**     | 10.31% | 14.65% | N/A        | 29.2         |\n\n\u003e [!NOTE]\n\u003e 1. \\* `whisper-ort` model ([model types](#supported-model-types)).\n\u003e 2. ** `whisper` model ([model types](#supported-model-types)) with `fp16` precision.\n\u003e 3. All other models were run with the default precision - `fp32` on CPU and `fp32` or `fp16` (some of the original models) on GPU.\n\n## Benchmarks\n\nHardware:\n1. Arm tests were run on an Orange Pi Zero 3 with a Cortex-A53 processor.\n2. x64 tests were run on a laptop with an Intel i7-7700HQ processor.\n3. T4 tests were run in Google Colab on Nvidia T4 with CUDA and TensorRT.\n\n\u003e [!NOTE]\n\u003e In T4 tests, preprocessors are always run using the TensorRT provider.\n\n### Russian ASR models\nNotebook with benchmark code - [benchmark-ru](examples/benchmark-ru.ipynb)\n\n[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/istupakov/onnx-asr/blob/main/examples/benchmark-ru.ipynb)\n\n| Model                     | Arm RTFx   | x64 RTFx   | T4 RTFx (CUDA) | T4 RTFx (TensorRT) | T4 RTFx (TensorRT, fp16) |\n|---------------------------|------------|------------|----------------|--------------------|--------------------------|\n| GigaAM v2 CTC             | 0.8        | 11.6       | 127.6          | 197.0              | 619.8                    |\n| GigaAM v2 RNN-T           | 0.8        | 10.7       | 52.6           | 84.1               | 101.6                    |\n| GigaAM v3 CTC             | N/A        | 14.5       | 134.8          | 223.1              | 706.3                    |\n| GigaAM v3 RNN-T           | N/A        | 13.3       | 52.4           | 92.1               | 99.6                     |\n| GigaAM v3 E2E CTC         | N/A        | N/A        | 135.6          | 222.8              | 716.5                    |\n| GigaAM v3 E2E RNN-T       | N/A        | N/A        | 63.8           | 98.5               | 119.3                    |\n| Nemo FastConformer CTC    | 4.0        | 45.8       | 127.7          | 484.7              | 777.7                    |\n| Nemo FastConformer RNN-T  | 3.2        | 27.2       | 57.1           | 119.4              | 124.9                    |\n| Nemo Parakeet TDT 0.6B V3 | N/A        | 9.7        | 63.5           | 97.3               | 181.3                    |\n| Nemo Canary 1B V2         | N/A        | N/A        | 18.6           | N/A                | N/A                      |\n| T-Tech T-one              | N/A        | 11.7       | 15.2           | 40.6               | N/A                      |\n| Vosk 0.52 small           | 5.1        | 45.5       | 115.0          | N/A                | N/A                      |\n| Vosk 0.54                 | 3.8        | 33.6       | 97.6           | N/A                | N/A                      |\n| Whisper base              | 0.8        | 6.6        | 58.0           | N/A                | N/A                      |\n| Whisper large-v3-turbo    | N/A        | N/A        | 19.5           | N/A                | N/A                      |\n\n### English ASR models\n\nNotebook with benchmark code - [benchmark-en](examples/benchmark-en.ipynb)\n\n[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/istupakov/onnx-asr/blob/main/examples/benchmark-en.ipynb)\n\n| Model                     | Arm RTFx   | x64 RTFx   | T4 RTFx (CUDA)  | T4 RTFx (TensorRT) | T4 RTFx (TensorRT, fp16) |\n|---------------------------|------------|------------|-----------------|--------------------|--------------------------|\n| Nemo Parakeet CTC 0.6B    | 1.1        | 11.5       | 106.1           | 154.7              | N/A                      |\n| Nemo Parakeet RNN-T 0.6B  | 1.0        | 8.7        | 49.7            | 69.7               | N/A                      |\n| Nemo Parakeet TDT 0.6B V2 | 1.1        | 10.5       | 77.9            | 116.7              | 233.8                    |\n| Nemo Parakeet TDT 0.6B V3 | N/A        | 9.5        | 77.4            | 106.2              | 227.4                    |\n| Nemo Canary 1B V2         | N/A        | N/A        | 22.1            | N/A                | N/A                      |\n| Whisper base              | 1.2        | 9.2        | 92.2            | N/A                | N/A                      |\n| Whisper large-v3-turbo    | N/A        | N/A        | 29.2            | N/A                | N/A                      |\n\n## Convert model to ONNX\n\nSave the model according to the instructions below and add config.json:\n\n```json\n{\n    \"model_type\": \"nemo-conformer-rnnt\", // See \"Supported model types\"\n    \"features_size\": 80, // Size of preprocessor features for Whisper or Nemo models, supported 80 and 128\n    \"subsampling_factor\": 8, // Subsampling factor - 4 for conformer models and 8 for fastconformer and parakeet models\n    \"max_tokens_per_step\": 10 // Max tokens per step for RNN-T decoder\n}\n```\nThen you can upload the model into Hugging Face and use `load_model` to download it.\n\n### Nvidia NeMo Conformer/FastConformer/Parakeet\nInstall **NeMo Toolkit**\n```shell\npip install nemo_toolkit['asr']\n```\n\nDownload model and export to ONNX format\n```py\nimport nemo.collections.asr as nemo_asr\nfrom pathlib import Path\n\nmodel = nemo_asr.models.ASRModel.from_pretrained(\"nvidia/stt_ru_fastconformer_hybrid_large_pc\")\n\n# For export Hybrid models with CTC decoder\n# model.set_export_config({\"decoder_type\": \"ctc\"})\n\nonnx_dir = Path(\"nemo-onnx\")\nonnx_dir.mkdir(exist_ok=True)\nmodel.export(str(Path(onnx_dir, \"model.onnx\")))\n\nwith Path(onnx_dir, \"vocab.txt\").open(\"wt\") as f:\n    for i, token in enumerate([*model.tokenizer.vocab, \"\u003cblk\u003e\"]):\n        f.write(f\"{token} {i}\\n\")\n```\n\n### Sber GigaAM v2/v3\nInstall **GigaAM**\n```shell\ngit clone https://github.com/salute-developers/GigaAM.git\npip install ./GigaAM --extra-index-url https://download.pytorch.org/whl/cpu\n```\n\nDownload model and export to ONNX format\n```py\nimport gigaam\nfrom pathlib import Path\n\nonnx_dir = \"gigaam-onnx\"\nmodel_type = \"rnnt\"  # or \"ctc\"\n\nmodel = gigaam.load_model(\n    model_type,\n    fp16_encoder=False,  # only fp32 tensors\n    use_flash=False,  # disable flash attention\n)\nmodel.to_onnx(dir_path=onnx_dir)\n\nwith Path(onnx_dir, \"v2_vocab.txt\").open(\"wt\") as f:\n    for i, token in enumerate([\"\\u2581\", *(chr(ord(\"а\") + i) for i in range(32)), \"\u003cblk\u003e\"]):\n        f.write(f\"{token} {i}\\n\")\n```\n\n### OpenAI Whisper (with `onnxruntime` export)\n\nRead onnxruntime [instruction](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/models/whisper/README.md) for convert Whisper to ONNX.\n\nDownload model and export with *Beam Search* and *Forced Decoder Input Ids*:\n```shell\npython3 -m onnxruntime.transformers.models.whisper.convert_to_onnx -m openai/whisper-base --output ./whisper-onnx --use_forced_decoder_ids --optimize_onnx --precision fp32\n```\n\nSave tokenizer config\n```py\nfrom transformers import WhisperTokenizer\n\nprocessor = WhisperTokenizer.from_pretrained(\"openai/whisper-base\")\nprocessor.save_pretrained(\"whisper-onnx\")\n```\n\n### OpenAI Whisper (with `optimum` export)\n\nExport model to ONNX with Hugging Face `optimum-cli`\n```shell\noptimum-cli export onnx --model openai/whisper-base ./whisper-onnx\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fistupakov%2Fonnx-asr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fistupakov%2Fonnx-asr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fistupakov%2Fonnx-asr/lists"}