{"id":15646813,"url":"https://github.com/pinto0309/whisper-onnx-tensorrt","last_synced_at":"2025-07-08T08:36:11.529Z","repository":{"id":168946334,"uuid":"643853187","full_name":"PINTO0309/whisper-onnx-tensorrt","owner":"PINTO0309","description":"ONNX and TensorRT implementation of Whisper","archived":false,"fork":false,"pushed_at":"2023-05-27T14:57:59.000Z","size":1268,"stargazers_count":61,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-30T12:31:49.285Z","etag":null,"topics":["cupy","numpy","onnx","stt","tensorrt","whisper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PINTO0309.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-22T09:42:08.000Z","updated_at":"2024-12-16T08:51:23.000Z","dependencies_parsed_at":null,"dependency_job_id":"24158ca4-4203-4842-930d-eb5d6f1240fd","html_url":"https://github.com/PINTO0309/whisper-onnx-tensorrt","commit_stats":null,"previous_names":["pinto0309/whisper-onnx-tensorrt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PINTO0309%2Fwhisper-onnx-tensorrt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PINTO0309%2Fwhisper-onnx-tensorrt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PINTO0309%2Fwhisper-onnx-tensorrt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PINTO0309%2Fwhisper-onnx-tensorrt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PINTO0309","download_url":"https://codeload.github.com/PINTO0309/whisper-onnx-tensorrt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251701834,"owners_count":21629902,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cupy","numpy","onnx","stt","tensorrt","whisper"],"created_at":"2024-10-03T12:15:09.748Z","updated_at":"2025-04-30T12:34:26.073Z","avatar_url":"https://github.com/PINTO0309.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# whisper-onnx-tensorrt\nONNX and TensorRT implementation of Whisper.\n\nThis repository has been reimplemented with ONNX and TensorRT using [zhuzilin/whisper-openvino](https://github.com/zhuzilin/whisper-openvino) as a reference.\n\nEnables execution only with onnxruntime with CUDA and TensorRT Excecution Provider enabled, no need to install PyTorch or TensorFlow. All backend logic using PyTorch was rewritten to a Numpy/CuPy implementation from scratch.\n\nClick here for CPU version: https://github.com/PINTO0309/whisper-onnx-cpu\n\n## 1. Environment\nAlthough it can run directly on the host PC, I strongly recommend the use of Docker to avoid breaking the environment.\n\n1. Docker\n2. NVIDIA GPU (VRAM 16 GB or more recommended)\n3. onnx 1.13.1\n4. onnxruntime-gpu 1.13.1 (TensorRT Execution Provider custom)\n5. CUDA 11.8\n6. cuDNN 8.9\n7. TensorRT 8.5.3\n8. onnx-tensorrt 8.5-GA\n9. cupy v12.0.0\n10. etc (See Dockerfile.xxx)\n\n## 2. Converted Models\nhttps://github.com/PINTO0309/PINTO_model_zoo/tree/main/381_Whisper\n\n## 3. Docker run\n```bash\ngit clone https://github.com/PINTO0309/whisper-onnx-tensorrt.git \u0026\u0026 cd whisper-onnx-tensorrt\n```\n### 3-1. CUDA ver\n```bash\ndocker run --rm -it --gpus all -v `pwd`:/workdir pinto0309/whisper-onnx-cuda\n```\n### 3-2. TensorRT ver\n```bash\ndocker run --rm -it --gpus all -v `pwd`:/workdir pinto0309/whisper-onnx-tensorrt\n```\n\n## 4. Docker build\nIf you do not need to build the docker image by yourself, you do not need to perform this step.\n### 4-1. CUDA ver\n```bash\ndocker build -t whisper-onnx -f Dockerfile.gpu .\n```\n### 4-2. TensorRT ver\n```bash\ndocker build -t whisper-onnx -f Dockerfile.tensorrt .\n```\n### 4-3. docker run\n```bash\ndocker run --rm -it --gpus all -v `pwd`:/workdir whisper-onnx\n```\n\n## 5. Transcribe\n- `--model` option\n    ```\n    tiny.en\n    tiny\n    base.en\n    base\n    small.en\n    small\n    medium.en\n    medium\n    large-v1\n    large-v2\n    ```\n- command\n\n    The onnx file is automatically downloaded when the sample is run. Note that `Decoder` is run in CUDA, not TensorRT, because the shape of all input tensors must be undefined. When running the TensorRT version, there is a 5 to 10 minute wait for the compilation process from ONNX to the TensorRT Engine during the first inference. If `--language` is not specified, the tokenizer will auto-detect the language.\n    ```bash\n    python whisper/transcribe.py xxxx.mp4 --model small --beam_size 3\n    ```\n- results\n    ```\n    Detecting language using up to the first 30 seconds. Use `--language` to specify the language\n    Detected language: Japanese\n    [00:00.000 --\u003e 00:07.200] ストレオシンの推定モデルの最適化 としまして 後半のパート2は 実際\n    [00:07.200 --\u003e 00:11.600] のデモを交えまして 普段私がどのように モデルを最適化して 様々な\n    [00:11.600 --\u003e 00:15.600] フレームワークの環境でプロイしてる かというのを実際に操作をこの\n    [00:15.600 --\u003e 00:18.280] 画面上で見ていただきながら ご理解いただけるように努めたい\n    [00:18.280 --\u003e 00:21.600] と思います それでは早速ですが こちらの\n    [00:21.600 --\u003e 00:26.320] GitHubの方に本日の公演内容について は すべてチュートリアルをまとめて\n    [00:26.320 --\u003e 00:31.680] コミットしております 2021.0.20.28 インテルティブラーニング\n    [00:31.680 --\u003e 00:35.200] でヒットネットデモという ちょっと長い名前なんですけれども 現状\n    [00:35.200 --\u003e 00:39.120] はプライベートになってますが この公演のタイミングでパブリック\n    [00:39.120 --\u003e 00:43.440] の方に変更したいと思っております 基本的にはこちらの上から順前\n    [00:43.440 --\u003e 00:48.000] ですね チュートリアルを謎って いくという形になります\n    [00:48.000 --\u003e 00:52.640] まず本日対象にするモデルの内容 なんですけれども Google Research\n    ```\n- parameters\n    ```\n    usage: transcribe.py\n        [-h]\n        [--model {tiny.en,tiny,base.en,base,small.en,small,medium.en,medium,large-v1,large-v2}]\n        [--output_dir OUTPUT_DIR]\n        [--verbose VERBOSE]\n        [--disable_cupy]\n        [--task {transcribe,translate}]\n        [--language {af, am, ...}]\n        [--temperature TEMPERATURE]\n        [--best_of BEST_OF]\n        [--beam_size BEAM_SIZE]\n        [--patience PATIENCE]\n        [--length_penalty LENGTH_PENALTY]\n        [--suppress_tokens SUPPRESS_TOKENS]\n        [--initial_prompt INITIAL_PROMPT]\n        [--condition_on_previous_text CONDITION_ON_PREVIOUS_TEXT]\n        [--temperature_increment_on_fallback TEMPERATURE_INCREMENT_ON_FALLBACK]\n        [--compression_ratio_threshold COMPRESSION_RATIO_THRESHOLD]\n        [--logprob_threshold LOGPROB_THRESHOLD]\n        [--no_speech_threshold NO_SPEECH_THRESHOLD]\n        audio [audio ...]\n\n    positional arguments:\n      audio\n        audio file(s) to transcribe\n\n    optional arguments:\n      -h, --help\n        show this help message and exit\n      --model {tiny.en,tiny,base.en,base,small.en,small,medium.en,medium,large-v1,large-v2}\n        name of the Whisper model to use\n        (default: small)\n      --output_dir OUTPUT_DIR, -o OUTPUT_DIR\n        directory to save the outputs\n        (default: .)\n      --verbose VERBOSE\n        whether to print out the progress and debug messages\n        (default: True)\n      --disable_cupy\n        When Out of Memory occurs due to insufficient GPU RAM, this option suppresses GPU\n        RAM consumption.\n      --task {transcribe,translate}\n        whether to perform X-\u003eX speech recognition ('transcribe') or\n        X-\u003eEnglish translation ('translate')\n        (default: transcribe)\n      --language {af, am, ...}\n        language spoken in the audio, specify None to perform language detection\n        (default: None)\n      --temperature TEMPERATURE\n        temperature to use for sampling\n        (default: 0)\n      --best_of BEST_OF\n        number of candidates when sampling with non-zero temperature\n        (default: 5)\n      --beam_size BEAM_SIZE\n        number of beams in beam search, only applicable when temperature is zero\n        (default: 5)\n      --patience PATIENCE\n        optional patience value to use in beam decoding,\n        as in https://arxiv.org/abs/2204.05424,\n        the default (1.0) is equivalent to conventional beam search\n        (default: None)\n      --length_penalty LENGTH_PENALTY\n        optional token length penalty coefficient (alpha) as in\n        https://arxiv.org/abs/1609.08144, uses simple lengt normalization by default\n        (default: None)\n      --suppress_tokens SUPPRESS_TOKENS\n        comma-separated list of token ids to suppress during sampling;\n        '-1' will suppress most special characters except common punctuations\n        (default: -1)\n      --initial_prompt INITIAL_PROMPT\n        optional text to provide as a prompt for the first window.\n        (default: None)\n      --condition_on_previous_text CONDITION_ON_PREVIOUS_TEXT\n        if True, provide the previous output of the model as a prompt for the next window;\n        disabling may make the text inconsistent across windows, but the model becomes\n        less prone to getting stuck in a failure loop\n        (default: True)\n      --temperature_increment_on_fallback TEMPERATURE_INCREMENT_ON_FALLBACK\n        temperature to increase when falling back when the decoding fails to meet either of\n        the thresholds below\n        (default: 0.2)\n      --compression_ratio_threshold COMPRESSION_RATIO_THRESHOLD\n        if the gzip compression ratio is higher than this value, treat the decoding as failed\n        (default: 2.4)\n      --logprob_threshold LOGPROB_THRESHOLD\n        if the average log probability is lower than this value, treat the decoding as failed\n        (default: -1.0)\n      --no_speech_threshold NO_SPEECH_THRESHOLD\n        if the probability of the \u003c|nospeech|\u003e token is higher than this value AND\n        the decoding has failed due to `logprob_threshold`, consider the segment as silence\n        (default: 0.6)\n    ```\n## 6. Languages\nhttps://github.com/PINTO0309/whisper-onnx-tensorrt/blob/main/whisper/tokenizer.py\n```\nLANGUAGES = {\n    \"en\": \"english\",\n    \"zh\": \"chinese\",\n    \"de\": \"german\",\n    \"es\": \"spanish\",\n    \"ru\": \"russian\",\n    \"ko\": \"korean\",\n    \"fr\": \"french\",\n    \"ja\": \"japanese\",\n    \"pt\": \"portuguese\",\n    \"tr\": \"turkish\",\n    \"pl\": \"polish\",\n    \"ca\": \"catalan\",\n    \"nl\": \"dutch\",\n    \"ar\": \"arabic\",\n    \"sv\": \"swedish\",\n    \"it\": \"italian\",\n    \"id\": \"indonesian\",\n    \"hi\": \"hindi\",\n    \"fi\": \"finnish\",\n    \"vi\": \"vietnamese\",\n    \"iw\": \"hebrew\",\n    \"uk\": \"ukrainian\",\n    \"el\": \"greek\",\n    \"ms\": \"malay\",\n    \"cs\": \"czech\",\n    \"ro\": \"romanian\",\n    \"da\": \"danish\",\n    \"hu\": \"hungarian\",\n    \"ta\": \"tamil\",\n    \"no\": \"norwegian\",\n    \"th\": \"thai\",\n    \"ur\": \"urdu\",\n    \"hr\": \"croatian\",\n    \"bg\": \"bulgarian\",\n    \"lt\": \"lithuanian\",\n    \"la\": \"latin\",\n    \"mi\": \"maori\",\n    \"ml\": \"malayalam\",\n    \"cy\": \"welsh\",\n    \"sk\": \"slovak\",\n    \"te\": \"telugu\",\n    \"fa\": \"persian\",\n    \"lv\": \"latvian\",\n    \"bn\": \"bengali\",\n    \"sr\": \"serbian\",\n    \"az\": \"azerbaijani\",\n    \"sl\": \"slovenian\",\n    \"kn\": \"kannada\",\n    \"et\": \"estonian\",\n    \"mk\": \"macedonian\",\n    \"br\": \"breton\",\n    \"eu\": \"basque\",\n    \"is\": \"icelandic\",\n    \"hy\": \"armenian\",\n    \"ne\": \"nepali\",\n    \"mn\": \"mongolian\",\n    \"bs\": \"bosnian\",\n    \"kk\": \"kazakh\",\n    \"sq\": \"albanian\",\n    \"sw\": \"swahili\",\n    \"gl\": \"galician\",\n    \"mr\": \"marathi\",\n    \"pa\": \"punjabi\",\n    \"si\": \"sinhala\",\n    \"km\": \"khmer\",\n    \"sn\": \"shona\",\n    \"yo\": \"yoruba\",\n    \"so\": \"somali\",\n    \"af\": \"afrikaans\",\n    \"oc\": \"occitan\",\n    \"ka\": \"georgian\",\n    \"be\": \"belarusian\",\n    \"tg\": \"tajik\",\n    \"sd\": \"sindhi\",\n    \"gu\": \"gujarati\",\n    \"am\": \"amharic\",\n    \"yi\": \"yiddish\",\n    \"lo\": \"lao\",\n    \"uz\": \"uzbek\",\n    \"fo\": \"faroese\",\n    \"ht\": \"haitian creole\",\n    \"ps\": \"pashto\",\n    \"tk\": \"turkmen\",\n    \"nn\": \"nynorsk\",\n    \"mt\": \"maltese\",\n    \"sa\": \"sanskrit\",\n    \"lb\": \"luxembourgish\",\n    \"my\": \"myanmar\",\n    \"bo\": \"tibetan\",\n    \"tl\": \"tagalog\",\n    \"mg\": \"malagasy\",\n    \"as\": \"assamese\",\n    \"tt\": \"tatar\",\n    \"haw\": \"hawaiian\",\n    \"ln\": \"lingala\",\n    \"ha\": \"hausa\",\n    \"ba\": \"bashkir\",\n    \"jw\": \"javanese\",\n    \"su\": \"sundanese\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpinto0309%2Fwhisper-onnx-tensorrt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpinto0309%2Fwhisper-onnx-tensorrt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpinto0309%2Fwhisper-onnx-tensorrt/lists"}