{"id":13492649,"url":"https://github.com/huggingface/distil-whisper","last_synced_at":"2025-04-29T14:19:56.537Z","repository":{"id":204741330,"uuid":"712526447","full_name":"huggingface/distil-whisper","owner":"huggingface","description":"Distilled variant of Whisper for speech recognition. 6x faster, 50% smaller, within 1% word error rate.","archived":false,"fork":false,"pushed_at":"2025-01-08T10:09:47.000Z","size":2696,"stargazers_count":3840,"open_issues_count":73,"forks_count":321,"subscribers_count":67,"default_branch":"main","last_synced_at":"2025-04-29T14:19:51.709Z","etag":null,"topics":["audio","speech-recognition","whisper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-31T16:34:45.000Z","updated_at":"2025-04-29T10:49:28.000Z","dependencies_parsed_at":"2023-11-16T20:04:03.018Z","dependency_job_id":"eaa43aaf-cc23-48fb-b577-51cc4f7d997b","html_url":"https://github.com/huggingface/distil-whisper","commit_stats":{"total_commits":151,"total_committers":9,"mean_commits":16.77777777777778,"dds":0.3708609271523179,"last_synced_commit":"9422916d714bde70ea9d88e13c56f877ced1e76b"},"previous_names":["huggingface/distil-whisper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdistil-whisper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdistil-whisper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdistil-whisper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdistil-whisper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/distil-whisper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251516953,"owners_count":21601912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audio","speech-recognition","whisper"],"created_at":"2024-07-31T19:01:07.888Z","updated_at":"2025-04-29T14:19:56.519Z","avatar_url":"https://github.com/huggingface.png","language":"Python","funding_links":[],"categories":["Python","Recognition","Speech-to-Text (STT)","Repos","STT (Speech-to-Text) | 语音转文本","For Developers"],"sub_categories":["Open-Source Models \u0026 Libraries","Open Source STT Models | 开源 STT 模型","Model Variants \u0026 Performance Optimizations"],"readme":"# Distil-Whisper\n\n[[Paper]](https://arxiv.org/abs/2311.00430)\n[[Models]](https://huggingface.co/collections/distil-whisper/distil-whisper-models-65411987e6727569748d2eb6)\n[[Colab]](https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/Distil_Whisper_Benchmark.ipynb)\n[[Training Code]](training)\n\nDistil-Whisper is a distilled version of Whisper for English speech recognition that is **6 times faster**, 49% smaller, and performs **within 1% word \nerror rate (WER)** on out-of-distribution evaluation sets:\n\n| Model                                                                      | Params / M | Rel. Latency ↑ | Short-Form WER ↓ | Long-Form WER ↓ |\n|----------------------------------------------------------------------------|------------|----------------|------------------|-----------------|\n| [large-v3](https://huggingface.co/openai/whisper-large-v3)                 | 1550       | 1.0            | **8.4**          | 11.0            |\n|                                                                            |            |                |                  |                 |\n| [distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)   | 756        | 6.3            | 9.7              | **10.8**        |\n| [distil-large-v2](https://huggingface.co/distil-whisper/distil-large-v2)   | 756        | 5.8            | 10.1             | 11.6            |\n| [distil-medium.en](https://huggingface.co/distil-whisper/distil-medium.en) | 394        | **6.8**        | 11.1             | 12.4            |\n| [distil-small.en](https://huggingface.co/distil-whisper/distil-small.en)   | **166**    | 5.6            | 12.1             | 12.8            |\n\nFor most applications, we recommend the latest [distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3) checkpoint,\nsince it is the most performant distilled checkpoint and compatible across all Whisper libraries. The only exception is \nresource-constrained applications with very little memory, such as on-device or mobile applications, where the \n[distil-small.en](https://huggingface.co/distil-whisper/distil-small.en) is a great choice, since it is only 166M \nparameters and performs within 4% WER of Whisper large-v3.\n\n\u003e [!NOTE]  \n\u003e Distil-Whisper is only available for English speech recognition. For multilingual speech recognition, we recommend using the [Whisper Turbo](https://huggingface.co/openai/whisper-large-v3-turbo) checkpoint, which was released by OpenAI and leverages the same principles as Distil-Whisper. For details, refer to the Whisper turbo [release statement](https://github.com/openai/whisper/discussions/2363).\n\n## 1. Usage\n\nDistil-Whisper is supported in Hugging Face 🤗 Transformers from version 4.35 onwards. To run the model, first \ninstall the latest version of the Transformers library. For this example, we'll also install 🤗 Datasets to load a toy \naudio dataset from the Hugging Face Hub:\n\n```bash\npip install --upgrade pip\npip install --upgrade transformers accelerate datasets[audio]\n```\n\n### Short-Form Transcription\n\nShort-form transcription is the process of transcribing audio samples that are less than 30-seconds long, which is the \nmaximum receptive field of the Whisper models. This means the entire audio clip can be processed in one go without the \nneed for chunking.\n\nFirst, we load Distil-Whisper via the convenient [`AutoModelForSpeechSeq2Seq`](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForSpeechSeq2Seq) and [`AutoProcessor`](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoProcessor) classes.\n\nWe load the model in `float16` precision and make sure that loading time takes as little time as possible by passing `low_cpu_mem_usage=True`.\nIn addition, we want to make sure that the model is loaded in [`safetensors`](https://github.com/huggingface/safetensors) format by passing `use_safetensors=True`:\n\n```python\nimport torch\nfrom transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline\n\ndevice = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\ntorch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32\n\nmodel_id = \"distil-whisper/distil-large-v3\"\n\nmodel = AutoModelForSpeechSeq2Seq.from_pretrained(\n    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True\n)\nmodel.to(device)\n\nprocessor = AutoProcessor.from_pretrained(model_id)\n```\n\nThe model and processor can then be passed to the [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline).\nNote that if you would like to have more control over the generation process, you can directly make use of model + processor API as shown below.\n\n```python\npipe = pipeline(\n    \"automatic-speech-recognition\",\n    model=model,\n    tokenizer=processor.tokenizer,\n    feature_extractor=processor.feature_extractor,\n    max_new_tokens=128,\n    torch_dtype=torch_dtype,\n    device=device,\n)\n```\n\nNext, we load an example short-form audio from the LibriSpeech corpus:\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\nsample = dataset[0][\"audio\"]\n```\n\nFinally, we can call the pipeline to transcribe the audio:\n\n```python\nresult = pipe(sample)\nprint(result[\"text\"])\n```\n\nTo transcribe a local audio file, simply pass the path to your audio file when you call the pipeline:\n\n```python\nresult = pipe(\"audio.mp3\")\nprint(result[\"text\"])\n```\n\nFor more information on how to customize the automatic speech recognition pipeline, please refer to the ASR pipeline [docs](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline).\nWe also provide an end-to-end [Google Colab](https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/Distil_Whisper_Benchmark.ipynb) that benchmarks Whisper against Distil-Whisper.\n\n\u003cdetails\u003e\n\u003csummary\u003e For more control over the generation parameters, use the model + processor API directly: \u003c/summary\u003e\n\nAd-hoc generation arguments can be passed to `model.generate`, including `num_beams` for beam-search, `return_timestamps` \nfor segment-level timestamps, and `prompt_ids` for prompting. See the [docstrings](https://huggingface.co/docs/transformers/en/model_doc/whisper#transformers.WhisperForConditionalGeneration.generate)\nfor more details.\n\n```python\nimport torch\nfrom transformers import AutoModelForSpeechSeq2Seq, AutoProcessor\nfrom datasets import Audio, load_dataset\n\n\ndevice = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\ntorch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32\n\nmodel_id = \"distil-whisper/distil-large-v3\"\n\nmodel = AutoModelForSpeechSeq2Seq.from_pretrained(\n    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True\n)\nmodel.to(device)\n\nprocessor = AutoProcessor.from_pretrained(model_id)\n\ndataset = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\ndataset = dataset.cast_column(\"audio\", Audio(processor.feature_extractor.sampling_rate))\nsample = dataset[0][\"audio\"]\n\ninput_features = processor(\n  sample[\"array\"], sampling_rate=sample[\"sampling_rate\"], return_tensors=\"pt\"\n).input_features\n\ninput_features = input_features.to(device, dtype=torch_dtype)\n\ngen_kwargs = {\n  \"max_new_tokens\": 128,\n  \"num_beams\": 1,\n  \"return_timestamps\": False,\n}\n\npred_ids = model.generate(input_features, **gen_kwargs)\npred_text = processor.batch_decode(pred_ids, skip_special_tokens=True, decode_with_timestamps=gen_kwargs[\"return_timestamps\"])\n\nprint(pred_text)\n```\n\n\u003c/details\u003e\n\n### Sequential Long-Form\n\nThe latest [distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3) checkpoint is specifically designed \nto be compatible with OpenAI's sequential long-form transcription algorithm. This algorithm uses a sliding window for \nbuffered inference of long audio files (\u003e 30-seconds), and returns more accurate transcriptions compared to the \n[chunked long-form algorithm](#chunked-long-form).\n\nThe sequential long-form algorithm should be used in either of the following scenarios:\n1. Transcription accuracy is the most important factor, and latency is less of a consideration\n2. You are transcribing **batches** of long audio files, in which case the latency of sequential is comparable to chunked, while being up to 0.5% WER more accurate\n\nIf you are transcribing single long audio files and latency is the most important factor, you should use the chunked algorithm\ndescribed [below](#chunked-long-form). For a detailed explanation of the different algorithms, refer to Sections 5 of \nthe [Distil-Whisper paper](https://arxiv.org/pdf/2311.00430.pdf).\n\nWe start by loading the model and processor as before:\n\n```python\nimport torch\nfrom transformers import AutoModelForSpeechSeq2Seq, AutoProcessor\n\ndevice = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\ntorch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32\n\nmodel_id = \"distil-whisper/distil-large-v3\"\n\nmodel = AutoModelForSpeechSeq2Seq.from_pretrained(\n    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True\n)\nmodel.to(device)\n\nprocessor = AutoProcessor.from_pretrained(model_id)\n```\n\nThe model and processor can then be passed to the [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline).\nNote that if you would like to have more control over the generation process, you can directly make use of `model.generate(...)` API as shown below.\n\n```python\npipe = pipeline(\n    \"automatic-speech-recognition\",\n    model=model,\n    tokenizer=processor.tokenizer,\n    feature_extractor=processor.feature_extractor,\n    max_new_tokens=128,\n    torch_dtype=torch_dtype,\n    device=device,\n)\n```\n\nNext, we load a long-form audio sample. Here, we use an example of concatenated samples from the LibriSpeech corpus:\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"distil-whisper/librispeech_long\", \"clean\", split=\"validation\")\nsample = dataset[0][\"audio\"]\n```\n\nFinally, we can call the pipeline to transcribe the audio:\n\n```python\nresult = pipe(sample)\nprint(result[\"text\"])\n```\n\nTo transcribe a local audio file, simply pass the path to your audio file when you call the pipeline:\n\n```python\nresult = pipe(\"audio.mp3\")\nprint(result[\"text\"])\n```\n\n\u003cdetails\u003e\n\n\u003csummary\u003e For more control over the generation parameters, use the model + processor API directly: \u003c/summary\u003e\n\n```python\nimport torch\nfrom transformers import AutoModelForSpeechSeq2Seq, AutoProcessor\nfrom datasets import Audio, load_dataset\n\n\ndevice = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\ntorch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32\n\nmodel_id = \"distil-whisper/distil-large-v3\"\n\nmodel = AutoModelForSpeechSeq2Seq.from_pretrained(\n    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True\n)\nmodel.to(device)\n\nprocessor = AutoProcessor.from_pretrained(model_id)\n\ndataset = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\ndataset = dataset.cast_column(\"audio\", Audio(processor.feature_extractor.sampling_rate))\nsample = dataset[0][\"audio\"]\n\ninputs = processor(\n    sample[\"array\"],\n    sampling_rate=sample[\"sampling_rate\"],\n    return_tensors=\"pt\",\n    truncation=False,\n    padding=\"longest\",\n    return_attention_mask=True,\n)\ninputs = inputs.to(device, dtype=torch_dtype)\n\ngen_kwargs = {\n    \"max_new_tokens\": 448,\n    \"num_beams\": 1,\n    \"condition_on_prev_tokens\": False,\n    \"compression_ratio_threshold\": 1.35,  # zlib compression ratio threshold (in token space)\n    \"temperature\": (0.0, 0.2, 0.4, 0.6, 0.8, 1.0),\n    \"logprob_threshold\": -1.0,\n    \"no_speech_threshold\": 0.6,\n    \"return_timestamps\": True,\n}\n\npred_ids = model.generate(**inputs, **gen_kwargs)\npred_text = processor.batch_decode(pred_ids, skip_special_tokens=True, decode_with_timestamps=False)\n\nprint(pred_text)\n```\n\n\u003c/details\u003e\n\n### Chunked Long-Form\n\ndistil-large-v3 remains compatible with the Transformers chunked long-form algorithm. This algorithm should be used when \na single large audio file is being transcribed and the fastest possible inference is required. In such circumstances, \nthe chunked algorithm is up to 9x faster than OpenAI's sequential long-form implementation (see Table 7 of the \n[Distil-Whisper paper](https://arxiv.org/pdf/2311.00430.pdf)).\n\nWe can load the model and processor as before:\n\n```python\nimport torch\nfrom transformers import AutoModelForSpeechSeq2Seq, AutoProcessor\n\ndevice = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\ntorch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32\n\nmodel_id = \"distil-whisper/distil-large-v3\"\n\nmodel = AutoModelForSpeechSeq2Seq.from_pretrained(\n    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True\n)\nmodel.to(device)\n\nprocessor = AutoProcessor.from_pretrained(model_id)\n```\n\nTo enable chunking, pass the `chunk_length_s` parameter to the `pipeline`. For distil-large-v3, a chunk length of 25-seconds\nis optimal. To activate batching, pass the argument `batch_size`:\n\n```python\npipe = pipeline(\n    \"automatic-speech-recognition\",\n    model=model,\n    tokenizer=processor.tokenizer,\n    feature_extractor=processor.feature_extractor,\n    max_new_tokens=128,\n    chunk_length_s=25,\n    batch_size=16,\n    torch_dtype=torch_dtype,\n    device=device,\n)\n```\n\nThe argument `max_new_tokens` controls the maximum number of generated tokens *per-chunk*. In the typical speech setting,\nwe have no more than 3 words spoken per-second. Therefore, for a 30-second input, we have at most 90 words (approx 128 tokens).\nWe set the maximum number of generated tokens per-chunk to 128 to truncate any possible hallucinations that occur at the \nend of the segment. These tokens get removed at the chunk borders using the long-form chunking transcription algorithm, \nso it is more efficient to truncate them directly during generation to avoid redundant generation steps in the decoder.\n\nNow, let's load a long-form audio sample. Here, we use an example of concatenated samples from the LibriSpeech corpus:\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"distil-whisper/librispeech_long\", \"clean\", split=\"validation\")\nsample = dataset[0][\"audio\"]\n```\n\nFinally, we can call the pipeline to transcribe the audio:\n\n```python\nresult = pipe(sample)\nprint(result[\"text\"])\n```\n\nFor more information on how to customize the automatic speech recognition pipeline, please refer to the ASR pipeline [docs](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline).\n\n### Speculative Decoding\n\nDistil-Whisper can be used as an assistant model to Whisper for [speculative decoding](https://huggingface.co/blog/whisper-speculative-decoding). \nSpeculative decoding mathematically ensures the exact same outputs as Whisper are obtained while being 2 times faster. \nThis makes it the perfect drop-in replacement for existing Whisper pipelines, since the same outputs are guaranteed.\n\nFor speculative decoding, we need to load both the teacher: [`openai/whisper-large-v3`](https://huggingface.co/openai/whisper-large-v3).\nAs well as the assistant (*a.k.a* student) [`distil-whisper/distil-large-v3`](https://huggingface.co/distil-whisper/distil-large-v3).\n\nLet's start by loading the teacher model and processor. We do this in much the same way we loaded the Distil-Whisper \nmodel in the previous examples:\n\n```python\nfrom transformers import AutoModelForSpeechSeq2Seq, AutoProcessor\nimport torch\n\ndevice = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\ntorch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32\n\nmodel_id = \"openai/whisper-large-v3\"\n\nmodel = AutoModelForSpeechSeq2Seq.from_pretrained(\n    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True\n)\nmodel.to(device)\n\nprocessor = AutoProcessor.from_pretrained(model_id)\n```\n\nNow let's load the assistant. Since Distil-Whisper shares exactly same encoder as the teacher model, we only need \nto load the 2-layer decoder as a \"Decoder-only\" model:\n\n```python\nfrom transformers import AutoModelForCausalLM\nassistant_model_id = \"distil-whisper/distil-large-v2\"\n\nassistant_model = AutoModelForCausalLM.from_pretrained(\n    assistant_model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True\n)\nassistant_model.to(device)\n```\n\nThe assistant model shares the same processor as the teacher, so there's no need to load a student processor.\n\nWe can now pass the assistant model to the pipeline to be used for speculative decoding. We pass it as a `generate_kwarg`\nwith the key [`\"assistant_model\"`](https://huggingface.co/docs/transformers/main/en/main_classes/text_generation#transformers.GenerationMixin.generate.assistant_model) \nso that speculative decoding is enabled:\n\n```python\npipe = pipeline(\n    \"automatic-speech-recognition\",\n    model=model,\n    tokenizer=processor.tokenizer,\n    feature_extractor=processor.feature_extractor,\n    max_new_tokens=128,\n    generate_kwargs={\"assistant_model\": assistant_model},\n    torch_dtype=torch_dtype,\n    device=device,\n)\n```\n\nAs before, we can pass any sample to the pipeline to be transcribed:\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\nsample = dataset[0][\"audio\"]\n\nresult = pipe(sample)\nprint(result[\"text\"])\n```\n\n**Note:** speculative decoding should be on average 2x faster than using \"only\" Whisper large-v2 at a mere 8% increase \nin VRAM memory usage while mathematically ensuring the same results. This makes it the perfect replacement for Whisper large-v2\nin existing speech recognition pipelines.\n\nFor more details on speculative decoding, refer to the following resources:\n* [Speculative decoding for 2x faster Whisper inference](https://huggingface.co/blog/whisper-speculative-decoding) blog post by Sanchit Gandhi\n* [Assisted Generation: a new direction toward low-latency text generation](https://huggingface.co/blog/assisted-generation) blog post by Joao Gante\n* [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192) paper by Leviathan et. al.\n\n### Additional Speed \u0026 Memory Improvements\n\nYou can apply additional speed and memory improvements to Distil-Whisper which we cover in the following.\n\n#### Flash Attention\n\nWe recommend using [Flash Attention 2](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#flashattention-2) if your GPU allows for it.\nTo do so, you first need to install [Flash Attention](https://github.com/Dao-AILab/flash-attention):\n\n```\npip install flash-attn --no-build-isolation\n```\n\nYou can then pass `use_flash_attention_2=True` to `from_pretrained` to enable Flash Attention 2:\n\n```diff\n- model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True)\n+ model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True, use_flash_attention_2=True)\n```\n\n#### Torch Scale-Product-Attention (SDPA)\n\nIf your GPU does not support Flash Attention, we recommend making use of [BetterTransformers](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#bettertransformer).\nTo do so, you first need to install optimum:\n\n```\npip install --upgrade optimum\n```\n\nAnd then convert your model to a \"BetterTransformer\" model before using it:\n\n```diff\nmodel = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True)\n+ model = model.to_bettertransformer()\n```\n\n### Exporting to Other Libraries\n\nDistil-Whisper has support in the following libraries with the original \"sequential\" long-form transcription algorithm. \nClick the links in the table to see the relevant code-snippets for each:\n\n| Library         | distil-small.en                                                                                 | distil-medium.en                                                                                 | distil-large-v2                                                                                 |\n|-----------------|-------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|\n| OpenAI Whisper  | [link](https://huggingface.co/distil-whisper/distil-small.en#running-whisper-in-openai-whisper) | [link](https://huggingface.co/distil-whisper/distil-medium.en#running-whisper-in-openai-whisper) | [link](https://huggingface.co/distil-whisper/distil-large-v2#running-whisper-in-openai-whisper) |\n| Whisper cpp     | [link](https://huggingface.co/distil-whisper/distil-small.en#whispercpp)                        | [link](https://huggingface.co/distil-whisper/distil-medium.en#whispercpp)                        | [link](https://huggingface.co/distil-whisper/distil-large-v2#whispercpp)                        |\n| Transformers js | [link](https://huggingface.co/distil-whisper/distil-small.en#transformersjs)                    | [link](https://huggingface.co/distil-whisper/distil-medium.en#transformersjs)                    | [link](https://huggingface.co/distil-whisper/distil-large-v2#transformersjs)                    |\n| Candle (Rust)   | [link](https://huggingface.co/distil-whisper/distil-small.en#candle)                            | [link](https://huggingface.co/distil-whisper/distil-medium.en#candle)                            | [link](https://huggingface.co/distil-whisper/distil-large-v2#candle)                            |\n\nUpdates will be posted here with the integration of the \"chunked\" long-form transcription algorithm into the respective \nlibraries.\n\nFor the 🤗 Transformers code-examples, refer to the sections [Short-Form](#short-form-transcription) and [Long-Form](#long-form-transcription) Transcription.\n\n## 2. Why use Distil-Whisper? ⁉️\n\nDistil-Whisper is designed to be a drop-in replacement for Whisper on English speech recognition. Here are 5 reasons for making the\nswitch to Distil-Whisper:\n\n1. **Faster inference:** 6 times faster inference speed, while performing to within 1% WER of Whisper on out-of-distribution audio:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://huggingface.co/datasets/distil-whisper/figures/resolve/main/main_table.png?raw=true\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n2. **Robustness to noise:** demonstrated by strong WER performance at low signal-to-noise ratios:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://huggingface.co/datasets/distil-whisper/figures/resolve/main/noise.png?raw=true\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n3. **Robustness to hallucinations:** quantified by 1.3 times fewer repeated 5-gram word duplicates (5-Dup.) and 2.1% lower insertion error rate (IER) than Whisper:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://huggingface.co/datasets/distil-whisper/figures/resolve/main/hallucination.png?raw=true\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n4. **Designed for speculative decoding:** Distil-Whisper can be used as an assistant model to Whisper, giving 2 times faster inference speed while mathematically ensuring the same outputs as the Whisper model.\n5. **Permissive license:** Distil-Whisper is [MIT licensed](./LICENSE), meaning it can be used for commercial applications.\n\n## 3. Approach ✍️\n\nTo distill Whisper, we copy the entire encoder module and freeze it during training. We copy only two decoder layers, \nwhich are initialised from the first and last decoder layers from Whisper. All other decoder layers from Whisper\nare discarded:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://huggingface.co/datasets/distil-whisper/figures/resolve/main/architecture.png?raw=true\" width=\"600\"/\u003e\n\u003c/p\u003e\n\nDistil-Whisper is trained on a *knowledge distillation* objective. Specifically, it is trained to minimise the KL divergence\nbetween the distilled model and the Whisper model, as well as the cross-entropy loss on pseudo-labelled audio data.\n\nWe train Distil-Whisper on a total of 22k hours of pseudo-labelled audio data, spanning 10 domains with over 18k speakers:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://huggingface.co/datasets/distil-whisper/figures/resolve/main/datasets.png?raw=true\" width=\"600\"/\u003e\n\u003c/p\u003e\n\nThis diverse audio dataset is paramount to ensuring robustness of Distil-Whisper to different datasets and domains. \n\nIn addition, we use a WER filter to discard pseudo-labels where Whisper mis-transcribes or hallucinates. This greatly \nimproves WER performance of the downstream distilled model.\n\nFor full details on the distillation set-up and evaluation results, refer to the [Distil-Whisper paper](https://arxiv.org/abs/2311.00430).\n\n## 4. Training Code\n\nTraining code to reproduce Distil-Whisper can be found in the directory [training](training). This code has been adapted \nbe general enough to distill Whisper for multilingual speech recognition, facilitating anyone in the community to distill \nWhisper on their choice of language.\n\n## 5. Acknowledgements\n* OpenAI for the Whisper [model](https://huggingface.co/openai/whisper-large-v3) and [original codebase](https://github.com/openai/whisper)\n* Hugging Face 🤗 [Transformers](https://github.com/huggingface/transformers) for the model integration\n* Google's [TPU Research Cloud (TRC)](https://sites.research.google/trc/about/) program for Cloud TPU v4s\n\n## 6. Citation\n\nIf you use this model, please consider citing the Distil-Whisper paper:\n```\n@misc{gandhi2023distilwhisper,\n      title={Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling}, \n      author={Sanchit Gandhi and Patrick von Platen and Alexander M. Rush},\n      year={2023},\n      eprint={2311.00430},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n```\n\nAnd also the Whisper paper:\n```\n@misc{radford2022robust,\n      title={Robust Speech Recognition via Large-Scale Weak Supervision}, \n      author={Alec Radford and Jong Wook Kim and Tao Xu and Greg Brockman and Christine McLeavey and Ilya Sutskever},\n      year={2022},\n      eprint={2212.04356},\n      archivePrefix={arXiv},\n      primaryClass={eess.AS}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fdistil-whisper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Fdistil-whisper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fdistil-whisper/lists"}