{"id":13405557,"url":"https://github.com/m-bain/whisperX","last_synced_at":"2025-03-14T10:31:02.411Z","repository":{"id":64930458,"uuid":"576103395","full_name":"m-bain/whisperX","owner":"m-bain","description":"WhisperX:  Automatic Speech Recognition with Word-level Timestamps (\u0026 Diarization)","archived":false,"fork":false,"pushed_at":"2024-04-11T14:02:23.000Z","size":24060,"stargazers_count":8782,"open_issues_count":418,"forks_count":907,"subscribers_count":117,"default_branch":"main","last_synced_at":"2024-04-12T11:16:43.093Z","etag":null,"topics":["asr","speech","speech-recognition","speech-to-text","whisper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-4-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-bain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"custom":"https://www.buymeacoffee.com/maxhbain"}},"created_at":"2022-12-09T02:34:23.000Z","updated_at":"2024-06-18T21:11:15.171Z","dependencies_parsed_at":"2023-12-19T18:58:53.070Z","dependency_job_id":"3c0d0cef-319c-4432-bb46-431166f3f2aa","html_url":"https://github.com/m-bain/whisperX","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-bain%2FwhisperX","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-bain%2FwhisperX/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-bain%2FwhisperX/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-bain%2FwhisperX/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-bain","download_url":"https://codeload.github.com/m-bain/whisperX/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221458076,"owners_count":16825271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asr","speech","speech-recognition","speech-to-text","whisper"],"created_at":"2024-07-30T19:02:05.381Z","updated_at":"2025-03-14T10:31:02.401Z","avatar_url":"https://github.com/m-bain.png","language":"Python","funding_links":["https://www.buymeacoffee.com/maxhbain"],"categories":["Python","AI","Applications","Recognition","Repos","语音识别","others","Speech-to-Text (STT)","Whisper","Model variants","STT (Speech-to-Text) | 语音转文本","Speech Processing","🔧 Utilities \u0026 Miscellaneous","AI Caption \u0026 Subtitle Tools","App","For Developers","Audio Transcription"],"sub_categories":["AdBlock-Tester","网络服务_其他","Open-Source Models \u0026 Libraries","Open Source STT Models | 开源 STT 模型","Speech-to-Text","Open-Source Transcription","Model Variants \u0026 Performance Optimizations","Context-Relevant MCP Servers"],"readme":"\u003ch1 align=\"center\"\u003eWhisperX\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/m-bain/whisperX/stargazers\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/stars/m-bain/whisperX.svg?colorA=orange\u0026colorB=orange\u0026logo=github\"\n         alt=\"GitHub stars\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/m-bain/whisperX/issues\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/issues/m-bain/whisperx.svg\"\n             alt=\"GitHub issues\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/m-bain/whisperX/blob/master/LICENSE\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/license/m-bain/whisperX.svg\"\n             alt=\"GitHub license\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://arxiv.org/abs/2303.00747\"\u003e\n        \u003cimg src=\"http://img.shields.io/badge/Arxiv-2303.00747-B31B1B.svg\"\n             alt=\"ArXiv paper\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://twitter.com/intent/tweet?text=\u0026url=https%3A%2F%2Fgithub.com%2Fm-bain%2FwhisperX\"\u003e\n  \u003cimg src=\"https://img.shields.io/twitter/url/https/github.com/m-bain/whisperX.svg?style=social\" alt=\"Twitter\"\u003e\n  \u003c/a\u003e      \n\u003c/p\u003e\n\n\n\u003cimg width=\"1216\" align=\"center\" alt=\"whisperx-arch\" src=\"https://raw.githubusercontent.com/m-bain/whisperX/refs/heads/main/figures/pipeline.png\"\u003e\n\n\n\u003c!-- \u003cp align=\"left\"\u003eWhisper-Based Automatic Speech Recognition (ASR) with improved timestamp accuracy + quality via forced phoneme alignment and voice-activity based batching for fast inference.\u003c/p\u003e --\u003e\n\n\n\u003c!-- \u003ch2 align=\"left\", id=\"what-is-it\"\u003eWhat is it 🔎\u003c/h2\u003e --\u003e\n\n\nThis repository provides fast automatic speech recognition (70x realtime with large-v2) with word-level timestamps and speaker diarization.\n\n- ⚡️ Batched inference for 70x realtime transcription using whisper large-v2\n- 🪶 [faster-whisper](https://github.com/guillaumekln/faster-whisper) backend, requires \u003c8GB gpu memory for large-v2 with beam_size=5\n- 🎯 Accurate word-level timestamps using wav2vec2 alignment\n- 👯‍♂️ Multispeaker ASR using speaker diarization from [pyannote-audio](https://github.com/pyannote/pyannote-audio) (speaker ID labels) \n- 🗣️ VAD preprocessing, reduces hallucination \u0026 batching with no WER degradation\n\n\n\n**Whisper** is an ASR model [developed by OpenAI](https://github.com/openai/whisper), trained on a large dataset of diverse audio. Whilst it does produces highly accurate transcriptions, the corresponding timestamps are at the utterance-level, not per word, and can be inaccurate by several seconds. OpenAI's whisper does not natively support batching.\n\n**Phoneme-Based ASR** A suite of models finetuned to recognise the smallest unit of speech distinguishing one word from another, e.g. the element p in \"tap\". A popular example model is [wav2vec2.0](https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self).\n\n**Forced Alignment** refers to the process by which orthographic transcriptions are aligned to audio recordings to automatically generate phone level segmentation.\n\n**Voice Activity Detection (VAD)** is the detection of the presence or absence of human speech.\n\n**Speaker Diarization** is the process of partitioning an audio stream containing human speech into homogeneous segments according to the identity of each speaker.\n\n\u003ch2 align=\"left\", id=\"highlights\"\u003eNew🚨\u003c/h2\u003e\n\n- 1st place at [Ego4d transcription challenge](https://eval.ai/web/challenges/challenge-page/1637/leaderboard/3931/WER)  🏆\n- _WhisperX_ accepted at INTERSPEECH 2023 \n- v3 transcript segment-per-sentence: using nltk sent_tokenize for better subtitlting \u0026 better diarization\n- v3 released, 70x speed-up open-sourced. Using batched whisper with [faster-whisper](https://github.com/guillaumekln/faster-whisper) backend!\n- v2 released, code cleanup, imports whisper library VAD filtering is now turned on by default, as in the paper.\n- Paper drop🎓👨‍🏫! Please see our [ArxiV preprint](https://arxiv.org/abs/2303.00747) for benchmarking and details of WhisperX. We also introduce more efficient batch inference resulting in large-v2 with *60-70x REAL TIME speed.\n\n\u003ch2 align=\"left\" id=\"setup\"\u003eSetup ⚙️\u003c/h2\u003e\nTested for PyTorch 2.0, Python 3.10 (use other versions at your own risk!)\n\nGPU execution requires the NVIDIA libraries cuBLAS 11.x and cuDNN 8.x to be installed on the system. Please refer to the [CTranslate2 documentation](https://opennmt.net/CTranslate2/installation.html).\n\n\n### 1. Create Python3.10 environment\n\n`conda create --name whisperx python=3.10`\n\n`conda activate whisperx`\n\n\n### 2. Install PyTorch, e.g. for Linux and Windows CUDA11.8:\n\n`conda install pytorch==2.0.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia`\n\nSee other methods [here.](https://pytorch.org/get-started/previous-versions/#v200)\n\n### 3. Install WhisperX\n\nYou have several installation options:\n\n#### Option A: Stable Release (recommended)\nInstall the latest stable version from PyPI:\n\n```bash\npip install whisperx\n```\n\n#### Option B: Development Version\nInstall the latest development version directly from GitHub (may be unstable):\n\n```bash\npip install git+https://github.com/m-bain/whisperx.git\n```\n\nIf already installed, update to the most recent commit:\n\n```bash\npip install git+https://github.com/m-bain/whisperx.git --upgrade\n```\n\n#### Option C: Development Mode\nIf you wish to modify the package, clone and install in editable mode:\n```bash\ngit clone https://github.com/m-bain/whisperX.git\ncd whisperX\npip install -e .\n```\n\n\u003e **Note**: The development version may contain experimental features and bugs. Use the stable PyPI release for production environments.\n\nYou may also need to install ffmpeg, rust etc. Follow openAI instructions here https://github.com/openai/whisper#setup.\n\n### Speaker Diarization\nTo **enable Speaker Diarization**, include your Hugging Face access token (read) that you can generate from [Here](https://huggingface.co/settings/tokens) after the `--hf_token` argument and accept the user agreement for the following models: [Segmentation](https://huggingface.co/pyannote/segmentation-3.0) and [Speaker-Diarization-3.1](https://huggingface.co/pyannote/speaker-diarization-3.1) (if you choose to use Speaker-Diarization 2.x, follow requirements [here](https://huggingface.co/pyannote/speaker-diarization) instead.)\n\n\u003e **Note**\u003cbr\u003e\n\u003e As of Oct 11, 2023, there is a known issue regarding slow performance with pyannote/Speaker-Diarization-3.0 in whisperX. It is due to dependency conflicts between faster-whisper and pyannote-audio 3.0.0. Please see [this issue](https://github.com/m-bain/whisperX/issues/499) for more details and potential workarounds.\n\n\n\u003ch2 align=\"left\" id=\"example\"\u003eUsage 💬 (command line)\u003c/h2\u003e\n\n### English\n\nRun whisper on example segment (using default params, whisper small) add `--highlight_words True` to visualise word timings in the .srt file.\n\n    whisperx path/to/audio.wav\n\n\nResult using *WhisperX* with forced alignment to wav2vec2.0 large:\n\nhttps://user-images.githubusercontent.com/36994049/208253969-7e35fe2a-7541-434a-ae91-8e919540555d.mp4\n\nCompare this to original whisper out the box, where many transcriptions are out of sync:\n\nhttps://user-images.githubusercontent.com/36994049/207743923-b4f0d537-29ae-4be2-b404-bb941db73652.mov\n\n\nFor increased timestamp accuracy, at the cost of higher gpu mem, use bigger models (bigger alignment model not found to be that helpful, see paper) e.g.\n\n    whisperx path/to/audio.wav --model large-v2 --align_model WAV2VEC2_ASR_LARGE_LV60K_960H --batch_size 4\n\n\nTo label the transcript with speaker ID's (set number of speakers if known e.g. `--min_speakers 2` `--max_speakers 2`):\n\n    whisperx path/to/audio.wav --model large-v2 --diarize --highlight_words True\n\nTo run on CPU instead of GPU (and for running on Mac OS X):\n\n    whisperx path/to/audio.wav --compute_type int8\n\n### Other languages\n\nThe phoneme ASR alignment model is *language-specific*, for tested languages these models are [automatically picked from torchaudio pipelines or huggingface](https://github.com/m-bain/whisperX/blob/f2da2f858e99e4211fe4f64b5f2938b007827e17/whisperx/alignment.py#L24-L58).\nJust pass in the `--language` code, and use the whisper `--model large`.\n\nCurrently default models provided for `{en, fr, de, es, it}` via torchaudio pipelines and many other languages via Hugging Face. Please find the list of currently supported languages under `DEFAULT_ALIGN_MODELS_HF` on [alignment.py](https://github.com/m-bain/whisperX/blob/main/whisperx/alignment.py). If the detected language is not in this list, you need to find a phoneme-based ASR model from [huggingface model hub](https://huggingface.co/models) and test it on your data.\n\n\n#### E.g. German\n    whisperx --model large-v2 --language de path/to/audio.wav\n\nhttps://user-images.githubusercontent.com/36994049/208298811-e36002ba-3698-4731-97d4-0aebd07e0eb3.mov\n\n\nSee more examples in other languages [here](EXAMPLES.md).\n\n## Python usage  🐍\n\n```python\nimport whisperx\nimport gc \n\ndevice = \"cuda\" \naudio_file = \"audio.mp3\"\nbatch_size = 16 # reduce if low on GPU mem\ncompute_type = \"float16\" # change to \"int8\" if low on GPU mem (may reduce accuracy)\n\n# 1. Transcribe with original whisper (batched)\nmodel = whisperx.load_model(\"large-v2\", device, compute_type=compute_type)\n\n# save model to local path (optional)\n# model_dir = \"/path/\"\n# model = whisperx.load_model(\"large-v2\", device, compute_type=compute_type, download_root=model_dir)\n\naudio = whisperx.load_audio(audio_file)\nresult = model.transcribe(audio, batch_size=batch_size)\nprint(result[\"segments\"]) # before alignment\n\n# delete model if low on GPU resources\n# import gc; gc.collect(); torch.cuda.empty_cache(); del model\n\n# 2. Align whisper output\nmodel_a, metadata = whisperx.load_align_model(language_code=result[\"language\"], device=device)\nresult = whisperx.align(result[\"segments\"], model_a, metadata, audio, device, return_char_alignments=False)\n\nprint(result[\"segments\"]) # after alignment\n\n# delete model if low on GPU resources\n# import gc; gc.collect(); torch.cuda.empty_cache(); del model_a\n\n# 3. Assign speaker labels\ndiarize_model = whisperx.DiarizationPipeline(use_auth_token=YOUR_HF_TOKEN, device=device)\n\n# add min/max number of speakers if known\ndiarize_segments = diarize_model(audio)\n# diarize_model(audio, min_speakers=min_speakers, max_speakers=max_speakers)\n\nresult = whisperx.assign_word_speakers(diarize_segments, result)\nprint(diarize_segments)\nprint(result[\"segments\"]) # segments are now assigned speaker IDs\n```\n\n## Demos 🚀\n\n[![Replicate (large-v3](https://img.shields.io/static/v1?label=Replicate+WhisperX+large-v3\u0026message=Demo+%26+Cloud+API\u0026color=blue)](https://replicate.com/victor-upmeet/whisperx) \n[![Replicate (large-v2](https://img.shields.io/static/v1?label=Replicate+WhisperX+large-v2\u0026message=Demo+%26+Cloud+API\u0026color=blue)](https://replicate.com/daanelson/whisperx) \n[![Replicate (medium)](https://img.shields.io/static/v1?label=Replicate+WhisperX+medium\u0026message=Demo+%26+Cloud+API\u0026color=blue)](https://replicate.com/carnifexer/whisperx) \n\nIf you don't have access to your own GPUs, use the links above to try out WhisperX. \n\n\u003ch2 align=\"left\" id=\"whisper-mod\"\u003eTechnical Details 👷‍♂️\u003c/h2\u003e\n\nFor specific details on the batching and alignment, the effect of VAD, as well as the chosen alignment model, see the preprint [paper](https://www.robots.ox.ac.uk/~vgg/publications/2023/Bain23/bain23.pdf).\n\nTo reduce GPU memory requirements, try any of the following (2. \u0026 3. can affect quality):\n1.  reduce batch size, e.g. `--batch_size 4`\n2. use a smaller ASR model `--model base`\n3. Use lighter compute type `--compute_type int8`\n\nTranscription differences from openai's whisper:\n1. Transcription without timestamps. To enable single pass batching, whisper inference is performed `--without_timestamps True`, this ensures 1 forward pass per sample in the batch. However, this can cause discrepancies the default whisper output.\n2. VAD-based segment transcription, unlike the buffered transcription of openai's. In the WhisperX paper we show this reduces WER, and enables accurate batched inference\n3.  `--condition_on_prev_text` is set to `False` by default (reduces hallucination)\n\n\u003ch2 align=\"left\" id=\"limitations\"\u003eLimitations ⚠️\u003c/h2\u003e\n\n- Transcript words which do not contain characters in the alignment models dictionary e.g. \"2014.\" or \"£13.60\" cannot be aligned and therefore are not given a timing.\n- Overlapping speech is not handled particularly well by whisper nor whisperx\n- Diarization is far from perfect\n- Language specific wav2vec2 model is needed\n\n\n\u003ch2 align=\"left\" id=\"contribute\"\u003eContribute 🧑‍🏫\u003c/h2\u003e\n\nIf you are multilingual, a major way you can contribute to this project is to find phoneme models on huggingface (or train your own) and test them on speech for the target language. If the results look good send a pull request and some examples showing its success.\n\nBug finding and pull requests are also highly appreciated to keep this project going, since it's already diverging from the original research scope.\n\n\u003ch2 align=\"left\" id=\"coming-soon\"\u003eTODO 🗓\u003c/h2\u003e\n\n* [x] Multilingual init\n\n* [x] Automatic align model selection based on language detection\n\n* [x] Python usage\n\n* [x] Incorporating  speaker diarization\n\n* [x] Model flush, for low gpu mem resources\n\n* [x] Faster-whisper backend\n\n* [x] Add max-line etc. see (openai's whisper utils.py)\n\n* [x] Sentence-level segments (nltk toolbox)\n\n* [x] Improve alignment logic\n\n* [ ] update examples with diarization and word highlighting\n\n* [ ] Subtitle .ass output \u003c- bring this back (removed in v3)\n\n* [ ] Add benchmarking code (TEDLIUM for spd/WER \u0026 word segmentation)\n\n* [x] Allow silero-vad as alternative VAD option\n\n* [ ] Improve diarization (word level). *Harder than first thought...*\n\n\n\u003ch2 align=\"left\" id=\"contact\"\u003eContact/Support 📇\u003c/h2\u003e\n\n\nContact maxhbain@gmail.com for queries.\n\n\u003ca href=\"https://www.buymeacoffee.com/maxhbain\" target=\"_blank\"\u003e\u003cimg src=\"https://cdn.buymeacoffee.com/buttons/default-orange.png\" alt=\"Buy Me A Coffee\" height=\"41\" width=\"174\"\u003e\u003c/a\u003e\n\n\n\u003ch2 align=\"left\" id=\"acks\"\u003eAcknowledgements 🙏\u003c/h2\u003e\n\nThis work, and my PhD, is supported by the [VGG (Visual Geometry Group)](https://www.robots.ox.ac.uk/~vgg/) and the University of Oxford.\n\nOf course, this is builds on [openAI's whisper](https://github.com/openai/whisper).\nBorrows important alignment code from [PyTorch tutorial on forced alignment](https://pytorch.org/tutorials/intermediate/forced_alignment_with_torchaudio_tutorial.html)\nAnd uses the wonderful pyannote VAD / Diarization https://github.com/pyannote/pyannote-audio\n\n\nValuable VAD \u0026 Diarization Models from:\n- [pyannote audio][https://github.com/pyannote/pyannote-audio]\n- [silero vad][https://github.com/snakers4/silero-vad]\n\nGreat backend from [faster-whisper](https://github.com/guillaumekln/faster-whisper) and [CTranslate2](https://github.com/OpenNMT/CTranslate2)\n\nThose who have [supported this work financially](https://www.buymeacoffee.com/maxhbain) 🙏\n\nFinally, thanks to the OS [contributors](https://github.com/m-bain/whisperX/graphs/contributors) of this project, keeping it going and identifying bugs.\n\n\u003ch2 align=\"left\" id=\"cite\"\u003eCitation\u003c/h2\u003e\nIf you use this in your research, please cite the paper:\n\n```bibtex\n@article{bain2022whisperx,\n  title={WhisperX: Time-Accurate Speech Transcription of Long-Form Audio},\n  author={Bain, Max and Huh, Jaesung and Han, Tengda and Zisserman, Andrew},\n  journal={INTERSPEECH 2023},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-bain%2FwhisperX","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-bain%2FwhisperX","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-bain%2FwhisperX/lists"}