{"id":13715328,"url":"https://github.com/MiscellaneousStuff/openai-whisper-cpu","last_synced_at":"2025-05-07T04:30:40.074Z","repository":{"id":63066019,"uuid":"542117324","full_name":"MiscellaneousStuff/openai-whisper-cpu","owner":"MiscellaneousStuff","description":"Improving transcription performance of OpenAI Whisper for CPU based deployment","archived":false,"fork":false,"pushed_at":"2022-11-02T11:12:10.000Z","size":37,"stargazers_count":237,"open_issues_count":9,"forks_count":19,"subscribers_count":10,"default_branch":"main","last_synced_at":"2024-11-14T03:34:26.761Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MiscellaneousStuff.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-09-27T14:05:54.000Z","updated_at":"2024-10-30T18:19:47.000Z","dependencies_parsed_at":"2022-11-12T05:01:34.480Z","dependency_job_id":null,"html_url":"https://github.com/MiscellaneousStuff/openai-whisper-cpu","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MiscellaneousStuff%2Fopenai-whisper-cpu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MiscellaneousStuff%2Fopenai-whisper-cpu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MiscellaneousStuff%2Fopenai-whisper-cpu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MiscellaneousStuff%2Fopenai-whisper-cpu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MiscellaneousStuff","download_url":"https://codeload.github.com/MiscellaneousStuff/openai-whisper-cpu/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252813620,"owners_count":21808361,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T00:00:57.610Z","updated_at":"2025-05-07T04:30:39.730Z","avatar_url":"https://github.com/MiscellaneousStuff.png","language":"Jupyter Notebook","funding_links":[],"categories":["Model Variants"],"sub_categories":[],"readme":"# OpenAI Whisper - CPU\n\n## About\n\nExperiments applying quantization methods to OpenAI Whisper ASR model\nto improve the inference speed and throughput on CPU-based deployments.\nThis is motivated by the fact that, although the Whisper model greatly\nimproves the accessibility of SOTA ASR and doesn't require depending\non the cloud for high quality transcription, many end users can not\nrun this model out-of-the-box as most consumer computers only contain\nCPUs and do not contain high performance GPUs.\n\nThis could lead to allowing the larger Whisper models to run faster\non laptops without a GPU.\n\nHardware for experiments: \\\nCPU - AMD Ryzen 5 5600X \\\nRAM - 32GB DDR4 \\\nGPU - Nvidia GeForce RTX 3060 Ti \\\nHDD - M.2 SSD \n\n## Usage\n\nFirstly, get the fork of the OpenAI Whisper repo with the\nmodifications needed for CPU dynamic quantization:\n\n```bash\ngit submodule init\ngit submodule update\n```\n\nAnd then install the module using:\n\n```bash\npip install -e ./whisper\n```\n\n### Explanation\n\nQuantization of the Whisper model requires changing the `Linear()`\nlayers within the model to `nn.Linear()`. This is because you need\nto specifiy which layer types to dynamically quantize, such as:\n\n```python\nquantized_model = torch.quantization.quantize_dynamic(\n    model_fp32, {torch.nn.Linear}, dtype=torch.qint8\n)\n```\n\nHowever the whisper model is designed to be adaptable, i.e.\nit can run at different precisions, so the `Linear()` layer contains\ncustom code to account for this. However, this is not required for\nthe quantized model. You can either change the `Linear()` layers in\n\"/whisper/whisper/model.py\" yourself, or you can just use the above\ninstallation instructions.\n\n## Results\n\nTest audio is the first 30 seconds of: \\\nhttps://www.youtube.com/watch?v=oKOtzIo-uYw\n\n| Device | Whisper Model | Data Type | Linear Layer | Inference Time |\n| --- | --- | ----------- | --- | --- |\n| GPU | tiny | fp32 | Linear | 0.5 |\n| CPU | tiny  | fp32 | nn.Linear | 2.3 |\n| CPU | tiny  | qint8 (quant) | nn.Linear | 3.1 (0.74x slowdown) |\n\nTiny quantized model is 9.67x faster than real time. \\\nTiny quantized model is 0.74x slower than the original model.\n\n| Device | Whisper Model | Data Type | Linear Layer | Inference Time |\n| --- | --- | ----------- | --- | --- |\n| GPU | base | fp32 | Linear | 0.6 |\n| CPU | base  | fp32 | nn.Linear | 5.2 |\n| CPU | base  | qint8 (quant) | nn.Linear | 3.2 (1.62x speedup) |\n\nBase quantized model is 9.37x faster than real time. \\\nBase quantized model is 1.62x faster than the original model.\n\n| Device | Whisper Model | Data Type | Linear Layer | Inference Time |\n| --- | --- | ----------- | --- | --- |\n| GPU | small | fp32 | Linear | 0.7 |\n| CPU | small | fp32 | nn.Linear | 19.1s |\n| CPU | small | qint8 (quant) | nn.Linear | 6.9s (2.76x speedup) |\n\nSmall quantized model is 4.34x faster than real time. \\\nSmall quantized model is 2.76x faster than the original model.\n\n| Device | Whisper Model | Data Type | Linear Layer | Inference Time |\n| --- | --- | ----------- | --- | --- \n| GPU | medium | fp32 | Linear | 1.7s |\n| CPU | medium | fp32 | nn.Linear | 60.7 |\n| CPU | medium | qint8 (quant) | nn.Linear | 23.1 (2.62x speedup) |\n\nMedium quantized model is 1.29x faster than real time. \\\nMedium quantized model is 2.62x faster than the original model.\n\n# Docker\n\nBuild the docker image.   \n\n``` \ndocker build -t whisper-cpu . \n```\nRun the quantized model.   \n\n```\ndocker run --rm -v \"$(pwd)/audio\":/usr/src/app/audio -v \"$(pwd)/script\":/usr/src/app/script whisper-cpu python3 ./script/custom_whisper.py audio/path_to_dir_or_audio_file --language English --model medium.en \n```\n\n- ```-v \"$(pwd)/audio\":/usr/src/app/audio``` this creates a volume to give docker access to your audio files.\n- ```-v \"$(pwd)/script\":/usr/src/app/script``` this volume gives docker access to the custom start script. Transcription results are also stored here.\n\n- Note: you might want to adjust ```./script/custom_whisper.py``` for your own needs.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMiscellaneousStuff%2Fopenai-whisper-cpu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMiscellaneousStuff%2Fopenai-whisper-cpu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMiscellaneousStuff%2Fopenai-whisper-cpu/lists"}