{"id":15652049,"url":"https://github.com/kurianbenoy/whisper_normalizer","last_synced_at":"2025-04-11T02:07:33.132Z","repository":{"id":144273250,"uuid":"617026254","full_name":"kurianbenoy/whisper_normalizer","owner":"kurianbenoy","description":"A python package for whisper normalizer","archived":false,"fork":false,"pushed_at":"2025-03-02T14:04:23.000Z","size":574,"stargazers_count":54,"open_issues_count":8,"forks_count":11,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-11T02:07:22.604Z","etag":null,"topics":["asr","asr-benchmark","jupyter-notebook","nbdev","normalizers","openai","whisper"],"latest_commit_sha":null,"homepage":"https://kurianbenoy.github.io/whisper_normalizer/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kurianbenoy.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.bak","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"buy_me_a_coffee":"kurianbenoy2"}},"created_at":"2023-03-21T14:57:13.000Z","updated_at":"2025-04-09T09:12:59.000Z","dependencies_parsed_at":null,"dependency_job_id":"1f7a0354-96b3-433a-a8d6-a38a1b0e9a7e","html_url":"https://github.com/kurianbenoy/whisper_normalizer","commit_stats":{"total_commits":71,"total_committers":3,"mean_commits":"23.666666666666668","dds":0.09859154929577463,"last_synced_commit":"09861ac30a3bf95535e26467c51d0bfab25171fb"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kurianbenoy%2Fwhisper_normalizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kurianbenoy%2Fwhisper_normalizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kurianbenoy%2Fwhisper_normalizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kurianbenoy%2Fwhisper_normalizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kurianbenoy","download_url":"https://codeload.github.com/kurianbenoy/whisper_normalizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248328165,"owners_count":21085261,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asr","asr-benchmark","jupyter-notebook","nbdev","normalizers","openai","whisper"],"created_at":"2024-10-03T12:41:07.634Z","updated_at":"2025-04-11T02:07:33.107Z","avatar_url":"https://github.com/kurianbenoy.png","language":"Jupyter Notebook","funding_links":["https://buymeacoffee.com/kurianbenoy2"],"categories":[],"sub_categories":[],"readme":"# whisper_normalizer\n\n\n\u003c!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! --\u003e\n\n\u003e A python package for text standardisation/normalization. It uses\n\u003e normalization algorithm mentioned in OpenAI whisper paper. Using\n\u003e Whisper normalization can cause issues in Indic languages and other\n\u003e low resource languages when using\n\u003e [`BasicTextNormalizer`](https://kurianbenoy.github.io/whisper_normalizer/basic.html#basictextnormalizer).\n\u003e So normalization in Indic languages is also implemented in this\n\u003e package which was derived from\n\u003e [indic-nlp-library](https://github.com/anoopkunchukuttan/indic_nlp_library).\n\n[![Github\nlicense](https://img.shields.io/github/license/kurianbenoy/whisper_normalizer.svg)](https://github.com/kurianbenoy/whisper_normalizer/blob/main/LICENSE)\n[![Github\nStars](https://img.shields.io/github/stars/kurianbenoy/whisper_normalizer.svg?colorA=orange\u0026colorB=orange\u0026logo=github)](https://github.com/kurianbenoy/whisper_normalizer/stargazers)\n[![PyPI\nversion](https://img.shields.io/pypi/v/whisper-normalizer.svg?style=flat-square\u0026logo=pypi\u0026logoColor=white)](https://pypi.org/project/whisper-normalizer/)\n\u003c!-- [![Downloads](https://static.pepy.tech/badge/whisper-normalizer)](https://github.com/kurianbenoy/whisper_normalizer) --\u003e\n\u003c!-- [![python version](https://img.shields.io/badge/Python-%3E=3.7-blue)](https://github.com/kurianbenoy/whisper_normalizer)\n[![python version](https://img.shields.io/badge/Python-\u003c3.12-blue)](https://github.com/kurianbenoy/whisper_normalizer) --\u003e\n\nThis package is a python implementation of the text\nstandardisation/normalization approach which is being used in OpenAI\nwhisper. The code was originally being released as open-source in\n[Whisper source code](https://github.com/openai/whisper). More details\nabout the text normalization approach used by whisper can be found on\nAppendix Section C pp.21 the paper [Robust Speech Recognition via\nLarge-Scale Weak Supervision](https://cdn.openai.com/papers/whisper.pdf)\nby OpenAI team.\n\n## Installation of package\n\n``` sh\npip install whisper_normalizer\n```\n\nor from github repository\n\n``` sh\npip install git+https://github.com/kurianbenoy/whisper_normalizer.git\n```\n\n## How to use the package\n\n- I made a video walk through on how to use the `whisper_normalizer`\n  python package.\n\n[Colab Notebook Link of walk\nthrough](https://colab.research.google.com/gist/kurianbenoy/7d27d9ec193a4a97ec7821235bddc506/hello-world_whisper_normalizer.ipynb)\n\n[Github Gist Link of walk\nthrough](https://gist.github.com/kurianbenoy/7d27d9ec193a4a97ec7821235bddc506)\n\n[![Hello world to\nwhisper_normalizer](https://img.youtube.com/vi/c7trf0zul6g/0.jpg)](https://www.youtube.com/watch?v=c7trf0zul6g)\n\n## Why should we normalize/standardize text?\n\n- In ASR systems it’s important to normalize the text to reduce\n  unintentional penalties in metrics like WER, CER etc.\n- Text normalization/standardization is process of converting texts in\n  different styles into a standardized form, which is a best-effort\n  attempt to penalize only when a word error is caused by actually\n  mistranscribing a word, and not by formatting or punctuation\n  differences.(from [Whisper\n  paper](https://cdn.openai.com/papers/whisper.pdf))\n\n## Why use this python package?\n\nThis package is a python implementation of the text\nstandardisation/normalization approach which is being used in OpenAI\nwhisper text normalizer. If you want to use just text normalization\nalone, it’s better to use this instead reimplementing the same thing.\nOpenAI approach of text normalization is very helpful and is being used\nas normalization step when evaluating competitive models like\n[AssemblyAI Conformer-1\nmodel](https://www.assemblyai.com/blog/conformer-1/).\n\n## Models evaluated using Whisper normalization\n\n- OpenAI Whisper\n- Massively Multilingual Speech (MMS) models by Meta\n- Conformer 1 by AssemblyAI\n- Conformer 2 by AssemblyAI\n\n## How to use\n\nOpenAI open source approach of text normalization/standardization is\nmentioned in detail Appendix Section C pp.21 the paper [Robust Speech\nRecognition via Large-Scale Weak\nSupervision](https://cdn.openai.com/papers/whisper.pdf).\n\nWhisper Normalizer by default comes with two classes\n[`BasicTextNormalizer`](https://kurianbenoy.github.io/whisper_normalizer/basic.html#basictextnormalizer)\nand\n[`EnglishTextNormalizer`](https://kurianbenoy.github.io/whisper_normalizer/english.html#englishtextnormalizer)\n\nYou can use the same thing in this package as follows:\n\n``` python\nfrom whisper_normalizer.english import EnglishTextNormalizer\n\nenglish_normalizer = EnglishTextNormalizer()\nenglish_normalizer(\"I'm a little teapot, short and stout. Tip me over and pour me out!\")\n```\n\n    'i am a little teapot short and stout tip me over and pour me out'\n\n``` python\nfrom whisper_normalizer.basic import BasicTextNormalizer\n\nnormalizer = BasicTextNormalizer()\nnormalizer(\"I'm a little teapot, short and stout. Tip me over and pour me out!\")\n```\n\n    'i m a little teapot short and stout tip me over and pour me out '\n\n## Using BasicTextNormalizer in your mother tongue might be a bad idea\n\nWhisper Text Normalizer is not always recommended to be used. [Dr Kavya\nManohar](https://www.linkedin.com/in/kavya-manohar/) has written a\nblogpost on why it might be a bad idea on her [blopost titled Indian\nLanguages and Text Normalization: Part\n1](https://kavyamanohar.com/post/indic-normalizer/).\n\n## This model extends Whisper_normalizer to support Indic languages as well.\n\nThe logic for normalization in Indic languages is derived from\n[indic-nlp-library](https://github.com/anoopkunchukuttan/indic_nlp_library).\nThe logic for Malayalam normalization is expanded beyond the Indic NLP\nlibrary by\n[`MalayalamNormalizer`](https://kurianbenoy.github.io/whisper_normalizer/1b.indic_normalizer.html#malayalamnormalizer).\n\n``` python\nfrom whisper_normalizer.indic_normalizer import MalayalamNormalizer\n\nnormalizer = MalayalamNormalizer()\nnormalizer(\"എന്റെ കമ്പ്യൂട്ടറിനു് എന്റെ ഭാഷ.\")\n```\n\n    'എന്റെ കമ്പ്യൂട്ടറിന് എന്റെ ഭാഷ.'\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkurianbenoy%2Fwhisper_normalizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkurianbenoy%2Fwhisper_normalizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkurianbenoy%2Fwhisper_normalizer/lists"}