{"id":27945103,"url":"https://github.com/collabora/whisper-finetuning","last_synced_at":"2025-07-15T19:42:58.307Z","repository":{"id":276123615,"uuid":"924117895","full_name":"collabora/whisper-finetuning","owner":"collabora","description":"Whisper finetuning","archived":false,"fork":false,"pushed_at":"2025-04-09T07:36:25.000Z","size":25,"stargazers_count":12,"open_issues_count":0,"forks_count":1,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-05-07T12:55:46.519Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/collabora.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-29T12:57:54.000Z","updated_at":"2025-04-24T15:19:13.000Z","dependencies_parsed_at":"2025-04-09T08:37:28.541Z","dependency_job_id":null,"html_url":"https://github.com/collabora/whisper-finetuning","commit_stats":null,"previous_names":["collabora/whisper-finetuning"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collabora%2Fwhisper-finetuning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collabora%2Fwhisper-finetuning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collabora%2Fwhisper-finetuning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collabora%2Fwhisper-finetuning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/collabora","download_url":"https://codeload.github.com/collabora/whisper-finetuning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252882788,"owners_count":21819155,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T12:55:50.412Z","updated_at":"2025-05-07T12:55:51.875Z","avatar_url":"https://github.com/collabora.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Whisper Finetuning\n\n### Overview\n\nThis repo provides the scripts to fine-tune `openai/whisper`. The repository includes tools for data preprocessing, converting data to WebDataset format, and fine-tuning whisper.\n\n\n### Datasets\nThe model was fine-tuned on the following datasets:\n\n- **Shrutilipi (AI4Bharat)**\nA corpus of 6400+ hours across 12 Indian languages.\nHindi subset contains approximately 1600 hours.\n\n- **IITM Madras SpringLab**\nContains mock conversations and monologues on diverse topics.\nHindi subset covers approximately 900 hours.\n\n- **Mozilla Foundation’s Common Voice 11.0**\nReleased under CC-1.0, adding robustness through community-contributed voice data.\n\n- **Google Fleurs**\nUsed for testing, providing a comprehensive benchmark with its extensive multilingual dataset, licensed under CC-4.0.\n\n\n### Getting Started\n1. **System Requirements**\n- **CUDA**: \u003e= 11.8\n- **Python**: \u003e= 3.9\n- **Hardware**: NVIDIA 4090 recommended for training or any other NVIDIA GPU for acceleration.\n- **Docker Environment**: We used the Docker container `pytorch/pytorch:2.5.0-cuda12.1-cudnn9-devel` for our environment.\n\n2. **Install Requirements**\n\n```bash\n$ pip install -r requirements.txt\n```\n\n3. **Normalization**\n\nFor using Indic normalization, clone the `indic_nlp_resources` repository inside `whisper-finetuning`:\n\n```bash\n$ git clone https://github.com/anoopkunchukuttan/indic_nlp_resources.git\n```\nIn Hindi, the meaning of a sentence can drastically change if diacritics or characters are removed, making normalization a critical\npart of the pipeline. Consider this example:\n\n```\nहमने उस उम्मीदवार को चुना।\n```\n\n- Whisper's Default Normalization:\n\nWhisper applies aggressive text normalization, often stripping diacritics and compressing words. Here's how the same sentence looks\nafter Whisper normalization:\n```\n'हमन उस उमम दव र क चन'\n```\nThe removal of diacritics and loss of word boundaries result in text that is difficult to interpret and often meaningless.\n\n- Indic Normalization to the Rescue:\n\nInstead of Whisper's default normalization, we employed\n[Indic Normalization from the IndicNLP Library](https://github.com/anoopkunchukuttan/indic_nlp_library/blob/4cead0ae6c78fe9a19a51ef679f586206df9c476/indicnlp/normalize/indic_normalize.py#L325), which retains diacritics and complex characters, producing more linguistically accurate transcriptions:\n\n```\nहमने उस उम्मीदवार को चुना।\n```\n\nWhile Whisper's default normalization might reduce Word Error Rate (WER) on numeric benchmarks, it sacrifices semantic accuracy. For Hindi,\nmaintaining diacritics and preserving complex characters is vital for transcription quality, even if it slightly increases the WER. This\ntrade-off ensures that the transcriptions are meaningful and contextually accurate.\n\n\n### Data Preprocessing\n\nPreprocessing requires the dataset to be in Hugging Face audio dataset format. Please refer to the Hugging Face documentation for details on preparing your data.\n\nSince Whisper's maximum audio sample duration is 30 seconds, ensure that all audio samples are around 30 seconds:\n\n- If they are below 30 seconds, merge samples to reach ~30 seconds.\n- If they are over 30 seconds, either discard them or cut them short.\n\nAnd also make similar adjustments to their corresponding transcripts. Once we have the dataset in huggingface format we go ahead and preprocess the datasets i.e., preprocess all your huggingface format datasets(if you have multiple sources of data) and put them into a directory for e.g. `preprocessed_hf_datasets`:\n\n```bash\n$ python preprocess.py \\\n        --dataset_type huggingface \\\n        --dataset_name mozilla-foundation/common_voice_11_0 \\\n        --language hi \\\n        --split train \\\n        --save_dir ./preprocessed_hf_datasets/train/mozilla_common_voice_11_0 \\\n        --use_indic_norm\n```\nThis command preprocesses the Common Voice 11.0 Hindi dataset by ensuring the audio samples are the right duration, applying Indic normalization (if enabled), and saving the processed data to the specified directory.\n\n\n### Webdataset Conversion [Optional]\n\nAfter preprocessing all your train \u0026 dev datasets, convert them into a shuffled webdataset format which is crucial for faster training when using large datasets.\n\nThis step is optional and can be used if you have a large dataset i.e., thousands of hours of data otherwise stick to Hugging Face format, use the preprocessed dataset saved to the disk in the previous\nstep.\n\n- Write train shards\n\n```bash\n$ python convert_to_webdataset.py \\\n        --preprocessed_datasets ./preprocessed_hf_datasets/train/ \\\n        --output_dir ./webdataset_hi \\\n        --shard_size 1000 \\\n        --num_proc 4 \\\n        --shard_start_idx 0\n```\nLets assume that the train dataset had 50000 samples and we now have written 50 shards.\n\n- Write evaluation shards and set `--shard_start_idx=51`\n\n```bash\n$ python convert_to_webdataset.py \\\n        --preprocessed_datasets ./preprocessed_hf_datasets/test/ \\\n        --output_dir ./webdataset_hi \\\n        --shard_size 1000 \\\n        --num_proc 4 \\\n        --shard_start_idx 51\n```\nLets assume we have 2 test shards\n\n\n### Start training !!!\n\nThis command initiates the training process using the fine-tuning parameters provided, including the use of WebDataset data, specified model size, and Indic normalization. It also integrates with Weights \u0026 Biases for experiment tracking.\n\n- Train with webdataset:\n\n```bash\n$ python train.py \\\n        --model_size tiny \\\n        --language hi \\\n        --batch_size 32 \\\n        --grad_acc 4 \\\n        --learning_rate 3e-5 \\\n        --warmup_steps 500 \\\n        --max_steps 10000 \\\n        --save_steps 1000 \\\n        --eval_steps 1000 \\\n        --use_webdataset \\\n        --webdataset_path_or_url ./webdataset_hi \\\n        --train_shards 00000..00050 \\\n        --eval_shards 00051..00052 \\\n        --use_wandb \\\n        --wandb_project_name whisper-hindi \\\n        --wandb_run_name run0-tiny \\\n        --use_indic_norm\n```\n\n- Train with preprocessed huggingface train \u0026 test datasets:\n\n```bash\n$ python train.py \\\n        --model_size tiny \\\n        --language hi \\\n        --batch_size 32 \\\n        --grad_acc 4 \\\n        --learning_rate 3e-5 \\\n        --warmup_steps 500 \\\n        --max_steps 10000 \\\n        --save_steps 1000 \\\n        --eval_steps 1000 \\\n        --hf_train_dataset_path ./preprocessed_hf_datasets/train/mozilla_common_voice_11_0 \\\n        --hf_test_dataset_path ./preprocessed_hf_datasets/test/mozilla_common_voice_11_0 \\\n        --use_wandb \\\n        --wandb_project_name whisper-hindi \\\n        --wandb_run_name run0-tiny \\\n        --use_indic_norm\n```\n\n### Evaluation\n\nEvaluate your finetuned model on the `google/fleurs` test set using:\n\n```bash\n$ python test.py \\\n        --model_path ./whisper-tiny-hi-finetuned-run0-tiny \\\n        --model_size tiny \\\n        --language hi \\\n        --use_indic_norm\n```\n\n### Model Performance\n\n- Baseline WER results on `google/fleurs -- hindi` (Indic Normalization)\n\n| Model Size | Whisper Norm | Indic Norm | \n|------------|------------- |------------| \n| Tiny       | 172.60       | 196.57     | \n| Base       | 149.17       | 160.58     | \n| Small      | 67.37        | 89.73      |\n\n- Fine-tuned Whisper WER Results on `google/fleurs -- hindi`:\n\n| Model Size | Whisper Normalization | Indic Normalization |\n|------------|-----------------------|---------------------|\n| Tiny       | 14.21                 | 22.15              |\n| Base       | 11.78                 | 19.44              |\n| Small      | 10.11                 | 17.35              |\n\n### Acknowledgments\nWe acknowledge the contributions of:\n\n- [AI4Bharat](https://ai4bharat.iitm.ac.in/) for the [Shrutilipi dataset](https://ai4bharat.iitm.ac.in/datasets/shrutilipi).\n- [IIT Madras SpringLab](https://asr.iitm.ac.in/) for the [SpringX-Hindi dataset](https://asr.iitm.ac.in/dataset).\n- [mozilla-foundation](https://foundation.mozilla.org/en/) for the [common_voice_11_0](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0) hindi dataset.\n- [google](https://huggingface.co/google) for the [fleurs](https://huggingface.co/datasets/google/fleurs) dataset.\n- [IndicNLP Library](https://github.com/anoopkunchukuttan/indic_nlp_library) for their powerful Indic normalization tools.\n\n### Contributing\nContributions are welcome! If you would like to contribute to this project, please fork the repository and submit a pull request. For major changes, please open an issue first to discuss what you would like to change. For any bugs or issues, please report them via the repository's issue tracker.\n\n### License\nThis project is licensed under the MIT License.\n\n### Citations\n\n**Indic NLP LIbrary Citation**\n```bibtex\n@misc{kunchukuttan2020indicnlp,\n  author = \"Anoop Kunchukuttan\",\n  title = \"{The IndicNLP Library}\",\n  year = \"2020\",\n  howpublished={\\url{https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf}}\n}\n```\n\n**AI4Bharat - Shrutilipi dataset**\n```bibtex\n@misc{https://doi.org/10.48550/arxiv.2208.12666,\n  doi = {10.48550/ARXIV.2208.12666},\n  url = {https://arxiv.org/abs/2208.12666},\n  author = {Bhogale, Kaushal Santosh and Raman, Abhigyan and Javed, Tahir and Doddapaneni, Sumanth and Kunchukuttan, Anoop and Kumar, Pratyush and Khapra, Mitesh M.},\n  title = {Effectiveness of Mining Audio and Text Pairs from Public Data for Improving ASR Systems for Low-Resource Languages},\n  publisher = {arXiv},\n  year = {2022},\n  copyright = {arXiv.org perpetual, non-exclusive license}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcollabora%2Fwhisper-finetuning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcollabora%2Fwhisper-finetuning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcollabora%2Fwhisper-finetuning/lists"}