{"id":40374181,"url":"https://github.com/proger/uk","last_synced_at":"2026-01-20T11:33:52.002Z","repository":{"id":58634450,"uuid":"483001336","full_name":"proger/uk","owner":"proger","description":"Фонограми та синтагми: інструменти обробки","archived":false,"fork":false,"pushed_at":"2025-06-21T23:02:27.000Z","size":8499,"stargazers_count":21,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-22T00:17:58.480Z","etag":null,"topics":["dataset-generation","forced-alignment","hmm","kaldi","speech-recognition","ukrainian","ukrainian-language"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/proger.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-04-18T21:23:42.000Z","updated_at":"2025-06-21T23:02:31.000Z","dependencies_parsed_at":"2025-04-06T23:20:15.108Z","dependency_job_id":"62788910-de02-44a4-aa4e-8a2c1edf5f9a","html_url":"https://github.com/proger/uk","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/proger/uk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proger%2Fuk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proger%2Fuk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proger%2Fuk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proger%2Fuk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/proger","download_url":"https://codeload.github.com/proger/uk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proger%2Fuk/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28602469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-20T10:46:13.255Z","status":"ssl_error","status_checked_at":"2026-01-20T10:42:51.865Z","response_time":117,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset-generation","forced-alignment","hmm","kaldi","speech-recognition","ukrainian","ukrainian-language"],"created_at":"2026-01-20T11:33:51.829Z","updated_at":"2026-01-20T11:33:51.990Z","avatar_url":"https://github.com/proger.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Подбай про фонограми та синтагми з uk\n\nThis recipe trains a Ukrainian GMM-HMM on Common Voice 9.0 to use\nfor segmentation of long audio files into short utterances using its full transcript.\n\nIt avoids performing full large vocabulary speech recognition\nby limiting its search options to word sequences from the input transcript.\n\n## Recipe\n\n### Install Kaldi\n\n```bash\ngit clone https://github.com/kaldi-asr/kaldi\ncd kaldi\ncat INSTALL\n# install kaldi to $HOME/kaldi\n```\n\nSpeed run through Kaldi installation instructions:\n\n```\ncd tools\n./extras/check_dependencies.sh\n# Pay attention to dependency errors.\n# If you're on macOS you don't need OpenBLAS.\n# You won't need python2.7 and subversion.\n# Make sure python command runs some python:\nln -sf $(which python3) $HOME/.local/bin/python\n# On Ubuntu you can do this instead:\napt-get install python-is-python3\n\n# Ignore all subsequent dependency checks.\necho \u003e extras/check_dependencies.sh\n\n# Build all tools (primarily openfst and pocolm)\nmake -j8\n./extras/install_pocolm.sh\n\n# Build kaldi itself\ncd ../src\n./configure --shared\nmake -j clean depend\nmake -j8\n```\n\n### Install uk\n\n```\npip3 install --editable .\n```\n\n### Train Ukrainian using Common Voice 10.0\n\nPrerequisites:\n\n- Request access at https://huggingface.co/datasets/mozilla-foundation/common_voice_10_0\n- Get a Hugging Face token at https://huggingface.co/settings/tokens\n- run `huggingface-cli login`\n\n```bash\n# bring parts of kaldi into $PATH\nsource path.sh\n\n# prepare dataset for training\n# FWIW downloading takes longer than training :)\npython3 -m uk.prepare_dataset\n\n# progressively train mono, tri, tri2b, tri3b models\npython3 -m uk.train_gmm exp/tri3b\n```\n\n### Train English using LibriSpeech train-clean-100\n\n```bash\npython3 -m uk.prepare_dataset --dataset darkproger/librispeech_asr --subset train.clean.100 --split full --alphabet latin\n\n# bring parts of kaldi into $PATH\nsource path.sh\n\n# make a subset of librispeech\nutils/subset_data_dir.sh --per-spk data/darkproger/librispeech_asr/train.clean.100/full 30 data/librispeech_mini\n\n# progressively train mono, tri, tri2b, tri3b models\npython3 -m uk.train_gmm -d data/librispeech_mini --lexicon data/local/dict/english_mfa_reference.dict exp/english\n```\n\n### Align Text to Audio and Trim Silence\n\nSome Ukrainian unvoiced sounds are considered noise by VADs trained on English.\nIt's useful to perform segmentation using an acoustic model instead.\n\n\nUses the [Lada dataset](https://github.com/egorsmkv/ukrainian-tts-datasets/tree/main/lada).\n`data/dataset_lada` comes from `dataset_lada_ogg.zip` file.\n\nAssumes a Ukrainian model to be present in `exp/tri3b`, see above.\n\n```bash\n# Prepare data directory. We need text, utt2spk and wav.scp\nmkdir -p data/lada\n\u003c data/dataset_lada/accept/metadata.jsonl  jq  -rc '[.file,.orig_text] | @tsv' | python3 -m uk.clean_text | sed 's,.ogg,,' | sort \u003e data/lada/text\nfind data/dataset_lada/accept/ -name '*.ogg' | sort | awk -F/ '{s=$4;sub(\".ogg\",\"\",s); print s, \"lada\"}' \u003e data/lada/utt2spk\nfind data/dataset_lada/accept/ -name '*.ogg' | sort | awk -F/ '{s=$4;sub(\".ogg\",\"\",s); print s, \"ffmpeg -nostdin -i data/\"$0\" -ac 1 -acodec pcm_s16le -f wav - |\"}' \u003e data/lada/wav.scp\n\nutils/fix_data_dir.sh data/lada\n\n# Prepare lang. Makes every word pronunciation known by running G2P.\npython3 -m uk.prepare_dict -o exp/dict_base data/local/dict/lexicon_common_voice_uk.txt\npython3 -m uk.prepare_lang -d exp/dict_base -o data/lada --text data/lada/text\n\n# Compute and export alignments to exp/lada\nsteps/make_mfcc.sh data/lada\npython3 -m uk.align_utterances -a exp/ali -l data/lada/lang -m exp/tri3b data/lada\n\n# Write a segments file for utterances without leading and trailing silence\npython3 -m uk.trim_silence -l data/lada/lang/lang exp/ali/ \u003e data/lada/segments\n\n# Extract separate wavs\npython3 -m uk.extract_segments -o data/lada_seg -i data/lada/wav.scp data/lada/segments\n```\n\n### Segment Parallel Speech and Text Data\n\n```bash\n# get example data to align\ngit clone https://github.com/lang-uk/semesyuk-to-text\n\n# note: text can contain extra words\ncat semesyuk-to-text/texts/tokenized/semesyuk_farshrutka/01_prologue.txt | python3 -m uk.nlp_uk_tokens \\\n    \u003e data/local/semesyuk_farshrutka/01_prologue.txt\n\n# run segmentation using tri3b model\n# in this example it outputs a kaldi-style data directory to data/semesyuk_farshrutka_prologue\npython3 -m uk.segment_long_utterances -w exp/segment1 -o data/semesyuk_farshrutka_prologue \\\n    data/local/semesyuk_farshrutka/01_prologue.txt \\\n    semesyuk-to-text/audio/raw/semesyuk_farshrutka/01_prologue.mp3\n\n# upload result to wandb\npython3 -m uk.share data/semesyuk_farshrutka_prologue\n```\n\n## Acknowledgements\n\n- Common Voice Dataset https://commonvoice.mozilla.org/\n- G2P model https://github.com/kosti4ka/ukro_g2p\n- Baseline https://github.com/lang-uk/semesyuk-to-text\n- Alignment method (paper inside) https://github.com/kaldi-asr/kaldi/blob/master/egs/wsj/s5/steps/cleanup/segment_long_utterances.sh\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproger%2Fuk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproger%2Fuk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproger%2Fuk/lists"}