{"id":21529593,"url":"https://github.com/sooftware/ksponspeech","last_synced_at":"2025-04-09T23:51:40.983Z","repository":{"id":41059152,"uuid":"268727282","full_name":"sooftware/ksponspeech","owner":"sooftware","description":"Pre-processing KsponSpeech corpus (Korean Speech dataset) provided by AI Hub.","archived":false,"fork":false,"pushed_at":"2021-12-24T06:27:07.000Z","size":137,"stargazers_count":90,"open_issues_count":2,"forks_count":19,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-09T23:51:36.084Z","etag":null,"topics":["aihub","korean-speech","kospeech","ksponspeech"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sooftware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-02T07:09:49.000Z","updated_at":"2025-03-19T06:13:55.000Z","dependencies_parsed_at":"2022-08-27T03:08:07.498Z","dependency_job_id":null,"html_url":"https://github.com/sooftware/ksponspeech","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sooftware%2Fksponspeech","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sooftware%2Fksponspeech/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sooftware%2Fksponspeech/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sooftware%2Fksponspeech/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sooftware","download_url":"https://codeload.github.com/sooftware/ksponspeech/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248131454,"owners_count":21052819,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aihub","korean-speech","kospeech","ksponspeech"],"created_at":"2024-11-24T01:58:16.815Z","updated_at":"2025-04-09T23:51:40.962Z","avatar_url":"https://github.com/sooftware.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KsponSpeech\n#### Pre-processing KsponSpeech corpus provided by AI Hub\n   \nIt's been a while since [KsponSpeech](https://www.mdpi.com/2076-3417/10/19/6936) was released, but it's hard to compare performance because there's no established preprocessing method. So we're revealing the pre-processing method we used in the [KoSpeech](https://github.com/sooftware/KoSpeech) project. This project provides processing in characters, subwords, and grapheme units.    \n  \n## Intro\n\n`KsponSpeech-preprocess` is repository for pre-processing `KsponSpeech corpus` provided by AI Hub.  \n**KsponSpeech corpus** is a **1000h** Korean speech data corpus provided by [AI Hub](http://www.aihub.or.kr/) in Korea.   \nAnyone can download this dataset just by applying. The transcription rules can see [here](http://www.aihub.or.kr/sites/default/files/2019-12/%ED%95%9C%EA%B5%AD%EC%96%B4%20%EC%9D%8C%EC%84%B1%20%EC%A0%84%EC%82%AC%EA%B7%9C%EC%B9%99%20v1.0.pdf).  \n  \nYou can pre-process in various output-units, such as ***character, subword, grapheme***  \nWe will explain the details in the **Output-Unit** part below.\n   \n## Prerequisites\n* Pandas: `pip install pandas` (Refer [here](https://github.com/pandas-dev/pandas) for problem installing Pandas)  \n* Sentencepiece: `pip install sentencepiece` (Refer [here](https://github.com/google/sentencepiece) for problem installing Sentencepiece) \n  \n## Usage\n  \n1. Set options in [run.sh](https://github.com/sooftware/KsponSpeech-preprocess/blob/master/run.sh)  \n  \n```\npython main.py --dataset_path $DATASET_PATH --vocab_dest $VOCAB_DEST --output_unit $OUTPUT_UNIT --preprocess_mode $PREPROCESS_MODE --vocab_size $VOCAB_SIZE\n```\n  \n2. Run [run.sh](https://github.com/sooftware/KsponSpeech-preprocess/blob/master/run.sh)  \n```shell\n$ ./run.sh\n```\n  \n3. Leave the computer running for hours.  \n   \n## Preprocess\n  \nYou can choose between phonetic transcription and spelling transcription to preprocess.  \n  \n* Raw data\n```\nb/ (70%)/(칠 십 퍼센트) 확률이라니 아/ (뭐+ 뭔)/(모+ 몬) 소리야 진짜 (100%)(백 프로)가 왜 안돼? n/\n``` \n  \n* Delete noise labels, such as b/, n/, / ..\n```\n(70%)/(칠 십 퍼센트) 확률이라니 아/ (뭐+ 뭔)/(모+ 몬) 소리야 진짜 (100%)(백 프로)가 왜 안돼?\n```\n  \n* Delete labels such as '/', '*', '+', etc. (used for interjection representation)\n```\n(70%)/(칠 십 퍼센트) 확률이라니 아 (뭐 뭔)/(모 몬) 소리야 진짜 (100%)(백 프로)가 왜 안돼?\n```\n  \n* Option1 : phonetic transcript\n```\n칠 십 퍼센트 확률이라니 아 모 몬 소리야 진짜 백 프로가 왜 안돼?\n```\n\n* Option2 : spelling transcript\n```\n70% 확률이라니 아 뭐 뭔 소리야 진짜 100%가 왜 안돼?\n```\n  \n## Output-Unit\n   \nThis project provides processing in characters, subwords, and grapheme units.   \n  \n* Character-Unit\n```\n아 모 몬 소리야 칠 십 퍼센트 확률이라니\n```\n  \n* Subword-Unit\n```\n▁아 ▁모 ▁ 몬 ▁소리 야 ▁ 칠 ▁ 십 ▁퍼 센트 ▁확 률 이라 니\n```\n\n* Grapheme-Unit\n```\nㅇㅏ ㅁㅗ ㅁㅗㄴ ㅅㅗㄹㅣㅇㅑ ㅊㅣㄹ ㅅㅣㅂ ㅍㅓㅅㅔㄴㅌㅡ ㅎㅘㄱㄹㅠㄹㅇㅣㄹㅏㄴㅣ\n```\n   \n## Conversion to Numeric id\n  \n* transcript\n```\n아 모 몬 소리야 칠 십 퍼센트 확률이라니\n```\n\n* conversion\n```\n7 3 106 3 730 3 173 32 26 3 319 3 120 3 490 552 157 3 315 747 5 33 22\n```\n   \n## Troubleshoots and Contributing\n  \nIf you have any questions, bug reports, and feature requests, please [open an issue](https://github.com/sooftware/KsponSpeech.preprocess/issues) on Github.   \nFor live discussions, please go to our [gitter](https://gitter.im/Korean-Speech-Recognition/community) or Contacts sh951011@gmail.com please.  \n  \nI appreciate any kind of feedback or contribution.  Feel free to proceed with small issues like bug fixes, documentation improvement.  For major contributions and new features, please discuss with the collaborators in corresponding issues.  \n  \n## Author\n* [Soohwan Kim](https://github.com/sooftware), [Seyoung Bae](https://github.com/triplet02),  [Cheolhwang Won](https://github.com/wch18735), [Soyoung Cho](https://github.com/SoYoungCho), [Jeongwon Kwak](https://github.com/jeongwonkwak)\n* Contacts: sh951011@gmail.com\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsooftware%2Fksponspeech","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsooftware%2Fksponspeech","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsooftware%2Fksponspeech/lists"}