{"id":18538774,"url":"https://github.com/crownpku/somiao-pinyin","last_synced_at":"2025-04-10T03:56:55.707Z","repository":{"id":55055858,"uuid":"101725233","full_name":"crownpku/Somiao-Pinyin","owner":"crownpku","description":"Somiao Pinyin: Train your own Chinese Input Method with Seq2seq Model 搜喵拼音输入法","archived":false,"fork":false,"pushed_at":"2020-03-29T08:44:00.000Z","size":146,"stargazers_count":266,"open_issues_count":10,"forks_count":78,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-04-10T03:56:51.927Z","etag":null,"topics":["chinese","input-method","pinyin","seq2seq-model"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/crownpku.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"crownpku","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2017-08-29T06:08:38.000Z","updated_at":"2025-01-21T03:59:15.000Z","dependencies_parsed_at":"2022-08-14T10:20:34.238Z","dependency_job_id":null,"html_url":"https://github.com/crownpku/Somiao-Pinyin","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crownpku%2FSomiao-Pinyin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crownpku%2FSomiao-Pinyin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crownpku%2FSomiao-Pinyin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crownpku%2FSomiao-Pinyin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/crownpku","download_url":"https://codeload.github.com/crownpku/Somiao-Pinyin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248154998,"owners_count":21056542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chinese","input-method","pinyin","seq2seq-model"],"created_at":"2024-11-06T19:44:57.991Z","updated_at":"2025-04-10T03:56:55.685Z","avatar_url":"https://github.com/crownpku.png","language":"Python","funding_links":["https://github.com/sponsors/crownpku"],"categories":[],"sub_categories":[],"readme":"\n\n# Somiao Pinyin: Train your own Chinese Input Method with Seq2seq Model\n\n### [中文Blog](http://www.crownpku.com/2017/09/10/%E6%90%9C%E5%96%B5%E8%BE%93%E5%85%A5%E6%B3%95-%E7%94%A8seq2seq%E8%AE%AD%E7%BB%83%E8%87%AA%E5%B7%B1%E7%9A%84%E6%8B%BC%E9%9F%B3%E8%BE%93%E5%85%A5%E6%B3%95.html)\n\nPersonalized Chinese Pinyin Input Method with Seq2seq model\n\nOriginal code in https://github.com/Kyubyong/neural_chinese_transliterator for research purpose. \n\nThis repository intends to experiment with different training data and interactive user inputs, and possibly develop towards a real data-personalized and model-localized Pinyin Input product.\n\n\n![](http://www.crownpku.com/images/201709/1.jpg)\n\n\n## Requrements\n\n* Python (\u003e=3.5)\n\n* TensorFlow (\u003e=r1.2)\n\n* xpinyin (for Chinese pinyin annotation)\n\n* distance (for calculating the similarity score between two strings)\n\n* tqdm\n\n\n## Usage\n\n\n### Training:\n\n* STEP 1. Download [Leipzig Chinese Corpus](http://wortschatz.uni-leipzig.de/en/download/) \n  \n  Extract it and copy zho_news_2007-2009_1M-sentences.txt to data/ folder.\n  \n  Or use your own Chinese Corpus with the same format.\n\n* STEP 2. Build a Pinyin-Chinese parallel corpus.\n\n```\n#python3 build_corpus.py\n```\n\n* STEP 3. Run `prepro.py` to make vocabulary and training data.\n\n```\n#python3 prepro.py\n```\n\n* STEP 4. Adjust hyperparameters in `hyperparams.py` if necessary.\n\n\n* STEP 5. Train the model\n\n```\n#python3 train.py\n```\n\n\n### Inference with command line input:\n\nFor command line input testing, run:\n\n```\npython3 eval.py\n```\n\nYou may change the main function name to use the original testing data evaluation.\n\n\n### Testing with pre-trained models:\n\nDownload the pre-trained model from [blog](http://www.crownpku.com/2017/09/10/%E6%90%9C%E5%96%B5%E8%BE%93%E5%85%A5%E6%B3%95-%E7%94%A8seq2seq%E8%AE%AD%E7%BB%83%E8%87%AA%E5%B7%B1%E7%9A%84%E6%8B%BC%E9%9F%B3%E8%BE%93%E5%85%A5%E6%B3%95.html), unzip it to generate /log and /data.\n\nRemember to overwrite the pickle files in /data with the pre-trained model data.\n\nThen run for command line input testing:\n\n```\npython3 eval.py\n```\n\n\n## Sample Results\n\nModel is trained from Chinese News in 2007-2009. So many now common Chinese sayings are not learned.\n\n```\n请输入测试拼音：nihao\n你好\n\n请输入测试拼音：chenggongle\n成功了\n\n请输入测试拼音：wolegequ\n我了个曲\n\n请输入测试拼音：taibangla\n太棒啦\n\n请输入测试拼音：dacolehuizenmeyang\n打破了会怎么样\n\n请输入测试拼音：pujinghehujintaotongdianhua\n普京和胡锦涛通电话\n\n请输入测试拼音：xiangbuqilaishinianqianfashengleshenme\n想不起来十年前发生了什么\n\n请输入测试拼音：meiguohongzhawomenzainansilafudedashiguan\n美国轰炸我们在南斯拉夫的大事馆\n\n请输入测试拼音：liudehuanageshihouhaonianqing\n刘德华那个时候好年轻\n\n请输入测试拼音：shishihouxunlianyixiabilibilideyuliaole\n是时候训练一下比例比例的预料了\n\n```\n\n\n## TODOLIST\n\n* Pretrained models on different contexts\n\n* Model selection for using different models while input different things (chatting? writing scientific papers? etc...)\n\n* Function to record LOCALLY what user has input as personalized corpus\n\n* User Interface\n\n* ...","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrownpku%2Fsomiao-pinyin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcrownpku%2Fsomiao-pinyin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrownpku%2Fsomiao-pinyin/lists"}