{"id":20779439,"url":"https://github.com/freedomintelligence/x-tokenization","last_synced_at":"2026-03-15T14:39:47.461Z","repository":{"id":204427158,"uuid":"711824112","full_name":"FreedomIntelligence/X-tokenization","owner":"FreedomIntelligence","description":null,"archived":false,"fork":false,"pushed_at":"2023-11-03T07:56:17.000Z","size":2,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-01-18T08:38:32.227Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-30T08:48:19.000Z","updated_at":"2023-10-30T08:48:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"ad739534-8299-4bcf-babe-6dfef31967ed","html_url":"https://github.com/FreedomIntelligence/X-tokenization","commit_stats":null,"previous_names":["freedomintelligence/x-tokenization"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FX-tokenization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FX-tokenization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FX-tokenization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FX-tokenization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/X-tokenization/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243119862,"owners_count":20239379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T13:27:56.041Z","updated_at":"2025-12-16T12:59:28.850Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Testing tokenization in multilingual context\n\n\nThis is also inspired by [tokenmonster](https://github.com/alasdairforsythe/tokenmonster).\n\n## tokenizer 速度测试\n| 模型 | Arabic (203673) |  | English (403630) |  | Chinese (121106) |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  | token | time(s) | token | time(s) | token | time(s) |\n| Bloom (250k) | 50825 | 0.2077 | 89558 | 0.3469 | 77893 | 0.2185 |\n| Llama (32k) | 183904 | 0.4261 | 106614 | 0.6880 | 169253 | 0.3682 |\n| Baichuan2 (125k) | 172505 | 0.4330 | 99267 | 0.7666 | 80966 | 0.3087 |\n| mt5 | 73172 | 4.2103 | 104994 | 9.0237 | 90329 | 0.4704 |\n\n假设一个词表的大小，1k,2k,4k,8k, .. 训bpe，在给定的validation下算压缩率（暂时没找到合适的validation，以train dataset代替），饱和 (提升的比例随着词表大小不显著), 数据集wikipedia\n最后一个token的frequency\n###### 中文词表\n| num |  compression ratio | last word frequency |\n| --- | --- | --- |\n| 8000 | 0.7813 | 48 |\n| 16000 | 0.5964 | 2 |\n| 24000 | 0.5485 | 2 |\n| 32000 | 0.5212 | 1 |\n\n根据训练预料需要7000+token能够达到99.95%的覆盖率，故从8k开始。\n\n###### 阿拉伯语词表\n| num |  compression ratio | last word frequency |\n| --- | --- | --- |\n| 1000 | 0.4172 | 553 |\n| 2000 | 0.3603 | 553 |\n| 4000 | 0.3130 | 10 |\n| 8000 | 0.2752 | 10 |\n| 16000 | 0.2466 | 7 |\n| 24000 | 0.2338 | 6 |\n| 32000 | 0.2262 | 1 |\n\n\n## reference\n```\n@misc{X-tokenization-2023,\n  title={X-tokenization, towards a universal across languages.},\n  author={Jianqing Zhu and Benyou Wang},\n  year = {2023},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/FreedomIntelligence/X-tokenization}},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fx-tokenization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Fx-tokenization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fx-tokenization/lists"}