{"id":18264551,"url":"https://github.com/modeltc/greedy-tokenizer","last_synced_at":"2026-07-15T14:32:04.869Z","repository":{"id":208568398,"uuid":"721950088","full_name":"ModelTC/greedy-tokenizer","owner":"ModelTC","description":"Greedily tokenize strings with the longest tokens iteratively.","archived":false,"fork":false,"pushed_at":"2025-09-15T00:06:56.000Z","size":48,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-10-25T18:43:21.429Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ModelTC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-11-22T05:45:56.000Z","updated_at":"2025-03-22T15:13:45.000Z","dependencies_parsed_at":"2024-11-05T11:16:31.303Z","dependency_job_id":"5350ce94-2969-44e5-aa39-c5d3cec85914","html_url":"https://github.com/ModelTC/greedy-tokenizer","commit_stats":null,"previous_names":["modeltc/greedy-tokenizer"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/ModelTC/greedy-tokenizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2Fgreedy-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2Fgreedy-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2Fgreedy-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2Fgreedy-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ModelTC","download_url":"https://codeload.github.com/ModelTC/greedy-tokenizer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2Fgreedy-tokenizer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35509494,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-15T02:00:06.706Z","response_time":131,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T11:15:04.982Z","updated_at":"2026-07-15T14:32:04.845Z","avatar_url":"https://github.com/ModelTC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Greedy Tokenizer\n\n[![Source file](https://img.shields.io/badge/source_file-greedy__tokenizer.py-green)](./greedy_tokenizer.py)\n[![License](https://img.shields.io/badge/license-MIT%2FApache--2.0-informational.svg)](#license)\n[![Build status](https://github.com/ModelTC/greedy-tokenizer/actions/workflows/ci.yml/badge.svg)](https://github.com/ModelTC/greedy-tokenizer/actions)\n\nGreedily tokenize strings with the longest tokens iteratively,\ncompatible with `transformers.PretrainedTokenizer` and `transformers.AutoTokenizer`.\n\n## Requirements\n\n- [transformers](https://github.com/huggingface/transformers)\n- [general-sam](https://github.com/ModelTC/general-sam-py)\n\n## Installation\n\n```sh\ngit clone https://github.com/ModelTC/greedy-tokenizer.git\ncd greedy-tokenizer\npip install -e .\n```\n\nOr use the [source file](./greedy_tokenizer.py) directly.\n\n## Usage\n\n```python\nfrom greedy_tokenizer import GreedyTokenizer\nfrom transformers import AutoTokenizer\n\n# Construct GreedyTokenizer with other PretrainedTokenizer\ntokenizer = GreedyTokenizer.from_other_pretrained(\n    \"internlm/internlm2-chat-7b\",\n    trust_remote_code=True,\n    revision=\"main\",\n    use_fast=False,\n)\n# Or, you can use:\n# old_tokenizer = AutoTokenizer.from_pretrained(...)\n# tokenizer = GreedyTokenizer.mock_tokenizer(old_tokenizer)\n\nseq = \"Hello! 你好呀！🌠\"\ntokens = tokenizer.tokenize(seq)\n\nprint(tokens)\n# ['Hello', '!', ' ', '你好', '呀', '！', '\u003c0xF0\u003e', '\u003c0x9F\u003e', '\u003c0x8C\u003e', '\u003c0xA0\u003e']\n\nassert tokenizer.convert_tokens_to_string(tokens) == seq\n\n# GreedyTokenizer can also be saved and loaded\ntokenizer.save_pretrained(\"/tmp/internlm2-chat-gt\")\ntokenizer = AutoTokenizer.from_pretrained(\n    \"/tmp/internlm2-chat-gt\",\n    trust_remote_code=True,\n    use_fast=False,\n)\n\n# No subwords required!\ngt = GreedyTokenizer(vocab=[f'\u003c0x{i:02x}\u003e' for i in range(256)] + ['你好呀'])\nprint(gt.tokenize('你好你好呀'))\n# ['\u003c0xe4\u003e', '\u003c0xbd\u003e', '\u003c0xa0\u003e', '\u003c0xe5\u003e', '\u003c0xa5\u003e', '\u003c0xbd\u003e', '你好呀']\n```\n\n## Tests\n\n```sh\npip install -e \".[test]\"\npytest -s\n# You can set some environment variables\n# DATASET=happylkx/InstructCoder COLUMN=input pytest -s\n```\n\n## License\n\n- \u0026copy; 2023 Chielo Newctle \\\u003c[ChieloNewctle@gmail.com](mailto:ChieloNewctle@gmail.com)\\\u003e\n- \u0026copy; 2023 ModelTC Team\n\nThis project is licensed under either of\n\n- [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0) ([`LICENSE-APACHE`](LICENSE-APACHE))\n- [MIT license](https://opensource.org/licenses/MIT) ([`LICENSE-MIT`](LICENSE-MIT))\n\nat your option.\n\nThe [SPDX](https://spdx.dev) license identifier for this project is `MIT OR Apache-2.0`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodeltc%2Fgreedy-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmodeltc%2Fgreedy-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodeltc%2Fgreedy-tokenizer/lists"}