{"id":22469150,"url":"https://github.com/alula/tokenizers","last_synced_at":"2025-08-23T13:32:15.112Z","repository":{"id":111572835,"uuid":"231272185","full_name":"alula/tokenizers","owner":"alula","description":"A fast and easy to use implementation of today's most used tokenizers.","archived":false,"fork":false,"pushed_at":"2020-01-01T22:48:42.000Z","size":36,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-16T22:23:16.055Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alula.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-01T22:47:44.000Z","updated_at":"2020-01-01T22:48:44.000Z","dependencies_parsed_at":"2023-04-29T21:01:18.077Z","dependency_job_id":null,"html_url":"https://github.com/alula/tokenizers","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/alula/tokenizers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alula%2Ftokenizers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alula%2Ftokenizers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alula%2Ftokenizers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alula%2Ftokenizers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alula","download_url":"https://codeload.github.com/alula/tokenizers/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alula%2Ftokenizers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271749051,"owners_count":24814115,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-06T11:26:39.835Z","updated_at":"2025-08-23T13:32:15.088Z","avatar_url":"https://github.com/alula.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![PyPI version](https://badge.fury.io/py/tokenizers.svg)](https://badge.fury.io/py/tokenizers)\n\n# Tokenizers\n\nA fast and easy to use implementation of today's most used tokenizers.\n\n - High Level design: [master](https://github.com/huggingface/tokenizers)\n\nThis API is currently in the process of being stabilized. We might introduce breaking changes\nreally often in the coming days/weeks, so use at your own risks.\n\n### Installation\n\n#### With pip:\n\n```bash\npip install tokenizers\n```\n\n#### From sources:\n\nTo use this method, you need to have the Rust nightly toolchain installed.\n\n```bash\n# Install with:\ncurl https://sh.rustup.rs -sSf | sh -s -- -default-toolchain nightly-2019-11-01 -y\nexport PATH=\"$HOME/.cargo/bin:$PATH\"\n\n# Or select the right toolchain:\nrustup default nightly-2019-11-01\n```\n\nOnce Rust is installed and using the right toolchain you can do the following.\n\n```bash\ngit clone https://github.com/huggingface/tokenizers\ncd tokenizers/bindings/python\n\n# Create a virtual env (you can use yours as well)\npython -m venv .env\nsource .env/bin/activate\n\n# Install `tokenizers` in the current virtual env\npip install maturin\nmaturin develop --release\n```\n\n### Usage\n\n#### Use a pre-trained tokenizer\n\n```python\nfrom tokenizers import Tokenizer, models, pre_tokenizers, decoders\n\n# Load a BPE Model\nvocab = \"./path/to/vocab.json\"\nmerges = \"./path/to/merges.txt\"\nbpe = models.BPE.from_files(vocab, merges)\n\n# Initialize a tokenizer\ntokenizer = Tokenizer(bpe)\n\n# Customize pre-tokenization and decoding\ntokenizer.with_pre_tokenizer(pre_tokenizers.ByteLevel.new(add_prefix_space=True))\ntokenizer.with_decoder(decoders.ByteLevel.new())\n\n# And then encode:\nencoded = tokenizer.encode(\"I can feel the magic, can you?\")\nprint(encoded)\n\n# Or tokenize multiple sentences at once:\nencoded = tokenizer.encode_batch([\n\t\"I can feel the magic, can you?\",\n\t\"The quick brown fox jumps over the lazy dog\"\n])\nprint(encoded)\n```\n\n#### Train a new tokenizer\n\n```python\nfrom tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers\n\n# Initialize a tokenizer\ntokenizer = Tokenizer(models.BPE.empty())\n\n# Customize pre-tokenization and decoding\ntokenizer.with_pre_tokenizer(pre_tokenizers.ByteLevel.new(add_prefix_space=True))\ntokenizer.with_decoder(decoders.ByteLevel.new())\n\n# And then train\ntrainer = trainers.BpeTrainer.new(vocab_size=20000, min_frequency=2)\ntokenizer.train(trainer, [\n\t\"./path/to/dataset/1.txt\",\n\t\"./path/to/dataset/2.txt\",\n\t\"./path/to/dataset/3.txt\"\n])\n\n# Now we can encode\nencoded = tokenizer.encode(\"I can feel the magic, can you?\")\nprint(encoded)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falula%2Ftokenizers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falula%2Ftokenizers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falula%2Ftokenizers/lists"}