{"id":87494,"url":"https://github.com/nlpoptimize/awesome-tokenizers","name":"awesome-tokenizers","description":"A curated list of tokenizer libraries for blazing-fast NLP processing.","projects_count":33,"last_synced_at":"2026-06-26T13:00:33.604Z","repository":{"id":286958037,"uuid":"959512163","full_name":"NLPOptimize/awesome-tokenizers","owner":"NLPOptimize","description":"A curated list of tokenizer libraries for blazing-fast NLP processing.","archived":false,"fork":false,"pushed_at":"2025-04-10T09:55:33.000Z","size":19,"stargazers_count":12,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-09T14:03:36.828Z","etag":null,"topics":["awesome","collections","python","python-library","pythonframework","sentence","tokenizer","wordpiece"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NLPOptimize.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-02T22:53:07.000Z","updated_at":"2026-04-11T05:57:00.000Z","dependencies_parsed_at":"2025-04-23T06:01:21.341Z","dependency_job_id":"b0160dfc-ffa8-4458-99a7-a8d5d748d7a7","html_url":"https://github.com/NLPOptimize/awesome-tokenizers","commit_stats":null,"previous_names":["nlpoptimize/awesome-tokenizers"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/NLPOptimize/awesome-tokenizers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NLPOptimize%2Fawesome-tokenizers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NLPOptimize%2Fawesome-tokenizers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NLPOptimize%2Fawesome-tokenizers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NLPOptimize%2Fawesome-tokenizers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NLPOptimize","download_url":"https://codeload.github.com/NLPOptimize/awesome-tokenizers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NLPOptimize%2Fawesome-tokenizers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34817641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-26T02:00:06.560Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2025-04-10T14:11:09.763Z","updated_at":"2026-06-26T13:00:33.605Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["🔹 **WordPiece Tokenizer Implementations**","🔹 **SentencePiece Implementations**","🔹 **BPE (Byte Pair Encoding) Implementations**"],"sub_categories":[],"readme":"\u003chr\u003e\n\n\u003ca href=\"https://github.com/NLPOptimize/awesome-tokenizers\"\u003e\n  \u003cimg src=\"https://github.com/user-attachments/assets/fda66cb0-e785-4e52-bcaf-4deed208a278\" align=\"center\" alt=\"awesome_tokenizers\" title=\"https://github.com/NLPOptimize/awesome-tokenizers\"\u003e\n\u003c/a\u003e\n\n\u003chr\u003e\n\n\n\n\n\n# Awesome-tokenizer  [![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome)\n\nA repository with the 🔥 symbol is a tokenizer that is significantly faster than other tokenizers.\n\n\n## 🔹 **WordPiece Tokenizer Implementations**\n\n* 🔥 **[FlashTokenizer](https://github.com/NLPOptimize/flash-tokenizer)** (C++/Python)\n  *  The world's fastest CPU tokenizer library!\n- **[huggingface/tokenizers](https://github.com/huggingface/tokenizers)** *(Rust/Python)*\n  - Official Hugging Face tokenizer, fast Rust implementation with Python bindings.\n- 🔥 **[FastBertTokenizer](https://github.com/kekyo/FastBertTokenizer)** *(C#)*\n  - Highly optimized tokenizer for speed, reduced accuracy on non-English inputs.\n- **[BertTokenizers](https://github.com/microsoft/BertTokenizers)** *(C#)*\n  - Microsoft's original C# tokenizer implementation (slower than FastBertTokenizer).\n- 🔥 **[rust-tokenizers](https://github.com/guillaume-be/rust-tokenizers)** *(Rust/Python)*\n  - Rust tokenizer library; faster than pure Python but slower than BlingFire or Flash.\n- **[tokenizers-cpp](https://github.com/monologg/tokenizers-cpp)** *(C++)*\n  - Wrapper around SentencePiece and Hugging Face’s tokenizers; not a standalone implementation.\n- **[bertTokenizer (Java)](https://github.com/robrua/easy-bert)** *(Java)*\n  - Java-based Bert tokenizer implementation.\n- **[ZhuoruLin/fast-wordpiece](https://github.com/ZhuoruLin/fast-wordpiece)** *(Rust)*\n  - Rust implementation using LinMaxMatching; likely comparable or slower than optimized C++ versions.\n- **[huggingface_tokenizer_cpp](https://github.com/BlinkDL/huggingface_tokenizer_cpp)** *(C++)*\n  - Naive pure C++ implementation; slow performance.\n- **[SeanLee97/BertWordPieceTokenizer.jl](https://github.com/SeanLee97/BertWordPieceTokenizer.jl)** *(Julia)*\n  - Julia implementation, not widely benchmarked.\n- 🔥 **[BlingFire](https://github.com/microsoft/BlingFire)** *(C++/Python)*\n  - Microsoft's high-speed tokenizer optimized for batch processing, available as Python bindings.\n- **[tensorflow-text WordpieceTokenizer](https://github.com/tensorflow/text)** *(C++/Python)*\n  - TensorFlow-integrated Google's tokenizer optimized for use in TensorFlow pipelines.\n- **[transformers BertTokenizer](https://github.com/huggingface/transformers)** *(Python)*\n  - Hugging Face's Python implementation; easy to use but slower due to pure Python nature.\n- **[Deep Java Library (DJL) BertTokenizer](https://github.com/deepjavalibrary/djl)** *(Java)*\n  - Amazon’s Java implementation, integrated within DJL framework.\n- **[tokenizers.net](https://github.com/ScottLogic/tokenizers.net)** *(C#)*\n  - .NET/C# binding of Hugging Face tokenizers optimized for .NET runtimes.\n- **[Tokenizers.jl](https://github.com/JuliaText/Tokenizers.jl)** *(Julia)*\n  - Julia tokenizer library inspired by Hugging Face implementations.\n- **[fast-bert-tokenizer-py](https://github.com/kakaobrain/fast-bert-tokenizer-py)** *(Python/Cython)*\n  - Python tokenizer accelerated with Cython.\n- **[ml-commons/tokenizer](https://github.com/mlcommons/tokenizer)** *(C++)*\n  - High-performance C++ tokenizer supporting WordPiece and other algorithms.\n\n------\n\n## 🔹 **BPE (Byte Pair Encoding) Implementations**\n\n- **[OpenAI TikToken](https://github.com/openai/tiktoken)** *(Rust/Python)*\n  - Official BPE tokenizer from OpenAI (used in GPT models), highly optimized.\n- **[huggingface/tokenizers](https://github.com/huggingface/tokenizers)** *(Rust/Python)*\n  - General-purpose tokenizer supporting BPE, from Hugging Face.\n- **[bpe-tokenizer (Rust)](https://docs.rs/bpe-tokenizer/latest/bpe_tokenizer/)** *(Rust)*\n  - Rust BPE tokenizer library, identifying frequent pairs effectively.\n- **[YouTokenToMe](https://github.com/VKCOM/YouTokenToMe)** *(C++/Python)*\n  - Efficient BPE tokenizer with fast training and inference, developed by VK.com.\n- 🔥  /**[fastBPE](https://github.com/glample/fastBPE)** *(C++/Python)*\n  - Facebook’s fast and memory-efficient BPE tokenizer, widely used in NLP research.\n- 🔥 **[sentencepiece](https://github.com/google/sentencepiece)** *(C++/Python)*\n  - Google's SentencePiece implementation also provides BPE as one of the algorithms.\n- **[Subword-nmt](https://github.com/rsennrich/subword-nmt)** *(Python)*\n  - Python implementation commonly used in MT research, simple but slower.\n- 🔥 **[rs-bpe](https://github.com/gweidart/rs-bpe)** (Rust)\n  - A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust\n------\n\n## 🔹 **SentencePiece Implementations**\n\n- **[google/sentencepiece](https://github.com/google/sentencepiece)** *(C++/Python)*\n  - Google's official, language-independent, neural-based subword tokenizer.\n- **[sentencepiece-rs](https://github.com/finalfusion/sentencepiece)** *(Rust)*\n  - Rust binding for Google's SentencePiece.\n- **[huggingface/tokenizers](https://github.com/huggingface/tokenizers)** *(Rust/Python)*\n  - Hugging Face tokenizer library supporting SentencePiece.\n- **[TensorFlow Text SentencepieceTokenizer](https://github.com/tensorflow/text)** *(C++/Python)*\n  - Google's TensorFlow Text includes SentencePiece tokenizer optimized for TF environments.\n- **[sentencepiece.NET](https://github.com/Curiosity-ai/sentencepiece.NET)** *(C#)*\n  - .NET binding for SentencePiece tokenizer.\n- **[sentencepiece-jni](https://github.com/go-skynet/sentencepiece-jni)** *(Java)*\n  - JNI bindings for Google's SentencePiece tokenizer for Java applications.\n- **[sentencepiece-swift](https://github.com/xenova/sentencepiece-swift)** *(Swift)*\n  - Swift bindings for Google's SentencePiece tokenizer.\n\n------\n\n## Contributing\n\nYour contributions are always welcome! Please take a look at the [contribution guidelines](./CONTRIBUTING.md) first.\n\n\n## Question\n\nAlso, if you have any questions, please send a message directly to WeChat, Line, or Telegram below.\n\n\n\u003cdetails\u003e\n\u003csummary\u003e💬 LINE\u003c/summary\u003e\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/user-attachments/assets/f8bf4afb-7719-4ceb-8baa-8aeb197514ca\" width=\"512\" height=\"512\"\u003e\n\u003c/p\u003e\n\n\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e💬 Telegram\u003c/summary\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/user-attachments/assets/9516c0a3-6268-446e-a526-f0afa11c9a0e\" width=\"512\" height=\"512\"\u003e\n\u003c/p\u003e\n\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e💬 WeChat\u003c/summary\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/user-attachments/assets/f8e3a896-1820-47ab-b4f4-9ec0b8741e7d\" width=\"512\" height=\"512\"\u003e\n\u003c/p\u003e\n\n\n\u003c/details\u003e\n\n\n\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/nlpoptimize%2Fawesome-tokenizers/projects"}