https://github.com/NLPOptimize/flash-tokenizer

EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING
https://github.com/NLPOptimize/flash-tokenizer

bert berttokenizer cpp cpp17 deep-learning flash huggingface nlp pybind11 python tokenizer trie wordpiece wordpiece-tokenization

Last synced: 6 months ago
JSON representation

EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING

• Host: GitHub
• URL: https://github.com/NLPOptimize/flash-tokenizer
• Owner: NLPOptimize
• Created: 2025-03-09T00:26:27.000Z (7 months ago)
• Default Branch: main
• Last Pushed: 2025-04-09T09:26:39.000Z (6 months ago)
• Last Synced: 2025-04-09T10:39:14.707Z (6 months ago)
• Topics: bert, berttokenizer, cpp, cpp17, deep-learning, flash, huggingface, nlp, pybind11, python, tokenizer, trie, wordpiece, wordpiece-tokenization
• Language: C++
• Homepage: https://github.com/NLPOptimize/flash-tokenizer
• Size: 195 MB
• Stars: 287
• Watchers: 2
• Forks: 3
• Open Issues: 1
• Metadata Files:
  • Readme: README.ja-JP.md

Awesome Lists containing this project

• awesome-tokenizers - FlashTokenizer