https://github.com/shivendrra/shredword

Fast & efficient BPE tokenizer written in C & python for LLM tranining

c c-tokenizer cpp open-source tiktoken tokenizer

Last synced: 09 Apr 2025