Projects in Awesome Lists tagged with byte-pair-tokenizer
A curated list of projects in awesome lists tagged with byte-pair-tokenizer .
https://github.com/gweidart/rs-bpe
A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust
bpe bpe-tokenizer byte-pair-encoding byte-pair-tokenizer huggingface llm openai pypi-package python rust tiktoken tokenizers
Last synced: 28 Apr 2025
https://github.com/sameermanan/rs-bpe
A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust
bpe bpe-tokenizer byte-pair-encoding byte-pair-tokenizer huggingface llm openai pypi-package python rust tiktoken tokenizers
Last synced: 22 Mar 2025
