https://github.com/sameermanan/rs-bpe
A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust
https://github.com/sameermanan/rs-bpe
bpe bpe-tokenizer byte-pair-encoding byte-pair-tokenizer huggingface llm openai pypi-package python rust tiktoken tokenizers
Last synced: 2 months ago
JSON representation
A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust
- Host: GitHub
- URL: https://github.com/sameermanan/rs-bpe
- Owner: SameerManan
- License: mit
- Created: 2025-03-21T21:24:14.000Z (2 months ago)
- Default Branch: main
- Last Pushed: 2025-03-21T22:18:55.000Z (2 months ago)
- Last Synced: 2025-03-21T22:30:08.086Z (2 months ago)
- Topics: bpe, bpe-tokenizer, byte-pair-encoding, byte-pair-tokenizer, huggingface, llm, openai, pypi-package, python, rust, tiktoken, tokenizers
- Language: Python
- Size: 2.44 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0