https://github.com/zurawiki/tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken
https://github.com/zurawiki/tiktoken-rs

bpe openai rust tokenizer

Last synced: 3 months ago
JSON representation

Ready-made tokenizer library for working with GPT and tiktoken

• Host: GitHub
• URL: https://github.com/zurawiki/tiktoken-rs
• Owner: zurawiki
• License: mit
• Created: 2023-02-02T17:20:46.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2026-04-08T06:22:53.000Z (3 months ago)
• Last Synced: 2026-04-08T08:00:43.152Z (3 months ago)
• Topics: bpe, openai, rust, tokenizer
• Language: Rust
• Homepage:
• Size: 3.73 MB
• Stars: 381
• Watchers: 4
• Forks: 68
• Open Issues: 5
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md
  • Security: SECURITY.md

Awesome Lists containing this project

• awesome-ChatGPT-repositories - tiktoken-rs - Ready-made tokenizer library for working with GPT and tiktoken (Openai)
• awesome-rust - zurawiki/tiktoken-rs - rs](https://crates.io/crates/tiktoken-rs)] - Library for tokenizing text with OpenAI models using tiktoken. [](https://github.com/zurawiki/tiktoken-rs/actions/workflows/ci.yml) (Libraries / Artificial Intelligence)
• awesome-rust - zurawiki/tiktoken-rs - rs](https://crates.io/crates/tiktoken-rs)] — Library for tokenizing text with OpenAI models using tiktoken. [](https://github.com/zurawiki/tiktoken-rs/actions/workflows/ci.yml) (Libraries / Artificial Intelligence)
• awesome-rust - zurawiki/tiktoken-rs - rs](https://crates.io/crates/tiktoken-rs)] — Library for tokenizing text with OpenAI models using tiktoken. [](https://github.com/zurawiki/tiktoken-rs/actions/workflows/ci.yml) (Libraries / Artificial Intelligence)
• fucking-awesome-rust - zurawiki/tiktoken-rs - rs](crates.io/crates/tiktoken-rs)] - Library for tokenizing text with OpenAI models using tiktoken. [](https://github.com/zurawiki/tiktoken-rs/actions/workflows/ci.yml) (Libraries / Artificial Intelligence)
• awesome-rust-with-stars - zurawiki/tiktoken-rs - rs) | Library for tokenizing text with OpenAI models using tiktoken. | 2025-11-09 | (Libraries / Artificial Intelligence)
• awesome-rust-ai-libraries - tiktoken-rs - Library for tokenizing text with OpenAI models using tiktoken. Essential for LLM input processing in Rust. ([Read more](/details/tiktoken-rs.md)) `Tokenization` `Openai` `Open Source` (NLP)
• awesome-llm-cost - tiktoken-rs - Rust port of tiktoken. (Calculators and Estimators / Tokenizers)

README