https://github.com/zurawiki/tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
https://github.com/zurawiki/tiktoken-rs
bpe openai rust tokenizer
Last synced: 2 days ago
JSON representation
Ready-made tokenizer library for working with GPT and tiktoken
- Host: GitHub
- URL: https://github.com/zurawiki/tiktoken-rs
- Owner: zurawiki
- License: mit
- Created: 2023-02-02T17:20:46.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-10-25T21:55:29.000Z (6 months ago)
- Last Synced: 2024-10-29T15:49:16.935Z (6 months ago)
- Topics: bpe, openai, rust, tokenizer
- Language: Rust
- Homepage:
- Size: 3.68 MB
- Stars: 253
- Watchers: 5
- Forks: 47
- Open Issues: 19
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
- awesome-ChatGPT-repositories - tiktoken-rs - Ready-made tokenizer library for working with GPT and tiktoken (Openai)
- awesome-rust - zurawiki/tiktoken-rs - rs](https://crates.io/crates/tiktoken-rs)] - Library for tokenizing text with OpenAI models using tiktoken. [](https://github.com/zurawiki/tiktoken-rs/actions/workflows/ci.yml) (Libraries / Artificial Intelligence)
- fucking-awesome-rust - zurawiki/tiktoken-rs - rs](crates.io/crates/tiktoken-rs)] - Library for tokenizing text with OpenAI models using tiktoken. [](https://github.com/zurawiki/tiktoken-rs/actions/workflows/ci.yml) (Libraries / Artificial Intelligence)
- fucking-awesome-rust - zurawiki/tiktoken-rs - rs](crates.io/crates/tiktoken-rs)] - Library for tokenizing text with OpenAI models using tiktoken. [](https://github.com/zurawiki/tiktoken-rs/actions/workflows/ci.yml) (Libraries / Artificial Intelligence)
- awesome-rust - zurawiki/tiktoken-rs - rs](https://crates.io/crates/tiktoken-rs)] — Library for tokenizing text with OpenAI models using tiktoken. [](https://github.com/zurawiki/tiktoken-rs/actions/workflows/ci.yml) (Libraries / Artificial Intelligence)
README