https://github.com/sewenew/tokenizer

C++ implementation of tokenizers, including tiktoken.
https://github.com/sewenew/tokenizer

openai tiktoken tokenizer

Last synced: about 1 year ago
JSON representation

C++ implementation of tokenizers, including tiktoken.

• Host: GitHub
• URL: https://github.com/sewenew/tokenizer
• Owner: sewenew
• License: apache-2.0
• Created: 2023-08-20T09:36:49.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2023-12-07T14:24:08.000Z (over 2 years ago)
• Last Synced: 2025-03-28T06:24:05.383Z (over 1 year ago)
• Topics: openai, tiktoken, tokenizer
• Language: C++
• Homepage:
• Size: 761 KB
• Stars: 17
• Watchers: 5
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# tokenizer

C++ implementation of tokenizers, including tiktoken.