Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/kojix2/tiktoken-cr
Tiktoken for Crystalists
https://github.com/kojix2/tiktoken-cr
crystal tiktoken
Last synced: 13 days ago
JSON representation
Tiktoken for Crystalists
- Host: GitHub
- URL: https://github.com/kojix2/tiktoken-cr
- Owner: kojix2
- License: mit
- Created: 2023-08-04T23:44:30.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-08-11T14:50:20.000Z (3 months ago)
- Last Synced: 2024-10-04T21:32:05.267Z (about 1 month ago)
- Topics: crystal, tiktoken
- Language: Crystal
- Homepage: https://kojix2.github.io/tiktoken-cr/
- Size: 136 KB
- Stars: 4
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE.txt
Awesome Lists containing this project
README
# tiktoken-cr
[](https://github.com/kojix2/tiktoken-cr/actions/workflows/test.yml)
Tiktoken for Crystalists.
- [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs)
- [tiktoken-c](https://github.com/kojix2/tiktoken-c)
## Usage
```crystal
require "../src/tiktoken"
text = "吾輩は猫である。名前はたぬき。"
encoding = Tiktoken.encoding_for_model("gpt-4")
tokens = encoding.encode_with_special_tokens(text)
# [7305, 122, 164, 120, 102, 15682, 163, 234, 104, 16556, 30591, 30369, 1811, 13372, 25580, 15682, 28713, 2243, 105, 50834, 1811]
encoding.decode(tokens)
# "吾輩は猫である。名前はたぬき。"
```
Count the number of ChatGPT tokens
```crystal
model = "gpt-4"
messages = [
{
"role" => "system",
"content" => "You are a helpful assistant that only speaks French.",
},
{
"role" => "user",
"content" => "Hello, how are you?",
},
{
"role" => "assistant",
"content" => "Parlez-vous francais?",
},
]
Tiktoken.num_tokens_from_messages(model, messages)
# 36
Tiktoken.chat_completion_max_tokens(model, messages)
# 8156
```
## Documentation
[API Documentation](https://kojix2.github.io/tiktoken-cr/)
## Contributing
- Report bugs
- Fix bugs and submit pull requests
- Write, clarify, or fix documentation
- Suggest or add new features
- Make a donation
## Acknowledgement
This project is inspired by [tiktoken_ruby](https://github.com/IAPark/tiktoken_ruby).
## LICENSE
MIT