https://github.com/bzp2010/lua-tiktoken
OpenAI tiktoken for Lua
https://github.com/bzp2010/lua-tiktoken
lua openai rust tokenizer
Last synced: about 2 months ago
JSON representation
OpenAI tiktoken for Lua
- Host: GitHub
- URL: https://github.com/bzp2010/lua-tiktoken
- Owner: bzp2010
- License: apache-2.0
- Created: 2023-12-21T10:54:42.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-08T16:12:49.000Z (about 2 years ago)
- Last Synced: 2026-05-04T22:43:24.256Z (about 2 months ago)
- Topics: lua, openai, rust, tokenizer
- Language: Rust
- Homepage:
- Size: 6.84 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# OpenAI tiktoken for Lua
The `tiktoken-rs` is a rust library for tokenizing text with OpenAI models using tiktoken.
This project provides `tiktoken-rs` bindings on the Lua, which enables out-of-the-box text tokenization for AI-related projects such as Apache APISIX.
## Installation
The preferred way to install this library is to use Luarocks:
```shell
luarocks install https://github.com/bzp2010/lua-tiktoken/raw/main/rockspec/lua-tiktoken-main-0.rockspec
```
## Synopsis
```lua
local tiktoken = require("tiktoken")
local models = tiktoken.models() -- {"r50k_base", "p50k_base", "p50k_edit", "cl100k_base"}
local count = tiktoken.count("cl100k_base", "Here is a message")
```
## Modules
### tiktoken
To load this module:
```lua
local tiktoken = require("tiktoken")
```
#### models
**syntax:** *models = tiktoken.models()*
It returns a string table containing all the tokenizer models.
#### count
**syntax:** *count = tiktoken.count(model_name, text)*
`model_name` is the tokenizer model to be used, and it should be one of the tables returned by `models()`.
`message` is the text to be tokenized.
It returns the number of tokens.