https://github.com/ceifa/tiktoken-node

OpenAI's tiktoken but with node bindings
https://github.com/ceifa/tiktoken-node

Last synced: 18 days ago
JSON representation

OpenAI's tiktoken but with node bindings

• Host: GitHub
• URL: https://github.com/ceifa/tiktoken-node
• Owner: ceifa
• Created: 2023-03-16T23:16:59.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-05-18T11:58:03.000Z (6 months ago)
• Last Synced: 2024-10-07T06:06:30.463Z (about 1 month ago)
• Language: Rust
• Homepage:
• Size: 2.03 MB
• Stars: 104
• Watchers: 2
• Forks: 10
• Open Issues: 6
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# ⏳ tiktoken-node

tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with

OpenAI's models.

```js

const tiktoken = require('tiktoken-node')

let enc = tiktoken.getEncoding("gpt2")

console.assert(enc.decode(enc.encode("hello world")) == "hello world")

// To get the tokeniser corresponding to a specific model in the OpenAI API:

enc = tiktoken.encodingForModel("text-davinci-003")

```

The open source version of `tiktoken-node` can be installed from npm:

```

npm install tiktoken-node

```