{"id":13567006,"url":"https://github.com/niieani/gpt-tokenizer","last_synced_at":"2025-05-14T01:03:32.208Z","repository":{"id":152234432,"uuid":"617290848","full_name":"niieani/gpt-tokenizer","owner":"niieani","description":"The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.","archived":false,"fork":false,"pushed_at":"2025-03-05T02:40:27.000Z","size":13836,"stargazers_count":568,"open_issues_count":7,"forks_count":40,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-05-08T18:53:24.804Z","etag":null,"topics":["bpe","decoder","encoder","gpt-2","gpt-3","gpt-4","gpt-4o","gpt-o1","machine-learning","openai","tokenizer"],"latest_commit_sha":null,"homepage":"https://gpt-tokenizer.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"latitudegames/GPT-3-Encoder","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/niieani.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"niieani"}},"created_at":"2023-03-22T04:32:09.000Z","updated_at":"2025-05-08T05:07:34.000Z","dependencies_parsed_at":"2023-11-15T06:29:54.003Z","dependency_job_id":"089e9147-030a-4606-8d62-3d470be72f81","html_url":"https://github.com/niieani/gpt-tokenizer","commit_stats":{"total_commits":99,"total_committers":12,"mean_commits":8.25,"dds":0.4747474747474747,"last_synced_commit":"3547826b37e829009a40d421a3733a54d13cd452"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niieani%2Fgpt-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niieani%2Fgpt-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niieani%2Fgpt-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niieani%2Fgpt-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/niieani","download_url":"https://codeload.github.com/niieani/gpt-tokenizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253967998,"owners_count":21992252,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bpe","decoder","encoder","gpt-2","gpt-3","gpt-4","gpt-4o","gpt-o1","machine-learning","openai","tokenizer"],"created_at":"2024-08-01T13:02:21.487Z","updated_at":"2025-05-14T01:03:32.170Z","avatar_url":"https://github.com/niieani.png","language":"TypeScript","funding_links":["https://github.com/sponsors/niieani"],"categories":["Openai","machine-learning","TypeScript","openai","Calculators and Estimators"],"sub_categories":["Tokenizers"],"readme":"# gpt-tokenizer\n\n[![Play with gpt-tokenizer](https://codesandbox.io/static/img/play-codesandbox.svg)](https://codesandbox.io/s/gpt-tokenizer-tjcjoz?fontsize=14\u0026hidenavigation=1\u0026theme=dark)\n\n`gpt-tokenizer` is a Token Byte Pair Encoder/Decoder supporting all OpenAI's models (including GPT-3.5, GPT-4, GPT-4o, and o1).\nIt's the [_fastest, smallest and lowest footprint_](#benchmarks) GPT tokenizer available for all JavaScript environments. It's written in TypeScript.\n\nThis library has been trusted by:\n\n- [CodeRabbit](https://www.coderabbit.ai/) (sponsor 🩷)\n- Microsoft ([Teams](https://github.com/microsoft/teams-ai), [GenAIScript](https://github.com/microsoft/genaiscript/))\n- Elastic ([Kibana](https://github.com/elastic/kibana))\n- [Effect TS](https://effect.website/)\n- [Rivet](https://github.com/Ironclad/rivet) by Ironclad\n\nPlease consider [🩷 sponsoring](https://github.com/sponsors/niieani) the project if you find it useful.\n\n#### Features\n\nAs of 2023, it is the most feature-complete, open-source GPT tokenizer on NPM. This package is a port of OpenAI's [tiktoken](https://github.com/openai/tiktoken), with some additional, unique features sprinkled on top:\n\n- Support for easily tokenizing chats thanks to the `encodeChat` function\n- Support for all current OpenAI models (available encodings: `r50k_base`, `p50k_base`, `p50k_edit`, `cl100k_base` and `o200k_base`)\n- Can be loaded and work synchronously! (i.e. in non async/await contexts)\n- Generator function versions of both the decoder and encoder functions\n- Provides the ability to decode an asynchronous stream of data (using `decodeAsyncGenerator` and `decodeGenerator` with any iterable input)\n- No global cache (no accidental memory leaks, as with the original GPT-3-Encoder implementation)\n- Includes a highly performant `isWithinTokenLimit` function to assess token limit without encoding the entire text/chat\n- Improves overall performance by eliminating transitive arrays\n- Type-safe (written in TypeScript)\n- Works in the browser out-of-the-box\n\n## Installation\n\n### As NPM package\n\n```bash\nnpm install gpt-tokenizer\n```\n\n### As a UMD module\n\n```html\n\u003cscript src=\"https://unpkg.com/gpt-tokenizer\"\u003e\u003c/script\u003e\n\n\u003cscript\u003e\n  // the package is now available as a global:\n  const { encode, decode } = GPTTokenizer_cl100k_base\n\u003c/script\u003e\n```\n\nIf you wish to use a custom encoding, fetch the relevant script.\n\n- https://unpkg.com/gpt-tokenizer/dist/o200k_base.js (for `gpt-4o` and `o1`)\n- https://unpkg.com/gpt-tokenizer/dist/cl100k_base.js (for `gpt-4-*` and `gpt-3.5-turbo`)\n- https://unpkg.com/gpt-tokenizer/dist/p50k_base.js\n- https://unpkg.com/gpt-tokenizer/dist/p50k_edit.js\n- https://unpkg.com/gpt-tokenizer/dist/r50k_base.js\n\nThe global name is a concatenation: `GPTTokenizer_${encoding}`.\n\nRefer to [supported models and their encodings](#Supported-models-and-their-encodings) section for more information.\n\n## Playground\n\nThe playground is published under a memorable URL: https://gpt-tokenizer.dev/\n\nYou can play with the package in the browser using the CodeSandbox [Playground](https://codesandbox.io/s/gpt-tokenizer-tjcjoz?fontsize=14\u0026hidenavigation=1\u0026theme=dark).\n\n[![GPT Tokenizer Playground](./docs/gpt-tokenizer.png)](https://codesandbox.io/s/gpt-tokenizer-tjcjoz?fontsize=14\u0026hidenavigation=1\u0026theme=dark)\n\nThe playground mimics the official [OpenAI Tokenizer](https://platform.openai.com/tokenizer).\n\n## Usage\n\nThe library provides various functions to transform text into (and from) a sequence of integers (tokens) that can be fed into an LLM model. The transformation is done using a Byte Pair Encoding (BPE) algorithm used by OpenAI.\n\n```typescript\nimport {\n  encode,\n  encodeChat,\n  decode,\n  isWithinTokenLimit,\n  encodeGenerator,\n  decodeGenerator,\n  decodeAsyncGenerator,\n} from 'gpt-tokenizer'\n// note: depending on the model, import from the respective file, e.g.:\n// import {...} from 'gpt-tokenizer/model/gpt-4o'\n\nconst text = 'Hello, world!'\nconst tokenLimit = 10\n\n// Encode text into tokens\nconst tokens = encode(text)\n\n// Decode tokens back into text\nconst decodedText = decode(tokens)\n\n// Check if text is within the token limit\n// returns false if the limit is exceeded, otherwise returns the actual number of tokens (truthy value)\nconst withinTokenLimit = isWithinTokenLimit(text, tokenLimit)\n\n// Example chat:\nconst chat = [\n  { role: 'system', content: 'You are a helpful assistant.' },\n  { role: 'assistant', content: 'gpt-tokenizer is awesome.' },\n] as const\n\n// Encode chat into tokens\nconst chatTokens = encodeChat(chat)\n\n// Check if chat is within the token limit\nconst chatWithinTokenLimit = isWithinTokenLimit(chat, tokenLimit)\n\n// Encode text using generator\nfor (const tokenChunk of encodeGenerator(text)) {\n  console.log(tokenChunk)\n}\n\n// Decode tokens using generator\nfor (const textChunk of decodeGenerator(tokens)) {\n  console.log(textChunk)\n}\n\n// Decode tokens using async generator\n// (assuming `asyncTokens` is an AsyncIterableIterator\u003cnumber\u003e)\nfor await (const textChunk of decodeAsyncGenerator(asyncTokens)) {\n  console.log(textChunk)\n}\n```\n\nBy default, importing from `gpt-tokenizer` uses `cl100k_base` encoding, used by `gpt-3.5-turbo` and `gpt-4`.\n\nTo get a tokenizer for a different model, import it directly, for example:\n\n```ts\nimport {\n  encode,\n  decode,\n  isWithinTokenLimit,\n  // etc...\n} from 'gpt-tokenizer/model/gpt-3.5-turbo'\n```\n\nIf you're dealing with a resolver that doesn't support package.json `exports` resolution, you might need to import from the respective `cjs` or `esm` directory, e.g.:\n\n```ts\nimport {\n  encode,\n  decode,\n  isWithinTokenLimit,\n  // etc...\n} from 'gpt-tokenizer/cjs/model/gpt-3.5-turbo'\n```\n\n#### Lazy loading\n\nIf you don't mind loading the tokenizer asynchronously, you can use a dynamic import inside your function, like so:\n\n```ts\nconst {\n  encode,\n  decode,\n  isWithinTokenLimit,\n  // etc...\n} = await import('gpt-tokenizer/model/gpt-3.5-turbo')\n```\n\n#### Loading an encoding\n\nIf your model isn't supported by the package, but you know which BPE encoding it uses, you can load the encoding directly, e.g.:\n\n```ts\nimport {\n  encode,\n  decode,\n  isWithinTokenLimit,\n  // etc...\n} from 'gpt-tokenizer/encoding/cl100k_base'\n```\n\n### Supported models and their encodings\n\n- `o1-*` (`o200k_base`)\n- `gpt-4o` (`o200k_base`)\n- `gpt-4-*` (`cl100k_base`)\n- `gpt-3.5-turbo` (`cl100k_base`)\n- `text-davinci-003` (`p50k_base`)\n- `text-davinci-002` (`p50k_base`)\n- `text-davinci-001` (`r50k_base`)\n- ...and many other models, see [models.ts](./src/models.ts) for an up-to-date list of supported models and their encodings.\n\nNote: if you're using `gpt-3.5-*` or `gpt-4-*` and don't see the model you're looking for, use the `cl100k_base` encoding directly.\n\n## API\n\n### `encode(text: string, encodeOptions?: EncodeOptions): number[]`\n\nEncodes the given text into a sequence of tokens. Use this method when you need to transform a piece of text into the token format that the GPT models can process.\n\nThe optional `encodeOptions` parameter allows you to specify special token handling (see [special tokens](#special-tokens)).\n\nExample:\n\n```typescript\nimport { encode } from 'gpt-tokenizer'\n\nconst text = 'Hello, world!'\nconst tokens = encode(text)\n```\n\n### `decode(tokens: number[]): string`\n\nDecodes a sequence of tokens back into text. Use this method when you want to convert the output tokens from GPT models back into human-readable text.\n\nExample:\n\n```typescript\nimport { decode } from 'gpt-tokenizer'\n\nconst tokens = [18435, 198, 23132, 328]\nconst text = decode(tokens)\n```\n\n### `isWithinTokenLimit(text: string, tokenLimit: number): false | number`\n\nChecks if the text is within the token limit. Returns `false` if the limit is exceeded, otherwise returns the number of tokens. Use this method to quickly check if a given text is within the token limit imposed by GPT models, without encoding the entire text.\n\nExample:\n\n```typescript\nimport { isWithinTokenLimit } from 'gpt-tokenizer'\n\nconst text = 'Hello, world!'\nconst tokenLimit = 10\nconst withinTokenLimit = isWithinTokenLimit(text, tokenLimit)\n```\n\n### `countTokens(text: string | Iterable\u003cChatMessage\u003e, encodeOptions?: EncodeOptions): number`\n\nCounts the number of tokens in the input text or chat. Use this method when you need to determine the number of tokens without checking against a limit.\nThe optional `encodeOptions` parameter allows you to specify custom sets of allowed or disallowed special tokens.\n\nExample:\n\n```typescript\nimport { countTokens } from 'gpt-tokenizer'\n\nconst text = 'Hello, world!'\nconst tokenCount = countTokens(text)\n```\n\n### `encodeChat(chat: ChatMessage[], model?: ModelName): number[]`\n\nEncodes the given chat into a sequence of tokens.\n\nIf you didn't import the model version directly, or if `model` wasn't provided during initialization, it must be provided here to correctly tokenize the chat for a given model. Use this method when you need to transform a chat into the token format that the GPT models can process.\n\nExample:\n\n```typescript\nimport { encodeChat } from 'gpt-tokenizer'\n\nconst chat = [\n  { role: 'system', content: 'You are a helpful assistant.' },\n  { role: 'assistant', content: 'gpt-tokenizer is awesome.' },\n]\nconst tokens = encodeChat(chat)\n```\n\nNote that if you encode an empty chat, it will still contain the minimum number of special tokens.\n\n### `encodeGenerator(text: string): Generator\u003cnumber[], void, undefined\u003e`\n\nEncodes the given text using a generator, yielding chunks of tokens.\nUse this method when you want to encode text in chunks, which can be useful for processing large texts or streaming data.\n\nExample:\n\n```typescript\nimport { encodeGenerator } from 'gpt-tokenizer'\n\nconst text = 'Hello, world!'\nconst tokens = []\nfor (const tokenChunk of encodeGenerator(text)) {\n  tokens.push(...tokenChunk)\n}\n```\n\n### `encodeChatGenerator(chat: Iterator\u003cChatMessage\u003e, model?: ModelName): Generator\u003cnumber[], void, undefined\u003e`\n\nSame as `encodeChat`, but uses a generator as output, and may use any iterator as the input `chat`.\n\n### `decodeGenerator(tokens: Iterable\u003cnumber\u003e): Generator\u003cstring, void, undefined\u003e`\n\nDecodes a sequence of tokens using a generator, yielding chunks of decoded text.\nUse this method when you want to decode tokens in chunks, which can be useful for processing large outputs or streaming data.\n\nExample:\n\n```typescript\nimport { decodeGenerator } from 'gpt-tokenizer'\n\nconst tokens = [18435, 198, 23132, 328]\nlet decodedText = ''\nfor (const textChunk of decodeGenerator(tokens)) {\n  decodedText += textChunk\n}\n```\n\n### `decodeAsyncGenerator(tokens: AsyncIterable\u003cnumber\u003e): AsyncGenerator\u003cstring, void, undefined\u003e`\n\nDecodes a sequence of tokens asynchronously using a generator, yielding chunks of decoded text. Use this method when you want to decode tokens in chunks asynchronously, which can be useful for processing large outputs or streaming data in an asynchronous context.\n\nExample:\n\n```javascript\nimport { decodeAsyncGenerator } from 'gpt-tokenizer'\n\nasync function processTokens(asyncTokensIterator) {\n  let decodedText = ''\n  for await (const textChunk of decodeAsyncGenerator(asyncTokensIterator)) {\n    decodedText += textChunk\n  }\n}\n```\n\n## Special tokens\n\nThere are a few special tokens that are used by the GPT models.\nNote that not all models support all of these tokens.\n\nBy default, **all special tokens are disallowed**.\n\nThe `encode`, `encodeGenerator` and `countTokens` functions accept an `EncodeOptions` parameter to customize special token handling:\n\n### Custom Allowed Sets\n\n`gpt-tokenizer` allows you to specify custom sets of allowed special tokens when encoding text. To do this, pass a\n`Set` containing the allowed special tokens as a parameter to the `encode` function:\n\n```ts\nimport {\n  EndOfPrompt,\n  EndOfText,\n  FimMiddle,\n  FimPrefix,\n  FimSuffix,\n  ImStart,\n  ImEnd,\n  ImSep,\n  encode,\n} from 'gpt-tokenizer'\n\nconst inputText = `Some Text ${EndOfPrompt}`\nconst allowedSpecialTokens = new Set([EndOfPrompt])\nconst encoded = encode(inputText, { allowedSpecialTokens })\nconst expectedEncoded = [8538, 2991, 220, 100276]\n\nexpect(encoded).toBe(expectedEncoded)\n```\n\nYou may also use a special shorthand for either disallowing or allowing all special tokens, by passing in the string `'all'`, e.g. `{ allowedSpecial: 'all' }`.\n\n### Custom Disallowed Sets\n\nSimilarly, you can specify custom sets of disallowed special tokens when encoding text. Pass a `Set`\ncontaining the disallowed special tokens as a parameter to the `encode` function:\n\n```ts\nimport { encode, EndOfText } from 'gpt-tokenizer'\n\nconst inputText = `Some Text ${EndOfText}`\nconst disallowedSpecial = new Set([EndOfText])\n// throws an error:\nconst encoded = encode(inputText, { disallowedSpecial })\n```\n\nIn this example, an Error is thrown, because the input text contains a disallowed special token.\n\nIf both `allowedSpecialTokens` and `disallowedSpecial` are provided, `disallowedSpecial` takes precedence.\n\n## Performance Optimization\n\n### LRU Merge Cache\n\nThe tokenizer uses an LRU (Least Recently Used) cache to improve encoding performance for similar strings. By default, it stores up to 100,000 merged token pairs. You can adjust this value to optimize for your specific use case:\n\n- Increasing the cache size will make encoding similar strings faster but consume more memory\n- Setting it to 0 will disable caching completely\n- For applications processing many unique strings, a smaller cache might be more efficient\n\nYou can modify the cache size using the `setMergeCacheSize` function:\n\n```ts\nimport { setMergeCacheSize } from 'gpt-tokenizer'\n\n// Set to 5000 entries\nsetMergeCacheSize(5000)\n\n// Disable caching completely\nsetMergeCacheSize(0)\n```\n\nThe cache is persisted between encoding calls. To explicitly clear the cache (e.g. to free up memory), use the `clearMergeCache` function:\n\n```ts\nimport { clearMergeCache } from 'gpt-tokenizer'\n\nclearMergeCache()\n```\n\n## Testing and Validation\n\n`gpt-tokenizer` includes a set of test cases in the [TestPlans.txt](./data/TestPlans.txt) file to ensure its compatibility with OpenAI's Python `tiktoken` library. These test cases validate the functionality and behavior of `gpt-tokenizer`, providing a reliable reference for developers.\n\nRunning the unit tests and verifying the test cases helps maintain consistency between the library and the original Python implementation.\n\n### Model Information\n\n`gpt-tokenizer` provides comprehensive data about all OpenAI models through the `models` export from [`gpt-tokenizer/models`](./src/models.ts). This includes detailed information about context windows, costs, training data cutoffs, and deprecation status.\n\nThe data is regularly maintained to match OpenAI's official documentation. Contributions to keep this data up-to-date are welcome - if you notice any discrepancies or have updates, please feel free to open a PR.\n\n## [Benchmarks](https://l8j6fv.csb.app/)\n\nSince version 2.4.0, `gpt-tokenizer` is the fastest tokenizer implementation available on NPM. It's even faster than the available WASM/node binding implementations.\nIt has the fastest encoding, decoding time and a tiny memory footprint. It also initializes faster than all other implementations.\n\nThe encodings themselves are also the smallest in size, due to the compact format they are stored in.\n\n![fastest benchmark](./docs/fastest.png)\n\n![lowest footprint benchmark](./docs/lowest-footprint.png)\n\n## License\n\nMIT\n\n## Contributing\n\nContributions are welcome! Please open a pull request or an issue to discuss your bug reports, or use the discussions feature for ideas or any other inquiries.\n\n## Thanks\n\nThanks to @dmitry-brazhenko's [SharpToken](https://github.com/dmitry-brazhenko/SharpToken), whose code was served as a reference for the port.\n\nHope you find the `gpt-tokenizer` useful in your projects!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fniieani%2Fgpt-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fniieani%2Fgpt-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fniieani%2Fgpt-tokenizer/lists"}