{"id":18488163,"url":"https://github.com/botisan-ai/gpt3-tokenizer","last_synced_at":"2026-02-08T17:19:07.970Z","repository":{"id":43842723,"uuid":"442091521","full_name":"botisan-ai/gpt3-tokenizer","owner":"botisan-ai","description":"Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.","archived":false,"fork":false,"pushed_at":"2023-01-27T04:07:55.000Z","size":2162,"stargazers_count":171,"open_issues_count":3,"forks_count":16,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-18T04:52:57.618Z","etag":null,"topics":["chatgpt","codex","gpt-3","gpt3","javascript","nodejs","openai","tokenizer","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/botisan-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":["xanthous-tech","lhr0909"]}},"created_at":"2021-12-27T07:59:18.000Z","updated_at":"2025-01-06T18:12:38.000Z","dependencies_parsed_at":"2023-02-15T05:01:42.815Z","dependency_job_id":null,"html_url":"https://github.com/botisan-ai/gpt3-tokenizer","commit_stats":null,"previous_names":["xanthous-tech/gpt3-tokenizer"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/botisan-ai%2Fgpt3-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/botisan-ai%2Fgpt3-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/botisan-ai%2Fgpt3-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/botisan-ai%2Fgpt3-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/botisan-ai","download_url":"https://codeload.github.com/botisan-ai/gpt3-tokenizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247923406,"owners_count":21018987,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatgpt","codex","gpt-3","gpt3","javascript","nodejs","openai","tokenizer","typescript"],"created_at":"2024-11-06T12:51:21.318Z","updated_at":"2026-02-08T17:19:07.959Z","avatar_url":"https://github.com/botisan-ai.png","language":"TypeScript","funding_links":["https://github.com/sponsors/xanthous-tech","https://github.com/sponsors/lhr0909"],"categories":["TypeScript"],"sub_categories":[],"readme":"# GPT3 Tokenizer\n\n[![Build](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml/badge.svg)](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml)\n[![NPM Version](https://img.shields.io/npm/v/gpt3-tokenizer.svg)](https://www.npmjs.com/package/gpt3-tokenizer)\n[![NPM Downloads](https://img.shields.io/npm/dt/gpt3-tokenizer.svg)](https://www.npmjs.com/package/gpt3-tokenizer)\n\nThis is a isomorphic TypeScript tokenizer for OpenAI's GPT-3 model. Including support for `gpt3` and `codex` tokenization. It should work in both NodeJS and Browser environments.\n## Usage\n\nFirst, install:\n\n```shell\nyarn add gpt3-tokenizer\n```\n\nIn code:\n\n```typescript\nimport GPT3Tokenizer from 'gpt3-tokenizer';\n\nconst tokenizer = new GPT3Tokenizer({ type: 'gpt3' }); // or 'codex'\nconst str = \"hello 👋 world 🌍\";\nconst encoded: { bpe: number[]; text: string[] } = tokenizer.encode(str);\nconst decoded = tokenizer.decode(encoded.bpe);\n```\n\n## Reference\n\nThis library is based on the following:\n- [OpenAI Tokenizer Page Source](https://beta.openai.com/tokenizer?view=bpe)\n- [gpt-3-encoder](https://github.com/latitudegames/GPT-3-Encoder)\n\nThe main difference between this library and gpt-3-encoder is that this library supports both `gpt3` and `codex` tokenization (The dictionary is taken directly from OpenAI so the tokenization result is on par with the OpenAI Playground). Also Map API is used instead of JavaScript objects, especially the `bpeRanks` object, which should see some performance improvement.\n\n## License\n\n[MIT](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbotisan-ai%2Fgpt3-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbotisan-ai%2Fgpt3-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbotisan-ai%2Fgpt3-tokenizer/lists"}