{"id":51619994,"url":"https://github.com/coder/ai-tokenizer","last_synced_at":"2026-07-12T18:02:15.905Z","repository":{"id":319726100,"uuid":"1073276146","full_name":"coder/ai-tokenizer","owner":"coder","description":"A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.","archived":false,"fork":false,"pushed_at":"2025-12-01T18:50:25.000Z","size":16190,"stargazers_count":54,"open_issues_count":3,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-02T14:07:36.041Z","etag":null,"topics":["ai","ai-sdk","llm","token","tokenizer"],"latest_commit_sha":null,"homepage":"https://coder.github.io/ai-tokenizer/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-09T21:45:12.000Z","updated_at":"2026-05-20T15:28:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"963f28bf-f4e9-4a93-b5b0-947328c4c45c","html_url":"https://github.com/coder/ai-tokenizer","commit_stats":null,"previous_names":["coder/ai-tokenizer"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/coder/ai-tokenizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coder%2Fai-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coder%2Fai-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coder%2Fai-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coder%2Fai-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coder","download_url":"https://codeload.github.com/coder/ai-tokenizer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coder%2Fai-tokenizer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35398566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-12T02:00:06.386Z","response_time":87,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-sdk","llm","token","tokenizer"],"created_at":"2026-07-12T18:02:15.100Z","updated_at":"2026-07-12T18:02:15.889Z","avatar_url":"https://github.com/coder.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ai-tokenizer\n\n[![CI](https://github.com/coder/ai-tokenizer/actions/workflows/ci.yml/badge.svg)](https://github.com/coder/ai-tokenizer/actions/workflows/ci.yml)\n[![npm version](https://img.shields.io/npm/v/ai-tokenizer)](https://www.npmjs.com/package/ai-tokenizer)\n[![npm downloads](https://img.shields.io/npm/dm/ai-tokenizer)](https://www.npmjs.com/package/ai-tokenizer)\n\nA faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.\n\n- Supports the [AI SDK](https://github.com/vercel/ai) tool and message schema\n- No WASM; highly portable; [5-7x faster](#performance) than tiktoken WASM\n- \\\u003e=97% [accuracy](#accuracy) for top models (GPT-5, Claude Sonnet, Gemini):\n\n\u003c!-- POPULAR_MODELS_TABLE_START --\u003e\n| Model | ~500 tokens | ~5k tokens | ~50k tokens |\n|-------|-------------|------------|-------------|\n| openai/gpt-5 | 99.16% | 99.43% | 99.47% |\n| anthropic/claude-opus-4.5 | 98.48% | 98.56% | 99.79% |\n| google/gemini-3-pro-preview | 98.70% | 98.77% | 98.91% |\n\n\u003c!-- POPULAR_MODELS_TABLE_END --\u003e\n\n[Try it on the website:](https://coder.github.io/ai-tokenizer/)\n\n[![Demo](https://raw.githubusercontent.com/coder/ai-tokenizer/main/demo.png)](https://coder.github.io/ai-tokenizer/)\n\n## Usage\n\nInstall:\n\n```sh\nbun install ai-tokenizer\n```\n\nEstimate tokens with the AI SDK:\n\n```ts\nimport Tokenizer, { models } from \"ai-tokenizer\";\nimport { count } from \"ai-tokenizer/sdk\";\nimport * as encoding from \"ai-tokenizer/encoding\";\n\n// Find the respective model.\nconst model = models[\"openai/gpt-5\"];\nconst tokenizer = new Tokenizer(encoding[model.encoding]);\nconst result = count({\n  tokenizer,\n  model,\n  // Every message is counted separately\n  messages,\n  // Tool descriptions and input schemas are counted\n  tools,\n})\n\n// {\n//   total: 150,\n//   messages: [\n//     {\n//       total: 120,\n//       content: [\n//         { type: \"text\", total: 100 },\n//         { type: \"tool-call\", total: 20, input: 15 }\n//       ]\n//     }\n//   ],\n//   tools: {\n//     total: 30,\n//     definitions: {\n//       myTool: {\n//         name: 5,\n//         description: 10,\n//         inputSchema: 15\n//       }\n//     }\n//   }\n// }\n\n// No API calls are performed; no keys.\n\nconst costUSD = result.total * model.pricing.input;\n```\n\nEstimate tokens using an encoding:\n\n```ts\nimport { Tokenizer } from \"ai-tokenizer\"\nimport * as o200k_base from \"ai-tokenizer/encoding/o200k_base\"\n\nconst tokenizer = new Tokenizer(o200k_base);\nconst total = tokenizer.count(\"some text input\");\n```\n\n\u003e [!WARNING]\n\u003e Import encodings selectively based on what you need. Each encoding is 2-8MB uncompressed.\n\n## Accuracy\n\nValidated against actual API responses with pseudo-random messages:\n\n\u003c!-- ACCURACY_TABLE_START --\u003e\n| Model | ~500 tokens | ~5k tokens | ~50k tokens |\n|-------|-------------|------------|-------------|\n| alibaba/qwen-3-14b | 96.45% | 96.45% | N/A |\n| alibaba/qwen-3-235b | 98.26% | 96.38% | N/A |\n| alibaba/qwen-3-30b | 96.45% | 96.27% | N/A |\n| alibaba/qwen-3-32b | 95.64% | 95.70% | N/A |\n| alibaba/qwen3-235b-a22b-thinking | 96.06% | 96.34% | 95.98% |\n| alibaba/qwen3-coder | 94.55% | 96.70% | 94.57% |\n| alibaba/qwen3-coder-plus | 94.84% | 96.76% | 94.54% |\n| alibaba/qwen3-max | 98.51% | 95.59% | 94.33% |\n| alibaba/qwen3-next-80b-a3b-instruct | 95.66% | 96.71% | 95.81% |\n| alibaba/qwen3-next-80b-a3b-thinking | 96.02% | 95.70% | 95.09% |\n| alibaba/qwen3-vl-instruct | 95.08% | 96.73% | 96.04% |\n| amazon/nova-lite | 98.10% | 92.67% | 90.17% |\n| amazon/nova-micro | 98.10% | 92.10% | 89.20% |\n| amazon/nova-pro | 98.10% | 92.82% | 90.16% |\n| anthropic/claude-3-haiku | 97.61% | 99.32% | 99.77% |\n| anthropic/claude-3-opus | 98.66% | 98.68% | 99.70% |\n| anthropic/claude-3.5-haiku | 97.61% | 99.80% | 99.86% |\n| anthropic/claude-3.5-sonnet | 97.93% | 99.26% | 99.67% |\n| anthropic/claude-3.5-sonnet-20240620 | 98.01% | 99.98% | 99.86% |\n| anthropic/claude-3.7-sonnet | 98.07% | 99.59% | 99.74% |\n| anthropic/claude-haiku-4.5 | 98.48% | 98.67% | 99.62% |\n| anthropic/claude-opus-4 | 98.07% | 99.39% | 99.74% |\n| anthropic/claude-opus-4.1 | 97.78% | 99.83% | 99.80% |\n| anthropic/claude-opus-4.5 | 98.48% | 98.56% | 99.79% |\n| anthropic/claude-sonnet-4 | 98.22% | 99.89% | 99.83% |\n| anthropic/claude-sonnet-4.5 | 98.48% | 98.91% | 99.70% |\n| deepseek/deepseek-r1 | 96.24% | 98.67% | 98.10% |\n| deepseek/deepseek-r1-distill-llama-70b | 98.20% | 99.08% | N/A |\n| deepseek/deepseek-v3 | 95.30% | 95.37% | 95.19% |\n| deepseek/deepseek-v3.1 | 97.40% | 95.85% | 96.46% |\n| deepseek/deepseek-v3.1-terminus | 96.20% | 97.57% | 97.55% |\n| deepseek/deepseek-v3.2-exp | 95.77% | 97.62% | 97.45% |\n| deepseek/deepseek-v3.2-exp-thinking | 95.98% | 97.55% | 97.56% |\n| google/gemini-2.0-flash | 97.97% | 99.69% | 99.95% |\n| google/gemini-2.0-flash-lite | 98.18% | 99.65% | 99.77% |\n| google/gemini-2.5-flash | 98.78% | 99.60% | 99.85% |\n| google/gemini-2.5-flash-lite | 99.19% | 99.33% | 99.84% |\n| google/gemini-2.5-flash-lite-preview-09-2025 | 92.74% | 99.63% | 99.74% |\n| google/gemini-2.5-flash-preview-09-2025 | 92.53% | 99.69% | 99.74% |\n| google/gemini-2.5-pro | 98.37% | 99.42% | 99.82% |\n| google/gemini-3-pro-preview | 98.70% | 98.77% | 98.91% |\n| meituan/longcat-flash-chat | 70.65% | 99.15% | 96.13% |\n| meta/llama-3.1-8b | 98.69% | 90.67% | 89.69% |\n| meta/llama-3.3-70b | 100.00% | 70.03% | 67.23% |\n| meta/llama-4-scout | 0.00% | 56.65% | 66.03% |\n| minimax/minimax-m2 | 94.13% | 74.53% | 72.83% |\n| mistral/codestral | 93.78% | 91.86% | 91.89% |\n| mistral/devstral-small | 98.77% | 94.09% | 93.72% |\n| mistral/magistral-medium | 98.24% | 94.23% | 93.71% |\n| mistral/magistral-small | 97.89% | 94.32% | 93.65% |\n| mistral/mistral-large | 98.23% | 94.09% | N/A |\n| mistral/mistral-medium | 98.06% | 93.96% | 93.79% |\n| mistral/mistral-small | 96.73% | 92.78% | N/A |\n| mistral/pixtral-large | 98.94% | 94.25% | 94.27% |\n| moonshotai/kimi-k2 | 89.51% | NaN% | 82.80% |\n| moonshotai/kimi-k2-0905 | 85.58% | 81.01% | 80.56% |\n| moonshotai/kimi-k2-thinking | 95.02% | 92.80% | 91.96% |\n| moonshotai/kimi-k2-thinking-turbo | 95.03% | 92.38% | 91.91% |\n| moonshotai/kimi-k2-turbo | 94.51% | 92.36% | 91.83% |\n| openai/gpt-4-turbo | 99.79% | 99.89% | 99.86% |\n| openai/gpt-4.1 | 100.00% | 99.93% | 99.92% |\n| openai/gpt-4.1-mini | 100.00% | 99.93% | 99.92% |\n| openai/gpt-4o | 100.00% | 99.93% | 99.92% |\n| openai/gpt-4o-mini | 100.00% | 99.93% | 99.92% |\n| openai/gpt-5 | 99.16% | 99.43% | 99.47% |\n| openai/gpt-5-chat | 100.00% | 99.93% | 99.92% |\n| openai/gpt-5-codex | 99.17% | 99.43% | 99.47% |\n| openai/gpt-5-mini | 99.16% | 99.43% | 99.47% |\n| openai/gpt-5-nano | 99.16% | 99.43% | 99.46% |\n| openai/gpt-5-pro | 99.17% | 99.43% | 99.47% |\n| openai/gpt-5.1-codex | 79.84% | 98.72% | 99.65% |\n| openai/gpt-5.1-codex-mini | 99.17% | 99.43% | 99.47% |\n| openai/gpt-5.1-instant | 79.84% | 98.72% | 99.65% |\n| openai/gpt-5.1-thinking | 79.84% | 98.72% | 99.64% |\n| openai/gpt-oss-120b | 80.77% | 69.64% | 69.62% |\n| openai/gpt-oss-20b | 68.55% | 60.79% | 60.98% |\n| openai/gpt-oss-safeguard-20b | 75.60% | 64.08% | 63.67% |\n| openai/o1 | 94.77% | 99.30% | 99.86% |\n| openai/o3 | 99.16% | 99.43% | 99.46% |\n| openai/o3-mini | 100.00% | NaN% | 99.92% |\n| openai/o4-mini | 99.16% | 99.43% | 99.47% |\n| prime-intellect/intellect-3 | 93.08% | 95.64% | 97.12% |\n| stealth/sonoma-dusk-alpha | 99.47% | 71.65% | 66.69% |\n| stealth/sonoma-sky-alpha | 99.46% | 69.52% | 66.49% |\n| vercel/v0-1.0-md | 98.99% | 53.45% | 92.42% |\n| vercel/v0-1.5-md | 98.96% | 53.55% | 92.39% |\n| xai/grok-2 | 95.43% | 96.09% | 95.28% |\n| xai/grok-2-vision | 94.44% | 95.68% | N/A |\n| xai/grok-3 | 99.57% | 98.67% | 99.23% |\n| xai/grok-3-fast | 99.57% | 98.59% | 99.13% |\n| xai/grok-3-mini | 99.59% | 98.76% | 99.18% |\n| xai/grok-3-mini-fast | 99.59% | 98.59% | 99.13% |\n| xai/grok-4 | 99.72% | 73.95% | 67.43% |\n| xai/grok-4-fast-non-reasoning | 90.02% | -22.27% | -52.27% |\n| xai/grok-4-fast-reasoning | 99.29% | 71.17% | 66.72% |\n| xai/grok-4.1-fast-non-reasoning | 99.65% | 71.70% | 66.76% |\n| xai/grok-4.1-fast-reasoning | 99.28% | 71.50% | 66.62% |\n| xai/grok-code-fast-1 | 80.14% | -13.63% | -51.13% |\n| zai/glm-4.5 | 94.55% | 96.26% | 96.16% |\n| zai/glm-4.5-air | 96.73% | 96.45% | 96.70% |\n| zai/glm-4.5v | 95.17% | 95.56% | 95.36% |\n\n\u003c!-- ACCURACY_TABLE_END --\u003e\n\n*Accuracy shows percentage within actual token count.*\n\n\u003e [!WARNING]\n\u003e Not every tool/token scenario is tested for accuracy. There will be edge-cases where this is more/less accurate. If you find this inaccurate for your scenario, please open an issue.\n\nRun `bun run scripts/generate-accuracy.ts` to update this table. Refer to [accuracy.json](./accuracy.json) for greater detail.\n\n## Performance\n\nai-tokenizer is **5-7x faster than tiktoken** for counting tokens. It is on par with gpt-tokenizer.\n\n```bash\n$ bun bench/versus\n\nclk: ~3.96 GHz\ncpu: AMD Ryzen AI 9 HX 370 w/ Radeon 890M\nruntime: bun 1.2.19 (x64-linux)\n\nbenchmark                   avg (min … max) p75 / p99    (min … top 1%)\n------------------------------------------- -------------------------------\n• initialization\n------------------------------------------- -------------------------------\nai-tokenizer                  13.99 µs/iter  14.14 µs  █                   \n                      (13.44 µs … 15.50 µs)  14.85 µs  █                   \n                    (  0.00  b …   1.24 kb) 308.13  b ██▁█▁█▁▁▁▁█▁▁▁▁▁▁▁▁▁█\n\ntiktoken                      85.05 ms/iter  85.68 ms         █      █     \n                      (81.61 ms … 89.19 ms)  86.55 ms ▅       █  ▅ ▅▅█▅▅  ▅\n                    (  0.00  b …   1.39 mb) 385.33 kb █▁▁▁▁▁▁▁█▁▁█▁█████▁▁█\n\n• encode: small text (~13 chars)\n------------------------------------------- -------------------------------\nai-tokenizer                   1.33 µs/iter   1.34 µs         ▇█           \n                        (1.11 µs … 1.90 µs)   1.60 µs         ██           \n                    (  0.00  b …   2.06 kb)   1.60 kb ▆▃▁▁▁▁▁▂███▃▄▂▃▂▃▂▁▁▂\n\ngpt-tokenizer                  1.05 µs/iter   1.13 µs █                    \n                      (926.82 ns … 1.65 µs)   1.53 µs ██    ▆              \n                    (  0.00  b … 864.00  b) 352.86  b ██▂▁▁▁██▆▃▂▂▁▁▁▁▁▁▁▁▁\n\ntiktoken                      32.23 µs/iter  30.19 µs  █                   \n                     (27.83 µs … 468.68 µs)  66.56 µs  █                   \n                    (  0.00  b …   1.13 mb)   5.27 kb ██▃▂▂▁▁▁▃▁▁▁▁▁▁▁▁▁▁▁▁\n\n• encode: medium text (~4.5KB)\n------------------------------------------- -------------------------------\nai-tokenizer                  89.81 µs/iter  92.53 µs    █                 \n                     (66.07 µs … 372.65 µs) 208.17 µs  █ █▇                \n                    (  0.00  b …   1.13 mb)  98.84 kb ▄█▃██▃▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁\n\ngpt-tokenizer                 76.97 µs/iter  80.01 µs  █   ▆               \n                     (66.79 µs … 279.16 µs) 115.17 µs  █   █▅              \n                    (  0.00  b … 384.00 kb)   5.44 kb ▆█▅█▆██▃▂▁▂▂▁▂▂▁▁▁▁▁▁\n\ntiktoken                     651.22 µs/iter 649.48 µs  ▄█                  \n                    (632.83 µs … 874.11 µs) 782.25 µs  ██                  \n                    (  0.00  b … 192.00 kb)   4.31 kb ▄██▆▂▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁\n\n• encode: large text (~500KB)\n------------------------------------------- -------------------------------\nai-tokenizer                  19.07 ms/iter  20.23 ms                  ▅█  \n                      (15.19 ms … 20.67 ms)  20.66 ms       ▃         ▃██▃▃\n                    (  0.00  b …  26.44 mb)  19.22 mb ▄▁▄▄█▁█▁▁▁▄▁▁▁▄▄█████\n\ngpt-tokenizer                 18.19 ms/iter  18.62 ms     █                \n                      (15.35 ms … 23.68 ms)  22.91 ms     █ ▃              \n                    (  0.00  b …  31.50 mb)   6.44 mb ▅▃▅▁█▁█▃▃▃▁▁█▁▁▁▁▅▁▁▅\n\ntiktoken                     135.38 ms/iter 135.35 ms █     █ █            \n                    (134.51 ms … 138.22 ms) 136.61 ms █▅ ▅ ▅█ █▅          ▅\n                    (  0.00  b … 768.00 kb) 648.00 kb ██▁█▁██▁██▁▁▁▁▁▁▁▁▁▁█\n\n• encode: unicode text\n------------------------------------------- -------------------------------\nai-tokenizer                 118.76 µs/iter 118.95 µs     █                \n                     (84.61 µs … 500.48 µs) 234.66 µs    ██                \n                    (  0.00  b … 768.00 kb) 129.37 kb ▃▄▂██▇▄▄▂▂▂▂▁▁▁▁▁▁▁▁▁\n\ngpt-tokenizer                231.35 µs/iter 256.95 µs   █▅▇█               \n                      (139.24 µs … 2.08 ms) 489.02 µs   ████  █            \n                    (  0.00  b …   2.41 mb) 260.61 kb ▃▅████▇▆█▆▃▂▂▂▁▁▁▂▁▁▁\n\ntiktoken                     836.84 µs/iter 819.36 µs █                    \n                      (799.44 µs … 1.97 ms)   1.39 ms █▅                   \n                    (  0.00  b … 192.00 kb)   5.87 kb ██▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁\n\n• encode: code\n------------------------------------------- -------------------------------\nai-tokenizer                 357.64 µs/iter 372.83 µs      █▄              \n                      (278.58 µs … 1.04 ms) 591.56 µs  ▃   ██              \n                    (  0.00  b …   1.31 mb) 364.10 kb ██▂▁▂██▅▃▂▂▁▂▂▁▁▁▁▁▁▁\n\ngpt-tokenizer                316.39 µs/iter 316.25 µs  █▅                  \n                    (279.98 µs … 919.92 µs) 622.31 µs  ██                  \n                    (  0.00  b … 192.00 kb)  54.12 kb ▄██▅▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁\n\ntiktoken                       3.12 ms/iter   3.12 ms  ▆█                  \n                        (3.05 ms … 3.72 ms)   3.52 ms  ██▃                 \n                    (  0.00  b … 192.00 kb)  20.06 kb ▆███▅▃▁▁▁▁▂▂▂▂▁▁▂▁▁▁▂\n\n• encode: mixed content\n------------------------------------------- -------------------------------\nai-tokenizer                   2.44 ms/iter   2.48 ms         ▇█▄          \n                        (2.06 ms … 2.89 ms)   2.81 ms        ▂████▃        \n                    (  0.00  b …   3.56 mb)   2.36 mb ▁▂▂▁▁▁▁██████▄▃▆▄▂▂▁▂\n\ngpt-tokenizer                  2.62 ms/iter   2.60 ms   █                  \n                        (2.38 ms … 4.34 ms)   3.96 ms  ▅█                  \n                    (576.00 kb …   4.88 mb)   1.34 mb ▄███▄▂▂▁▁▁▁▂▁▁▂▁▁▁▁▁▁\n\ntiktoken                      15.35 ms/iter  15.37 ms ▂█ ▂                 \n                      (15.17 ms … 16.35 ms)  15.80 ms ██▇█▂                \n                    (  0.00  b … 192.00 kb)  93.54 kb █████▇▄▁▄▄▄▁▄▄▄▁▁▇▄▁▄\n\n• decode: large token array\n------------------------------------------- -------------------------------\nai-tokenizer                   1.57 ms/iter   1.93 ms █                    \n                        (1.14 ms … 4.02 ms)   3.53 ms ██                   \n                    (  0.00  b …   6.94 mb)   1.44 mb ██▄▃▁▁▁▁▁▂▃▄▄▃▂▁▁▁▁▁▁\n\ngpt-tokenizer                876.67 µs/iter 857.62 µs  █                   \n                      (628.32 µs … 3.95 ms)   2.34 ms  █                   \n                    (  0.00  b …   6.00 mb)  27.15 kb ▁██▃▄▃▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁\n\ntiktoken                       1.64 ms/iter   1.67 ms  ▄  █                \n                        (1.47 ms … 2.49 ms)   2.33 ms ▆█ ██▃               \n                    (  0.00  b …   1.31 mb)   1.14 mb ██████▅▆▂▃▁▁▂▁▂▂▁▁▂▁▁\n\n• count: large text (~500KB)\n------------------------------------------- -------------------------------\nai-tokenizer                  18.33 ms/iter  19.52 ms                   █  \n                      (14.94 ms … 20.09 ms)  20.05 ms ▃▆              ▆██▃ \n                    (  0.00  b …  25.33 mb)  18.64 mb ██▁▁▁▄▁▁▁▁▁▁▄▄▄█████▄\n\ngpt-tokenizer                 16.65 ms/iter  18.26 ms  █               █   \n                      (14.63 ms … 19.14 ms)  19.01 ms  ███          ▅  █   \n                    (  0.00  b …   7.50 mb)   1.15 mb ▅████▁▅▅▁▁█▅▁▅██▁██▁█\n\ntiktoken                     131.95 ms/iter 131.85 ms        █             \n                    (130.24 ms … 138.35 ms) 133.42 ms      █ █             \n                    (576.00 kb … 768.00 kb) 720.00 kb █▁██▁█▁█▁▁█▁█▁▁▁▁▁▁▁█\n```\n\nRun this yourself with `bun bench/versus`.\n\n## License\n\n[MIT](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoder%2Fai-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoder%2Fai-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoder%2Fai-tokenizer/lists"}