{"id":14964608,"url":"https://github.com/lenml/tokenizers","last_synced_at":"2025-10-06T09:30:31.288Z","repository":{"id":234664568,"uuid":"789341239","full_name":"lenML/tokenizers","owner":"lenML","description":"a lightweight no-dependency fork from transformers.js (only tokenizers)","archived":false,"fork":false,"pushed_at":"2024-11-24T12:55:52.000Z","size":1370,"stargazers_count":12,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-20T21:09:47.819Z","etag":null,"topics":["baichuan","chatglm","chatgpt","gpt4","llama2","llama3","mistral","tokenizer","transfomers"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lenML.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-20T09:11:18.000Z","updated_at":"2025-01-03T08:06:20.000Z","dependencies_parsed_at":"2024-05-14T06:31:58.936Z","dependency_job_id":"108a078c-4e8a-4f13-9693-0a9dccf3b6fa","html_url":"https://github.com/lenML/tokenizers","commit_stats":{"total_commits":44,"total_committers":2,"mean_commits":22.0,"dds":0.4545454545454546,"last_synced_commit":"7962081f76c7c3d38b4fe75377714723d51af17d"},"previous_names":["lenml/tokenizers"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenML%2Ftokenizers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenML%2Ftokenizers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenML%2Ftokenizers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenML%2Ftokenizers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lenML","download_url":"https://codeload.github.com/lenML/tokenizers/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235515428,"owners_count":19002481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["baichuan","chatglm","chatgpt","gpt4","llama2","llama3","mistral","tokenizer","transfomers"],"created_at":"2024-09-24T13:33:29.594Z","updated_at":"2025-10-06T09:30:31.272Z","avatar_url":"https://github.com/lenML.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# @lenml/tokenizers\n\nThis is the central repository for the `@lenml/tokenizers` project, which provides tokenization libraries for various machine learning models.\n\n\u003e this repo fork from [huggingface/transformers.js](https://github.com/huggingface/transformers.js/)\n\n# Tokenizer Arena / Playground\n\nExplore our Tokenizer Arena / Playground! This interactive platform allows you to utilize various tokenizers from our @lenml/tokenizers library. Easily load and compare different tokenizers, seeing how they perform with diverse text inputs. Whether you're a professional developer or a machine learning enthusiast, this playground is perfect for gaining insights into the tokenization process of different models and experimenting with their functionalities.\n\n[click to arena page](https://lenml.github.io/tokenizers-arena/#/side-by-side)\n\n[![screenshot](./assets/playground.jpg)](https://lenml.github.io/tokenizers-arena/#/side-by-side)\n\n# When should I use this instead of transformers.js?\n\nFirstly, the interface and the actual code of the Tokenizer object are completely identical to those in transformers.js. However, when loading a tokenizer with this library, you're allowed to create your model directly from a JSON object without the need for internet access, and without relying on Hugging Face (hf) servers, or local files.\n\nTherefore, this library becomes more convenient when you need to operate offline and only require the use of a tokenizer without the need for ONNX models.\n\n# Packages\n\nBelow is a table showcasing all available packages, the models they support, and their respective locations within the repository:\n\n| Package Name            | Supported Model(s)                                                         | Repository Link                                                            | NPM Page                                                                                                                                                                                                                                           |\n| ----------------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `tokenizers` (core)     | N/A (Core Tokenization Library)                                            | [@lenml/tokenizers](./packages/tokenizers)                                 | [![npm](https://img.shields.io/npm/v/@lenml/tokenizers.svg)](https://www.npmjs.com/package/@lenml/tokenizers) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizers)                                                                |\n| `qwen3`                 | Qwen3                                                                      | [@lenml/tokenizer-qwen3](./packages/qwen3)                                 | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-qwen3.svg)](https://www.npmjs.com/package/@lenml/tokenizer-qwen3) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-qwen3)                                                 |\n| `gptoss`                | GPT-OSS-20B GPT-OSS-120B                                                   | [@lenml/tokenizer-gptoss](./packages/gptoss)                               | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-gptoss.svg)](https://www.npmjs.com/package/@lenml/tokenizer-gptoss) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-gptoss)                                              |\n| `minicpm_v4_5`          | minicpm_v4_5                                                               | [@lenml/tokenizer-minicpm_v4_5](./packages/minicpm_v4_5)                   | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-minicpm_v4_5.svg)](https://www.npmjs.com/package/@lenml/tokenizer-minicpm_v4_5) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-minicpm_v4_5)                            |\n| `gemma3`                | Gemma3                                                                     | [@lenml/tokenizer-gemma3](./packages/gemma3)                               | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-gemma3.svg)](https://www.npmjs.com/package/@lenml/tokenizer-gemma3) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-gemma3)                                              |\n| `deepseek_v3`           | DeepSeek-V3 / DeepSeek-R1                                                  | [@lenml/tokenizer-deepseek_v3](./packages/deepseek_v3)                     | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-deepseek_v3.svg)](https://www.npmjs.com/package/@lenml/tokenizer-deepseek_v3) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-deepseek_v3)                               |\n| `llama3_1`              | Llama 3.1                                                                  | [@lenml/tokenizer-llama3_1](./packages/llama3_1)                           | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-llama3_1.svg)](https://www.npmjs.com/package/@lenml/tokenizer-llama3_1) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-llama3_1)                                        |\n| `llama2`                | Llama 2 (mistral, zephyr, vicuna)                                          | [@lenml/tokenizer-llama2](./packages/llama2)                               | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-llama2.svg)](https://www.npmjs.com/package/@lenml/tokenizer-llama2) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-llama2)                                              |\n| `llama3`                | Llama 3                                                                    | [@lenml/tokenizer-llama3](./packages/llama3)                               | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-llama3.svg)](https://www.npmjs.com/package/@lenml/tokenizer-llama3) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-llama3)                                              |\n| `gpt4o`                 | GPT-4o                                                                     | [@lenml/tokenizer-gpt4o](./packages/gpt4o)                                 | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-gpt4o.svg)](https://www.npmjs.com/package/@lenml/tokenizer-gpt4o) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-gpt4o)                                                 |\n| `gpt4`                  | GPT-4                                                                      | [@lenml/tokenizer-gpt4](./packages/gpt4)                                   | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-gpt4.svg)](https://www.npmjs.com/package/@lenml/tokenizer-gpt4) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-gpt4)                                                    |\n| `gpt35turbo`            | GPT-3.5 Turbo                                                              | [@lenml/tokenizer-gpt35turbo](./packages/gpt35turbo)                       | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-gpt35turbo.svg)](https://www.npmjs.com/package/@lenml/tokenizer-gpt35turbo) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-gpt35turbo)                                  |\n| `gpt35turbo16k`         | GPT-3.5 Turbo 16k                                                          | [@lenml/tokenizer-gpt35turbo16k](./packages/gpt35turbo16k)                 | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-gpt35turbo16k.svg)](https://www.npmjs.com/package/@lenml/tokenizer-gpt35turbo16k) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-gpt35turbo16k)                         |\n| `gpt3`                  | GPT-3                                                                      | [@lenml/tokenizer-gpt3](./packages/gpt3)                                   | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-gpt3.svg)](https://www.npmjs.com/package/@lenml/tokenizer-gpt3) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-gpt3)                                                    |\n| `gemma`                 | Gemma                                                                      | [@lenml/tokenizer-gemma](./packages/gemma)                                 | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-gemma.svg)](https://www.npmjs.com/package/@lenml/tokenizer-gemma) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-gemma)                                                 |\n| `claude`                | Claude 2/2.1/2.5/3/3.5                                                     | [@lenml/tokenizer-claude](./packages/claude)                               | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-claude.svg)](https://www.npmjs.com/package/@lenml/tokenizer-claude) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-claude)                                              |\n| `claude1`               | Claude 1                                                                   | [@lenml/tokenizer-claude1](./packages/claude1)                             | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-claude1.svg)](https://www.npmjs.com/package/@lenml/tokenizer-claude1) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-claude1)                                           |\n| `gpt2`                  | GPT-2                                                                      | [@lenml/tokenizer-gpt2](./packages/gpt2)                                   | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-gpt2.svg)](https://www.npmjs.com/package/@lenml/tokenizer-gpt2) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-gpt2)                                                    |\n| `baichuan2`             | Baichuan 2                                                                 | [@lenml/tokenizer-baichuan2](./packages/baichuan2)                         | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-baichuan2.svg)](https://www.npmjs.com/package/@lenml/tokenizer-baichuan2) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-baichuan2)                                     |\n| `chatglm3`              | ChatGLM 3                                                                  | [@lenml/tokenizer-chatglm3](./packages/chatglm3)                           | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-chatglm3.svg)](https://www.npmjs.com/package/@lenml/tokenizer-chatglm3) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-chatglm3)                                        |\n| `command_r_plus`        | Command-R-Plus                                                             | [@lenml/tokenizer-command_r_plus](./packages/command_r_plus)               | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-command_r_plus.svg)](https://www.npmjs.com/package/@lenml/tokenizer-command_r_plus) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-command_r_plus)                      |\n| `internlm2`             | InternLM 2                                                                 | [@lenml/tokenizer-internlm2](./packages/internlm2)                         | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-internlm2.svg)](https://www.npmjs.com/package/@lenml/tokenizer-internlm2) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-internlm2)                                     |\n| `qwen1_5`               | Qwen 1.5                                                                   | [@lenml/tokenizer-qwen1_5](./packages/qwen1_5)                             | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-qwen1_5.svg)](https://www.npmjs.com/package/@lenml/tokenizer-qwen1_5) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-qwen1_5)                                           |\n| `yi`                    | Yi                                                                         | [@lenml/tokenizer-yi](./packages/yi)                                       | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-yi.svg)](https://www.npmjs.com/package/@lenml/tokenizer-yi) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-yi)                                                          |\n| `text_davinci002`       | Text-Davinci-002                                                           | [@lenml/tokenizer-text_davinci002](./packages/text_davinci002)             | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-text_davinci002.svg)](https://www.npmjs.com/package/@lenml/tokenizer-text_davinci002) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-text_davinci002)                   |\n| `text_davinci003`       | Text-Davinci-003                                                           | [@lenml/tokenizer-text_davinci003](./packages/text_davinci003)             | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-text_davinci003.svg)](https://www.npmjs.com/package/@lenml/tokenizer-text_davinci003) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-text_davinci003)                   |\n| `text_embedding_ada002` | Text-Embedding-Ada-002                                                     | [@lenml/tokenizer-text_embedding_ada002](./packages/text_embedding_ada002) | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-text_embedding_ada002.svg)](https://www.npmjs.com/package/@lenml/tokenizer-text_embedding_ada002) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-text_embedding_ada002) |\n| `gemma2`                | Gemma 2 / gemini-1.0-pro / gemini-1.5-pro / gemini-1.5-flash / gemini-nano | [@lenml/tokenizer-gemma2](./packages/gemma2)                               | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-gemma2.svg)](https://www.npmjs.com/package/@lenml/tokenizer-gemma2) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-gemma2)                                              |\n| `gemini`                | Gemma 2 / gemini-1.0-pro / gemini-1.5-pro / gemini-1.5-flash / gemini-nano | [@lenml/tokenizer-gemini](./packages/gemini)                               | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-gemini.svg)](https://www.npmjs.com/package/@lenml/tokenizer-gemini) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-gemini)                                              |\n| `qwen2_5`               | Qwen 2.5                                                                   | [@lenml/tokenizer-qwen2_5](./packages/qwen2_5)                             | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-qwen2_5.svg)](https://www.npmjs.com/package/@lenml/tokenizer-qwen2_5) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-qwen2_5)                                           |\n| `aya_expanse`           | Aya Expanse                                                                | [@lenml/tokenizer-aya_expanse](./packages/aya_expanse)                     | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-aya_expanse.svg)](https://www.npmjs.com/package/@lenml/tokenizer-aya_expanse) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-aya_expanse)                               |\n| `llama3_2`              | Llama 3.2                                                                  | [@lenml/tokenizer-llama3_2](./packages/llama3_2)                           | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-llama3_2.svg)](https://www.npmjs.com/package/@lenml/tokenizer-llama3_2) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-llama3_2)                                        |\n| `mistral_nemo`          | Mistral Nemo                                                               | [@lenml/tokenizer-mistral_nemo](./packages/mistral_nemo)                   | [![npm](https://img.shields.io/npm/v/@lenml/tokenizer-mistral_nemo.svg)](https://www.npmjs.com/package/@lenml/tokenizer-mistral_nemo) ![NPM Downloads](https://img.shields.io/npm/dw/%40lenml%2Ftokenizer-mistral_nemo)                            |\n\nIn addition to the pre-packaged models listed above, you can also utilize the interfaces in @lenml/tokenizers to load models independently.\n\n# Usage\n\n## install\n\n### npm/yarn/pnpm\n\n```\nnpm install @lenml/tokenizers\n```\n\n### ESM\n\n```html\n\u003cscript type=\"importmap\"\u003e\n  {\n    \"imports\": {\n      \"@lenml/tokenizers\": \"https://www.unpkg.com/@lenml/tokenizers@latest/dist/main.mjs\"\n    }\n  }\n\u003c/script\u003e\n\u003cscript type=\"module\"\u003e\n  import { TokenizerLoader, tokenizers } from \"@lenml/tokenizers\";\n  console.log(\"@lenml/tokenizers: \", tokenizers);\n\u003c/script\u003e\n```\n\n## load tokenizer\n\n### from json\n\n```ts\nimport { TokenizerLoader } from \"@lenml/tokenizers\";\nconst tokenizer = TokenizerLoader.fromPreTrained({\n  tokenizerJSON: {\n    /* ... */\n  },\n  tokenizerConfig: {\n    /* ... */\n  },\n});\n```\n\n### from urls\n\n```ts\nimport { TokenizerLoader } from \"@lenml/tokenizers\";\nconst sourceUrls = {\n  tokenizerJSON:\n    \"https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1/resolve/main/tokenizer.json?download=true\",\n  tokenizerConfig:\n    \"https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1/resolve/main/tokenizer_config.json?download=true\",\n};\nconst tokenizer = await TokenizerLoader.fromPreTrainedUrls(sourceUrls);\n// or from fetch\nconst tokenizer = TokenizerLoader.fromPreTrained({\n  tokenizerJSON: await fetch(sourceUrls.tokenizerJSON).then((r) =\u003e r.json()),\n  tokenizerConfig: await fetch(sourceUrls.tokenizerConfig).then((r) =\u003e\n    r.json()\n  ),\n});\n```\n\n### from pre-packaged tokenizer\n\n```ts\nimport { fromPreTrained } from \"@lenml/tokenizer-llama3\";\nconst tokenizer = fromPreTrained();\n```\n\n## chat template\n\n```ts\nconst tokens = tokenizer.apply_chat_template([\n  {\n    role: \"system\",\n    content: \"You are helpful assistant.\",\n  },\n  {\n    role: \"user\",\n    content: \"Hello, how are you?\",\n  },\n]) as number[];\n\nconst chat_content = tokenizer.decode(tokens);\n\nconsole.log(chat_content);\n```\n\noutput:\n\n```\n\u003c|begin_of_text|\u003e\u003c|start_header_id|\u003esystem\u003c|end_header_id|\u003e\n\nYou are helpful assistant.\u003c|eot_id|\u003e\u003c|start_header_id|\u003euser\u003c|end_header_id|\u003e\n\nHello, how are you?\u003c|eot_id|\u003e\u003c|start_header_id|\u003eassistant\u003c|end_header_id|\u003e\n```\n\n## tokenizer api\n\n```ts\nconsole.log(\n  \"encode() =\u003e \",\n  tokenizer.encode(\"Hello, my dog is cute\", null, {\n    add_special_tokens: true,\n  })\n);\nconsole.log(\n  \"_encode_text() =\u003e \",\n  tokenizer._encode_text(\"Hello, my dog is cute\")\n);\n```\n\n\u003e fully tokenizer api: [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)\n\n## get lightweight `transformers.tokenizers`\n\nIn the `@lenml/tokenizers` package, you can get a lightweight no-dependency implementation of tokenizers:\n\n\u003e Since all dependencies related to huggingface have been removed in this library, although the implementation is the same, it is not possible to load models using the form `hf_user/repo`.\n\n```ts\nimport { tokenizers } from \"@lenml/tokenizers\";\n\nconst {\n  CLIPTokenizer,\n  AutoTokenizer,\n  CohereTokenizer,\n  VitsTokenizer,\n  WhisperTokenizer,\n  // ...\n} = tokenizers;\n```\n\n## Manual Packaging\n\nIn some cases, you may need to use an older version of Node.js, so you might not be able to use pre-packaged packages. In such situations, you can manually package starting from the .ts files.\n\nHere's a simple example:\n\n```ts\nimport {\n  tokenizerJSON,\n  tokenizerConfig,\n} from \"@lenml/tokenizer-claude/src/data.ts\";\nimport { TokenizerLoader } from \"@lenml/tokenizers/src/main.ts\";\n\nexport const tokenizer = TokenizerLoader.fromPreTrained({\n  tokenizerConfig,\n  tokenizerJSON,\n});\n```\n\n## Performance Benchmark Results\n\nThe following table summarizes the performance benchmarks for the `Llama31` and `GPT4o` tokenizers across various datasets. The performance is measured in operations per second (ops/sec), indicating how efficiently each tokenizer processes the given input.\n\n| Tokenizer | Operation | Text    | Performance (ops/sec) | Margin of Error (±%) | Sampled Runs |\n| --------- | --------- | ------- | --------------------- | -------------------- | ------------ |\n| Llama31   | encode    | English | 27,260                | 0.81%                | 86           |\n| Llama31   | encode    | Chinese | 50,675                | 0.81%                | 89           |\n| Llama31   | encode    | French  | 22,836                | 0.58%                | 92           |\n| Llama31   | encode    | Code    | 17,677                | 0.30%                | 94           |\n| Llama31   | decode    | English | 16,542                | 0.61%                | 90           |\n| Llama31   | decode    | Chinese | 21,118                | 0.39%                | 90           |\n| Llama31   | decode    | French  | 12,994                | 0.24%                | 91           |\n| Llama31   | decode    | Code    | 10,350                | 2.80%                | 87           |\n| GPT4o     | encode    | English | 31,618                | 0.74%                | 92           |\n| GPT4o     | encode    | Chinese | 73,120                | 0.74%                | 92           |\n| GPT4o     | encode    | French  | 27,838                | 3.40%                | 91           |\n| GPT4o     | encode    | Code    | 19,589                | 3.05%                | 87           |\n| GPT4o     | decode    | English | 24,723                | 0.73%                | 91           |\n| GPT4o     | decode    | Chinese | 44,201                | 0.33%                | 92           |\n| GPT4o     | decode    | French  | 21,924                | 0.39%                | 90           |\n| GPT4o     | decode    | Code    | 15,785                | 0.55%                | 94           |\n\nThe benchmarking script used to generate these results can be found at `./packages/tests/benchmarks/main.ts`. You can use this script to replicate the benchmarks and validate the performance metrics for yourself.\n\n# License\n\nApache-2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flenml%2Ftokenizers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flenml%2Ftokenizers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flenml%2Ftokenizers/lists"}