{"id":25441950,"url":"https://github.com/openvinotoolkit/openvino_tokenizers","last_synced_at":"2025-04-05T10:06:55.855Z","repository":{"id":222320886,"uuid":"751427608","full_name":"openvinotoolkit/openvino_tokenizers","owner":"openvinotoolkit","description":"OpenVINO Tokenizers extension","archived":false,"fork":false,"pushed_at":"2025-04-03T08:08:16.000Z","size":256580,"stargazers_count":31,"open_issues_count":7,"forks_count":36,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-04-03T09:23:26.102Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openvinotoolkit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-01T15:38:57.000Z","updated_at":"2025-04-03T08:08:12.000Z","dependencies_parsed_at":"2024-03-12T14:02:38.287Z","dependency_job_id":"36af7430-16d0-4b50-93d5-5a6641d1c6ce","html_url":"https://github.com/openvinotoolkit/openvino_tokenizers","commit_stats":null,"previous_names":["openvinotoolkit/openvino_tokenizers"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fopenvino_tokenizers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fopenvino_tokenizers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fopenvino_tokenizers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fopenvino_tokenizers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openvinotoolkit","download_url":"https://codeload.github.com/openvinotoolkit/openvino_tokenizers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247318744,"owners_count":20919484,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-17T13:16:07.288Z","updated_at":"2025-04-05T10:06:55.839Z","avatar_url":"https://github.com/openvinotoolkit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenVINO Tokenizers\n\n[![Downloads](https://static.pepy.tech/badge/openvino-tokenizers)](https://pepy.tech/project/openvino-tokenizers)\n[![Anaconda-Server Badge](https://anaconda.org/conda-forge/openvino-tokenizers/badges/downloads.svg)](https://anaconda.org/conda-forge/openvino-tokenizers)\n\nOpenVINO Tokenizers adds text processing operations to OpenVINO.\n\n## Features\n\n- Perform tokenization and detokenization without third-party dependencies\n- Convert a HuggingFace tokenizer into OpenVINO model tokenizer and detokenizer\n- Combine OpenVINO models into a single model\n- Add greedy decoding pipeline to text generation model\n\n## Installation\n\n(Recommended) Create and activate virtual env:\n```bash\npython3 -m venv venv\nsource venv/bin/activate\n # or\nconda create --name openvino_tokenizers\nconda activate openvino_tokenizers\n```\n\n### Minimal Installation\n\nUse minimal installation when you have a converted OpenVINO tokenizer:\n```bash\npip install openvino-tokenizers\n # or\nconda install -c conda-forge openvino openvino-tokenizers\n```\n\n### Convert Tokenizers Installation\n\nIf you want to convert HuggingFace tokenizers into OpenVINO tokenizers:\n```bash\npip install openvino-tokenizers[transformers]\n # or\nconda install -c conda-forge openvino openvino-tokenizers \u0026\u0026 pip install transformers[sentencepiece] tiktoken\n```\n\n### Install Pre-release Version\n\nUse `openvino-tokenizers[transformers]` to install tokenizers conversion dependencies.\n```bash\npip install --pre -U openvino openvino-tokenizers --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly\n```\n\n### Build and Install from Source\n\n#### Using OpenVINO PyPI package\n\nopenvino-tokenizers build depends on [openvino](https://pypi.org/project/openvino/) package which will be automatically installed from PyPI during the build process. To install unreleased versions, you would need to install openvino package from the nightly distribution channel using `--extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly`\n\n```bash\ngit clone https://github.com/openvinotoolkit/openvino_tokenizers.git\ncd openvino_tokenizers\npip install . --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly\n```\nThis command is the equivalent of minimal installation. Install tokenizers conversion dependencies if needed:\n```bash\npip install transformers[sentencepiece] tiktoken\n```\n:warning: Latest commit of OpenVINO Tokenizers might rely on features that are not present in the release OpenVINO version.\nUse [a nightly build](https://docs.openvino.ai/2024/get-started/install-openvino.html?VERSION=NIGHTLY) of OpenVINO or build\nOpenVINO Tokenizers from a release branch if you have issues with the build process.\n\n#### Using OpenVINO archive\n\nInstall [OpenVINO archive](https://docs.openvino.ai/2024/get-started/install-openvino.html) distribution. Use `--no-deps` to avoid OpenVINO installation from PyPI into your current environment.\n`--extra-index-url` is needed to resolve build dependencies only.\n\n```bash\nsource path/to/installed/openvino/setupvars.sh\ngit clone https://github.com/openvinotoolkit/openvino_tokenizers.git\ncd openvino_tokenizers\npip install --no-deps . --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly\n```\nThis command is the equivalent of minimal installation. Install tokenizers conversion dependencies if needed:\n```bash\npip install transformers[sentencepiece] tiktoken\n```\n:warning: Latest commit of OpenVINO Tokenizers might rely on features that are not present in the release OpenVINO version.\nUse [a nightly build](https://docs.openvino.ai/2024/get-started/install-openvino.html?VERSION=NIGHTLY) of OpenVINO or build\nOpenVINO Tokenizers from a release branch if you have issues with the build process.\n\n### Build and install for development\n\n#### Using OpenVINO PyPI package\n\n```bash\ngit clone https://github.com/openvinotoolkit/openvino_tokenizers.git\ncd openvino_tokenizers\npip install -e .[all] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly\n# verify installation by running tests\ncd tests/\npytest .\n```\n\n#### Using OpenVINO archive\n\nInstall [OpenVINO archive](https://docs.openvino.ai/2024/get-started/install-openvino.html) distribution. Use `--no-deps` to avoid OpenVINO installation from PyPI into your current environment.\n`--extra-index-url` is needed to resolve build dependencies only.\n\n```bash\nsource path/to/installed/openvino/setupvars.sh\ngit clone https://github.com/openvinotoolkit/openvino_tokenizers.git\ncd openvino_tokenizers\npip install -e .[all] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly\n# verify installation by running tests\ncd tests/\npytest .\n```\n\n### C++ Installation\n\nYou can use converted tokenizers in C++ pipelines with prebuild binaries.\n\n1. Download OpenVINO archive distribution for your OS from [here](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/download.html) and extract the archive.\n2. Download OpenVINO Tokenizers prebuild libraries from [here](https://storage.openvinotoolkit.org/repositories/openvino_tokenizers/packages/). To ensure compatibility first three numbers of OpenVINO Tokenizers version should match OpenVINO version and OS.\n3. Extract OpenVINO Tokenizers archive into OpenVINO installation directory. OpenVINO Tokenizers archive maintains the structure to be aligned with OpenVINO archive:\n    - Windows: `\u003copenvino_dir\u003e\\runtime\\bin\\intel64\\Release\\`\n    - MacOS_x86: `\u003copenvino_dir\u003e/runtime/lib/intel64/Release`\n    - MacOS_arm64: `\u003copenvino_dir\u003e/runtime/lib/arm64/Release/`\n    - Linux_x86: `\u003copenvino_dir\u003e/runtime/lib/intel64/`\n    - Linux_arm64: `\u003copenvino_dir\u003e/runtime/lib/aarch64/`\n\nAfter that you can add binary extension in the code with:\n- `core.add_extension(\"openvino_tokenizers.dll\")` for Windows\n- `core.add_extension(\"libopenvino_tokenizers.dylib\")` for MacOS\n- `core.add_extension(\"libopenvino_tokenizers.so\")` for Linux\n\nand `read`/`compile` converted (de)tokenizers models.\nIf you use version `2023.3.0.0`, the binary extension file is called `(lib)user_ov_extension.(dll/dylib/so)`.\n\n### C++ Build\n\nTo build OpenVINO Tokenizers binaries locally, use this command:\n\n```bash\nsource path/to/installed/openvino/setupvars.sh\ngit clone https://github.com/openvinotoolkit/openvino_tokenizers.git\ncd openvino_tokenizers\nmkdir build \u0026\u0026 cd build\ncmake -DCMAKE_BUILD_TYPE=Release ..\nmake\n```\n\nAfter that, you can transfer all binaries from `build/src` to `\u003copenvino_dir\u003e` as described in the C++ installation instruction above.\n\n## Usage\n\n:warning: OpenVINO Tokenizers can be inferred on a `CPU` device only.\n\n### Convert HuggingFace tokenizer\n\nOpenVINO Tokenizers ships with CLI tool that can convert tokenizers from Huggingface Hub\nor Huggingface tokenizers saved on disk:\n\n```shell\nconvert_tokenizer codellama/CodeLlama-7b-hf --with-detokenizer -o output_dir\n```\n\nThere is also `convert_tokenizer` function that can convert tokenizer python object.\n\n```python\nimport numpy as np\nfrom transformers import AutoTokenizer\nfrom openvino import compile_model, save_model\nfrom openvino_tokenizers import convert_tokenizer\n\nhf_tokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\nov_tokenizer = convert_tokenizer(hf_tokenizer)\n\ncompiled_tokenzier = compile_model(ov_tokenizer)\ntext_input = [\"Test string\"]\n\nhf_output = hf_tokenizer(text_input, return_tensors=\"np\")\nov_output = compiled_tokenzier(text_input)\n\nfor output_name in hf_output:\n    print(f\"OpenVINO {output_name} = {ov_output[output_name]}\")\n    print(f\"HuggingFace {output_name} = {hf_output[output_name]}\")\n# OpenVINO input_ids = [[ 101 3231 5164  102]]\n# HuggingFace input_ids = [[ 101 3231 5164  102]]\n# OpenVINO token_type_ids = [[0 0 0 0]]\n# HuggingFace token_type_ids = [[0 0 0 0]]\n# OpenVINO attention_mask = [[1 1 1 1]]\n# HuggingFace attention_mask = [[1 1 1 1]]\n\n# save tokenizer for later use\nsave_model(ov_tokenizer, \"openvino_tokenizer.xml\")\n\nloaded_tokenizer = compile_model(\"openvino_tokenizer.xml\")\nloaded_ov_output = loaded_tokenizer(text_input)\nfor output_name in hf_output:\n    assert np.all(loaded_ov_output[output_name] == ov_output[output_name])\n```\n\n### Connect Tokenizer to a Model\n\nTo infer and convert the original model, install torch or torch-cpu to the virtual environment.\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\nfrom openvino import compile_model, convert_model\nfrom openvino_tokenizers import convert_tokenizer, connect_models\n\ncheckpoint = \"mrm8488/bert-tiny-finetuned-sms-spam-detection\"\nhf_tokenizer = AutoTokenizer.from_pretrained(checkpoint)\nhf_model = AutoModelForSequenceClassification.from_pretrained(checkpoint)\n\ntext_input = [\"Free money!!!\"]\nhf_input = hf_tokenizer(text_input, return_tensors=\"pt\")\nhf_output = hf_model(**hf_input)\n\nov_tokenizer = convert_tokenizer(hf_tokenizer)\nov_model = convert_model(hf_model, example_input=hf_input.data)\ncombined_model = connect_models(ov_tokenizer, ov_model)\ncompiled_combined_model = compile_model(combined_model)\n\nopenvino_output = compiled_combined_model(text_input)\n\nprint(f\"OpenVINO logits: {openvino_output['logits']}\")\n# OpenVINO logits: [[ 1.2007061 -1.4698029]]\nprint(f\"HuggingFace logits {hf_output.logits}\")\n# HuggingFace logits tensor([[ 1.2007, -1.4698]], grad_fn=\u003cAddmmBackward0\u003e)\n```\n\n### Use Extension With Converted (De)Tokenizer or Model With (De)Tokenizer\n\nImport `openvino_tokenizers` will register tokenizer-related operations to OpenVINO,\nafter which you can work with saved tokenizers and detokenizers.\n\n```python\nimport numpy as np\nimport openvino_tokenizers\nfrom openvino import Core\n\ncore = Core()\n\n# detokenizer from codellama sentencepiece model\ncompiled_detokenizer = core.compile_model(\"detokenizer.xml\")\n\ntoken_ids = np.random.randint(100, 1000, size=(3, 5))\nopenvino_output = compiled_detokenizer(token_ids)\n\nprint(openvino_output[\"string_output\"])\n# ['sc�ouition�', 'intvenord hasient', 'g shouldwer M more']\n```\n\n### Text Generation Pipeline\n\n```python\nimport numpy as np\nfrom openvino import compile_model, convert_model\nfrom openvino_tokenizers import add_greedy_decoding, convert_tokenizer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n\nmodel_checkpoint = \"JackFram/llama-68m\"\nhf_tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)\nhf_model = AutoModelForCausalLM.from_pretrained(model_checkpoint, use_cache=False)\n\n# convert hf tokenizer\ntext_input = [\"Quick brown fox jumped \"]\nov_tokenizer, ov_detokenizer = convert_tokenizer(hf_tokenizer, with_detokenizer=True)\ncompiled_tokenizer = compile_model(ov_tokenizer)\n\n# transform input text into tokens\nov_input = compiled_tokenizer(text_input)\nhf_input = hf_tokenizer(text_input, return_tensors=\"pt\")\n\n# convert Pytorch model to OpenVINO IR and add greedy decoding pipeline to it\nov_model = convert_model(hf_model, example_input=hf_input.data)\nov_model_with_greedy_decoding = add_greedy_decoding(ov_model)\ncompiled_model = compile_model(ov_model_with_greedy_decoding)\n\n# generate new tokens\nnew_tokens_size = 10\nprompt_size = ov_input[\"input_ids\"].shape[-1]\ninput_dict = {\n    output.any_name: np.hstack([tensor, np.zeros(shape=(1, new_tokens_size), dtype=np.int_)])\n    for output, tensor in ov_input.items()\n}\nfor idx in range(prompt_size, prompt_size + new_tokens_size):\n    output = compiled_model(input_dict)[\"token_ids\"]\n    input_dict[\"input_ids\"][:, idx] = output[:, idx - 1]\n    input_dict[\"attention_mask\"][:, idx] = 1\nov_token_ids = input_dict[\"input_ids\"]\n\nhf_token_ids = hf_model.generate(\n    **hf_input,\n    min_new_tokens=new_tokens_size,\n    max_new_tokens=new_tokens_size,\n    temperature=0,  # greedy decoding\n)\n\n# decode model output\ncompiled_detokenizer = compile_model(ov_detokenizer)\nov_output = compiled_detokenizer(ov_token_ids)[\"string_output\"]\nhf_output = hf_tokenizer.batch_decode(hf_token_ids, skip_special_tokens=True)\nprint(f\"OpenVINO output string: `{ov_output}`\")\n# OpenVINO output string: `['Quick brown fox was walking through the forest. He was looking for something']`\nprint(f\"HuggingFace output string: `{hf_output}`\")\n# HuggingFace output string: `['Quick brown fox was walking through the forest. He was looking for something']`\n```\n\n### TensorFlow Text Integration\n\nOpenVINO Tokenizers include converters for certain TensorFlow Text operations.\nCurrently, only the MUSE model is supported.\nHere is an example of model conversion and inference:\n\n```python\nimport numpy as np\nimport tensorflow_hub as hub\nimport tensorflow_text  # register tf text ops\nfrom openvino import convert_model, compile_model\nimport openvino_tokenizers  # register ov tokenizer ops and translators\n\n\nsentences = [\"dog\",  \"I cuccioli sono carini.\", \"私は犬と一緒にビーチを散歩するのが好きです\"]\ntf_embed = hub.load(\n    \"https://www.kaggle.com/models/google/universal-sentence-encoder/frameworks/\"\n    \"TensorFlow2/variations/multilingual/versions/2\"\n)\n# convert model that uses Sentencepiece tokenizer op from TF Text\nov_model = convert_model(tf_embed)\nov_embed = compile_model(ov_model, \"CPU\")\n\nov_result = ov_embed(sentences)[ov_embed.output()]\ntf_result = tf_embed(sentences)\n\nassert np.all(np.isclose(ov_result, tf_result, atol=1e-4))\n```\n\n### RWKV Tokenizer\n\n```python\nfrom urllib.request import urlopen\n\nfrom openvino import compile_model\nfrom openvino_tokenizers import build_rwkv_tokenizer\n\n\nrwkv_vocab_url = (\n    \"https://raw.githubusercontent.com/BlinkDL/ChatRWKV/main/tokenizer/rwkv_vocab_v20230424.txt\"\n)\n\nwith urlopen(rwkv_vocab_url) as vocab_file:\n    vocab = map(bytes.decode, vocab_file)\n    tokenizer, detokenizer = build_rwkv_tokenizer(vocab)\n\ntokenizer, detokenizer = compile_model(tokenizer), compile_model(detokenizer)\n\nprint(tokenized := tokenizer([\"Test string\"])[\"input_ids\"])  # [[24235 47429]]\nprint(detokenizer(tokenized)[\"string_output\"])  # ['Test string']\n```\n\n### Tokenizer From GGUF Model \n\n```python\nfrom transformers import AutoTokenizer\nimport openvino as ov\nfrom openvino_tokenizers import convert_tokenizer\n\n\nmodel_id = \"unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF\"\nfilename = \"DeepSeek-R1-Distill-Qwen-1.5B-Q2_K.gguf\"\nhf_tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)\n\nov_tokenizer, ov_detokenizer = convert_tokenizer(hf_tokenizer, with_detokenizer=True)\nov_tokenizer, ov_detokenizer = ov.compile_model(ov_tokenizer), ov.compile_model(ov_detokenizer)\n\nprint(ov_res := ov_tokenizer([\"Test string\"])[\"input_ids\"])  # [[2271  914]]\nprint(ov_detokenizer(ov_res)[\"string_output\"])  # ['Test string']\n```\n\n### C++ Usage Example\n\nThis example shows how to run inference with C++ on a text-classification model from Hugging Face. It\nexpects the path to a model directory as parameter, and prints the logits returned by the model inference.\n\nExport an example model by running the following command after `pip install optimum[openvino]`:\n\n```sh\noptimum-cli export openvino microsoft/deberta-base-mnli deberta-base-mnli-ov\n```\n\n```cpp\n#include \u003copenvino/openvino.hpp\u003e\n#include \u003ciostream\u003e\n#include \u003cfilesystem\u003e\n\nint main(int argc, char* argv[]) {\n   std::string dirname = argv[1];\n   std::filesystem::path dir_path(dirname);\n   std::filesystem::path model_xml = dir_path / \"openvino_model.xml\";\n   std::filesystem::path tokenizer_xml = dir_path / \"openvino_tokenizer.xml\";\n\n   ov::Core core;\n   // use \"openvino_tokenizers.dll\" on Windows, \"libopenvino_tokenizers.dylib\" on macOS\n   core.add_extension(\"libopenvino_tokenizers.so\");\n\n   ov::InferRequest tokenizer_request = core.compile_model(tokenizer_xml, \"CPU\").create_infer_request();\n\n   std::string prompt=\"Hello world!\";\n   tokenizer_request.set_input_tensor(ov::Tensor{ov::element::string, {1}, \u0026prompt});\n   tokenizer_request.infer();\n   ov::Tensor input_ids = tokenizer_request.get_tensor(\"input_ids\");\n   ov::Tensor attention_mask = tokenizer_request.get_tensor(\"attention_mask\");\n\n   ov::InferRequest infer_request = core.compile_model(model_xml, \"CPU\").create_infer_request();\n   infer_request.set_tensor(\"input_ids\", input_ids);\n   infer_request.set_tensor(\"attention_mask\", attention_mask);\n   infer_request.infer();\n\n   auto output = infer_request.get_tensor(\"logits\");\n   const float *output_buffer = output.data\u003cconst float\u003e();\n\n   size_t num_elements = output.get_size();\n\n   for (size_t i = 0; i \u003c num_elements; i++) {\n       std::cout \u003c\u003c output_buffer[i] \u003c\u003c \" \";\n   }\n\n   std::cout \u003c\u003c std::endl;\n   return 0;\n}\n```\n\n### Unicode Support\n\n- OpenVINO Tokenizers support UTF-8 encoded inputs. \n- Internal tokenizer vocabulary is stored in UTF-8 encoding:\n  - Providing a tokenizer model with  non-UTF-8 input may lead to unexpected outputs or errors,\n  - Detokenizer output is UTF-8 encoded; if your terminal does not expect UTF-8, you might see garbage characters.\n- By default, a detokenizer replaces invalid UTF-8 output with � character. You can change this behavior during conversion.\n\n## Supported Tokenizer Types\n\n| Huggingface \u003cbr/\u003eTokenizer Type | Tokenizer Model Type | Tokenizer | Detokenizer |\n|---------------------------------|----------------------|----------|-----------|\n| Fast                            | WordPiece            | ✅        | ✅          |\n|                                 | BPE                  | ✅        | ✅         |\n|                                 | Unigram              | ✅         | ✅         |\n|                                 | WordLevel*           | ✅         | ✅         |\n| Legacy                          | SentencePiece .model | ✅        | ✅         |\n| Custom                          | tiktoken             | ✅        | ✅         |\n| RWKV                            | Trie                 | ✅        | ✅         |\n\n## Test Results\n\nThis report is autogenerated and includes tokenizers and detokenizers tests. The `Output Matched, %` column shows the percent of test strings for which the results of OpenVINO and Huggingface Tokenizers are the same. To update the report run `pytest --update_readme tokenizers_test.py` in `tests` directory.\n\n### Output Match by Tokenizer Type\n\n\u003ctable\u003e\n  \u003cthead\u003e\n    \u003ctr\u003e\n      \u003cth \u003eTokenizer Type\u003c/th\u003e\n      \u003cth \u003eOutput Matched, %\u003c/th\u003e\n      \u003cth \u003eNumber of Tests\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003e99.28\u003c/td\u003e\n      \u003ctd \u003e5827\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003e89.82\u003c/td\u003e\n      \u003ctd \u003e5157\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eTiktoken\u003c/td\u003e\n      \u003ctd \u003e96.56\u003c/td\u003e\n      \u003ctd \u003e524\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eUnigram\u003c/td\u003e\n      \u003ctd \u003e95.24\u003c/td\u003e\n      \u003ctd \u003e1470\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eWordLevel\u003c/td\u003e\n      \u003ctd \u003e98.96\u003c/td\u003e\n      \u003ctd \u003e192\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eWordPiece\u003c/td\u003e\n      \u003ctd \u003e99.07\u003c/td\u003e\n      \u003ctd \u003e1289\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n### Output Match by Model\n\n\u003ctable\u003e\n  \u003cthead\u003e\n    \u003ctr\u003e\n      \u003cth \u003eTokenizer Type\u003c/th\u003e\n      \u003cth \u003eModel\u003c/th\u003e\n      \u003cth \u003eOutput Matched, %\u003c/th\u003e\n      \u003cth \u003eNumber of Tests\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003eNousResearch/Llama-2-13b-hf\u003c/td\u003e\n      \u003ctd \u003e97.55\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003eNousResearch/Meta-Llama-3-8B-Instruct\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e247\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003eSalesforce/codegen-16B-multi\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003eTinyLlama/TinyLlama-1.1B-Chat-v1.0\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e247\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003eXenova/gpt-4o\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003eai-forever/rugpt3large_based_on_gpt2\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003eallenai/OLMo-1B-hf\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003eanswerdotai/ModernBERT-base\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003ebigscience/bloom\u003c/td\u003e\n      \u003ctd \u003e97.55\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003edeepseek-ai/DeepSeek-V3-0324\u003c/td\u003e\n      \u003ctd \u003e99.24\u003c/td\u003e\n      \u003ctd \u003e263\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003edeepseek-ai/deepseek-coder-6.7b-instruct\u003c/td\u003e\n      \u003ctd \u003e99.24\u003c/td\u003e\n      \u003ctd \u003e263\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003efacebook/galactica-120b\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003egpt2\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003ekoalajun/Gemma-2-9b-it-Ko-Crypto-Translate\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e247\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003elaion/CLIP-ViT-bigG-14-laion2B-39B-b160k\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003emicrosoft/Phi-3-mini-128k-instruct\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e247\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003emicrosoft/deberta-base\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003emlx-community/quantized-gemma-7b-it\u003c/td\u003e\n      \u003ctd \u003e97.57\u003c/td\u003e\n      \u003ctd \u003e247\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003eroberta-base\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003estabilityai/stablecode-completion-alpha-3b-4k\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003estabilityai/stablelm-2-1_6b\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003etiiuae/Falcon3-7B-Instruct\u003c/td\u003e\n      \u003ctd \u003e96.20\u003c/td\u003e\n      \u003ctd \u003e263\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eBPE\u003c/td\u003e\n      \u003ctd \u003etiiuae/falcon-7b\u003c/td\u003e\n      \u003ctd \u003e96.17\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003eBAAI/bge-reranker-v2-m3\u003c/td\u003e\n      \u003ctd \u003e96.73\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003eBAAI/bge-reranker-v2-m3_legacy\u003c/td\u003e\n      \u003ctd \u003e96.73\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003eNousResearch/Llama-2-13b-hf\u003c/td\u003e\n      \u003ctd \u003e94.29\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003eNousResearch/Llama-2-13b-hf_legacy\u003c/td\u003e\n      \u003ctd \u003e97.55\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003eTinyLlama/TinyLlama-1.1B-Chat-v1.0\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e247\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003eTinyLlama/TinyLlama-1.1B-Chat-v1.0_legacy\u003c/td\u003e\n      \u003ctd \u003e98.38\u003c/td\u003e\n      \u003ctd \u003e247\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003ebaichuan-inc/Baichuan2-7B-Chat_legacy\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003ecamembert-base\u003c/td\u003e\n      \u003ctd \u003e55.10\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003ecamembert-base_legacy\u003c/td\u003e\n      \u003ctd \u003e78.37\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003efacebook/musicgen-small\u003c/td\u003e\n      \u003ctd \u003e82.45\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003efacebook/musicgen-small_legacy\u003c/td\u003e\n      \u003ctd \u003e77.14\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003egoogle/flan-t5-xxl\u003c/td\u003e\n      \u003ctd \u003e75.92\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003egoogle/flan-t5-xxl_legacy\u003c/td\u003e\n      \u003ctd \u003e75.51\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003emicrosoft/Phi-3-mini-128k-instruct\u003c/td\u003e\n      \u003ctd \u003e99.19\u003c/td\u003e\n      \u003ctd \u003e247\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003emicrosoft/Phi-3-mini-128k-instruct_legacy\u003c/td\u003e\n      \u003ctd \u003e97.57\u003c/td\u003e\n      \u003ctd \u003e247\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003emicrosoft/deberta-v3-base\u003c/td\u003e\n      \u003ctd \u003e95.10\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003emicrosoft/deberta-v3-base_legacy\u003c/td\u003e\n      \u003ctd \u003e98.37\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003emlx-community/quantized-gemma-7b-it\u003c/td\u003e\n      \u003ctd \u003e96.76\u003c/td\u003e\n      \u003ctd \u003e247\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003emlx-community/quantized-gemma-7b-it_legacy\u003c/td\u003e\n      \u003ctd \u003e97.57\u003c/td\u003e\n      \u003ctd \u003e247\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003erinna/bilingual-gpt-neox-4b\u003c/td\u003e\n      \u003ctd \u003e83.67\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eSentencePiece\u003c/td\u003e\n      \u003ctd \u003erinna/bilingual-gpt-neox-4b_legacy\u003c/td\u003e\n      \u003ctd \u003e89.39\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eTiktoken\u003c/td\u003e\n      \u003ctd \u003eQwen/Qwen-14B-Chat\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eTiktoken\u003c/td\u003e\n      \u003ctd \u003eTHUDM/glm-4-9b-chat\u003c/td\u003e\n      \u003ctd \u003e93.16\u003c/td\u003e\n      \u003ctd \u003e263\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eUnigram\u003c/td\u003e\n      \u003ctd \u003eBAAI/bge-reranker-v2-m3\u003c/td\u003e\n      \u003ctd \u003e98.37\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eUnigram\u003c/td\u003e\n      \u003ctd \u003ecamembert-base\u003c/td\u003e\n      \u003ctd \u003e84.49\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eUnigram\u003c/td\u003e\n      \u003ctd \u003efacebook/musicgen-small\u003c/td\u003e\n      \u003ctd \u003e98.37\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eUnigram\u003c/td\u003e\n      \u003ctd \u003egoogle/flan-t5-xxl\u003c/td\u003e\n      \u003ctd \u003e91.84\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eUnigram\u003c/td\u003e\n      \u003ctd \u003emicrosoft/deberta-v3-base\u003c/td\u003e\n      \u003ctd \u003e98.37\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eUnigram\u003c/td\u003e\n      \u003ctd \u003erinna/bilingual-gpt-neox-4b\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eWordLevel\u003c/td\u003e\n      \u003ctd \u003ecisco-ai/mini-bart-g2p\u003c/td\u003e\n      \u003ctd \u003e98.96\u003c/td\u003e\n      \u003ctd \u003e192\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eWordPiece\u003c/td\u003e\n      \u003ctd \u003ebert-base-multilingual-cased\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eWordPiece\u003c/td\u003e\n      \u003ctd \u003ecointegrated/rubert-tiny2\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eWordPiece\u003c/td\u003e\n      \u003ctd \u003egoogle/mobilebert-uncased\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e245\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eWordPiece\u003c/td\u003e\n      \u003ctd \u003erasa/LaBSE\u003c/td\u003e\n      \u003ctd \u003e95.40\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd \u003eWordPiece\u003c/td\u003e\n      \u003ctd \u003esentence-transformers/all-MiniLM-L6-v2\u003c/td\u003e\n      \u003ctd \u003e100.00\u003c/td\u003e\n      \u003ctd \u003e261\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n### Recreating Tokenizers From Tests\n\nIn some tokenizers, you need to select certain settings so that their output is closer to the Huggingface tokenizers:\n- `THUDM/chatglm3-6b` detokenizer don't skips special tokens. Use `skip_special_tokens=False` during conversion\n- All tested tiktoken based detokenizers leave extra spaces. Use `clean_up_tokenization_spaces=False` during conversion\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenvinotoolkit%2Fopenvino_tokenizers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenvinotoolkit%2Fopenvino_tokenizers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenvinotoolkit%2Fopenvino_tokenizers/lists"}