{"id":51617713,"url":"https://github.com/japlete/utf-token","last_synced_at":"2026-07-12T15:30:24.369Z","repository":{"id":359349812,"uuid":"1198894146","full_name":"japlete/utf-token","owner":"japlete","description":"LLM-friendly encoding for random identifiers (hex, base64, UUID). Built for agents, RAG, and NIAH-style retrieval.","archived":false,"fork":false,"pushed_at":"2026-06-20T20:41:36.000Z","size":2778,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-20T22:14:23.721Z","etag":null,"topics":["agent-hooks","agentic-rag","ai-tools","context-engineering","niah","pypi-package","python-library","rag","token-optimization"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/utf-token/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/japlete.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-04-01T21:33:22.000Z","updated_at":"2026-06-20T20:41:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/japlete/utf-token","commit_stats":null,"previous_names":["japlete/utf-token"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/japlete/utf-token","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/japlete%2Futf-token","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/japlete%2Futf-token/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/japlete%2Futf-token/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/japlete%2Futf-token/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/japlete","download_url":"https://codeload.github.com/japlete/utf-token/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/japlete%2Futf-token/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35395749,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-12T02:00:06.386Z","response_time":87,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-hooks","agentic-rag","ai-tools","context-engineering","niah","pypi-package","python-library","rag","token-optimization"],"created_at":"2026-07-12T15:30:23.005Z","updated_at":"2026-07-12T15:30:24.364Z","avatar_url":"https://github.com/japlete.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# utf-token\n\nConvert random string identifiers to a LLM-friendly format to reduce token usage in certain retrieval and agentic tasks.\n\n`utf-token` encodes the identifier into a 2-token sequence by default with 30 bits of entropy. Collisions are prevented automatically and the conversion is fully reversible.\n\n![token savings vs hex, base64, and uuid](docs/assets/benchmarks/token_savings.png)\n\n## Install\n\n```shell\nuv add utf-token\n```\n\n## Usage\n\nThe `IdTokenBiMap` class is used to encode identifiers and store the full original bytes so you can recover them later.\n\n```python\nfrom utf_token import IdTokenBiMap\n\nbimap = IdTokenBiMap()\n\nhex_str = \"215aada34d0987ebfb9de132d913e46b\"\n# 17 tokens: 215 a ada 34 d 098 7 eb fb 9 de 132 d 913 e 46 b\n\ntoken_hex = bimap.fromhex(hex_str)\nprint(token_hex)\n# 2 tokens: ao 691\n\nreconstructed_hex = bimap.tohex(token_hex) # Recovers the original hex string\n```\n\nForward methods: `frombytes`, `fromhex`, `frombase64`, `fromuuid`.\nReverse methods: `tobytes`, `tohex`, `tobase64`, `touuid`.\n\nBoth forward and reverse methods accept either:\n\n- a single value -\u003e returns one encoded `str` (or recovered value)\n- an iterable of values -\u003e returns a lazy iterator\n\n### Persisting the reversible map\n\nThe internal map in `IdTokenBiMap` can be saved and restored to transfer offline conversions for online usage:\n\n- `to_dict` / `from_dict`\n- `to_json` / `from_json`\n\n## Optional arguments\n\n### LLM - token vocabulary pairing\n\nPick the token vocabulary that matches the model you are using. Current options are:\n\n- Default: `o200k` (OpenAI GPT-5+)\n- `gemma4` (Google Gemma 4)\n\n```python\nbimap = IdTokenBiMap(vocab=\"gemma4\")\n```\n\n### Controlling how many bits are encoded with `keep_bits`\n\n`IdTokenBiMap` takes `keep_bits` at construction (default **30**):\n\n- a positive integer that is a multiple of the vocab's `pair_index_bits` (15 for shipped vocabs): you get 1 token per 15 bits\n- `None` or `\"all\"`: encode the full input\n\n```python\nshort_bimap = IdTokenBiMap()                                     # keep_bits=30\nlonger_bimap = IdTokenBiMap(keep_bits=45)\nfull_bimap = IdTokenBiMap(keep_bits=\"all\")\n\nshort = short_bimap.frombytes(b\"\\x01\\x02\\x03\\x04\\x05\\x06\")\nshort_bimap.tobytes(short) == b\"\\x01\\x02\\x03\\x04\\x05\\x06\"        # reverse returns the full input\n```\n\nThe default 30 bits (two full 15-bit chunks) is enough entropy for retrieval workloads where you only need a handful of distinct identifiers visible to the model at once, and is also the minimum we recommend for the healing logic described below to stay reliable. Use a larger multiple of 15 if you need more in-context disambiguation.\n\n### Healing transcription errors on reverse lookup\n\nLLMs occasionally make transcription errors when copying identifiers. Reverse methods accept an `errors` keyword to control what happens when the input is not an exact match in the reverse map:\n\n- `errors=\"fix\"` (default): return the closest previously encoded identifier by Levenshtein distance.\n- `errors=\"raise\"`: if the exact lookup misses, raise `KeyError`. Useful when you want to manage error handling yourself.\n\n```python\nbimap = IdTokenBiMap()\nencoded = bimap.fromuuid(\"123e4567-e89b-12d3-a456-426614174000\")\n\nbimap.touuid(encoded)                                    # exact match\nbimap.touuid(encoded[:-1] + \"Z\")                         # heals to nearest stored id\n\nbimap.touuid(\"not_a_real_id\", errors=\"raise\")            # raises KeyError\n\nif encoded in bimap:                                     # supports membership checks\n    print(\"This will print\")\n```\n\n## Standalone forward-only helpers\n\n`frombytes`, `fromhex`, `frombase64`, and `fromuuid` are also available as standalone module-level functions. They perform only the forward conversion, and they default to keeping the full input rather than truncating. They are useful when you want to plug `utf-token` into your own data flow or build your own reverse-lookup table:\n\n```python\nfrom utf_token import fromhex\n\nmy_hex = \"215aada34d0987ebfb9de132d913e46b\"\nencoded_hex = fromhex(my_hex)                            # full input\nshort_hex = fromhex(my_hex, keep_bits=30)                 # top 30 MSBs\n```\n\nBoth `keep_bits=None` and `keep_bits=\"all\"` keep the full input.\n\nFor the standalone functions, pass the `vocab` parameter in the call.\n\n## Included safe character set in tokens\n\nBoth `o200k` and `gemma4` lookup tables are restricted to ASCII (`A-Z`, `a-z`, `0-9`, `_`) to avoid LLM confusion.\n\nNeither vocabulary emits quotes, slashes, brackets, commas, pipes, whitespace, or other delimiter characters, which makes the output easy to embed in JSON, Markdown, logs, tables, and prompts where the LLM or code needs to see clearly where an identifier begins and ends.\n\n### Instructions to include in prompts/tools\n\nTo avoid confusion when your agent sees these IDs, you can adapt these instructions to your specific use case:\n\n\u003e Identifiers are random LLM token sequences containing only ASCII alphanumeric or `_` characters. They are delimited by `\u003cinsert your delimiters here\u003e`. Some identifiers may contain words or part of words, it's just a coincidence due to the use of tokens. Do not translate or fix typos in the identifiers. Transcribe them **verbatim**.\n\n#### Other recommendations for maximum reliability in identifier retrieval\n\n1. Use consistent delimiters to clearly separate identifiers from other text in the prompt.\n2. Keep the default `keep_bits=30` (or a higher multiple of 15) so the healing logic has enough signal to disambiguate identifiers.\n3. Use structured outputs / JSON tools to request the identifiers. Provide a regex pattern such as `^[A-Za-z0-9_]+$` for the output strings in the JSON schema.\n4. Use smart models. For OpenAI, use at least GPT-5.4-mini (not nano). For Gemini, use at least Gemma 4. For Anthropic, use at least Haiku 4.5.\n5. Use low temperature if the model supports it.\n\n## Retrieval benchmark\n\nA NIAH-style benchmark is included to test small LLMs (GPT-5.4-mini, Gemma 4, Claude Haiku) on retrieval accuracy. With 100 samples for each model, and both full-input and default `keep_bits=30` identifiers, the success rate is 100%. The context length is 32k tokens (calibrated for hex identifiers, then re-encoded for each encoding), and identifiers have 16 bytes of entropy.\n\nSee [`docs/benchmarks_niah.md`](docs/benchmarks_niah.md).\n\nThe synthetic NIAH benchmark was adapted from [NVIDIA/RULER](https://github.com/NVIDIA/RULER).\n\n## How it works\n\n`utf-token` encodes the underlying bytes directly. Each vocabulary ships two pre-built lookup tables, generated offline by [`scripts/process_token_vocab.py`](scripts/process_token_vocab.py): a large pair table indexed by either 15 or 16 bits (depending on how many clean tokens the vocabulary can supply) and a small tail table indexed by 8 bits.\n\nFor 15-bit pair tables (both shipped vocabs) the encoder treats the input as an MSB-first bitstream, splits it into 15-bit chunks for the pair table, and uses the tail table for any 1–8 bit residual at the end. A 16-bit fast path is also implemented for any future vocabulary that can fill a 16-bit pair table under the curated `latin_16bit` recipe.\n\n`IdTokenBiMap` keeps a forward map and a reverse map so the generated string can be resolved back to the original bytes later. Collisions can happen when different inputs produce the same encoded string, especially when `keep_bits` truncates them to a short prefix. When `IdTokenBiMap` sees that a new value would collide with an existing one, it deterministically moves to the next prefix until it finds an unused encoded string. The stored reverse map still points that generated string back to the original full input.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaplete%2Futf-token","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaplete%2Futf-token","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaplete%2Futf-token/lists"}