{"id":17874008,"url":"https://github.com/bminixhofer/zett","last_synced_at":"2025-04-05T21:05:49.978Z","repository":{"id":238468481,"uuid":"796619876","full_name":"bminixhofer/zett","owner":"bminixhofer","description":"Code for Zero-Shot Tokenizer Transfer","archived":false,"fork":false,"pushed_at":"2025-01-14T09:53:55.000Z","size":1092,"stargazers_count":125,"open_issues_count":8,"forks_count":10,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-29T20:03:26.692Z","etag":null,"topics":["language-model","llm","llms","multilingual","tokenization","transfer-learning"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2405.07883","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bminixhofer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-06T09:58:30.000Z","updated_at":"2025-03-20T08:04:11.000Z","dependencies_parsed_at":"2024-06-16T19:56:02.188Z","dependency_job_id":"fc7c95e8-aebe-4051-9e9d-b9096e02394c","html_url":"https://github.com/bminixhofer/zett","commit_stats":null,"previous_names":["bminixhofer/zett"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bminixhofer%2Fzett","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bminixhofer%2Fzett/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bminixhofer%2Fzett/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bminixhofer%2Fzett/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bminixhofer","download_url":"https://codeload.github.com/bminixhofer/zett/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247399871,"owners_count":20932876,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["language-model","llm","llms","multilingual","tokenization","transfer-learning"],"created_at":"2024-10-28T11:07:17.774Z","updated_at":"2025-04-05T21:05:49.947Z","avatar_url":"https://github.com/bminixhofer.png","language":"Python","funding_links":[],"categories":["LLM Development and Optimization"],"sub_categories":["Architecture Innovations"],"readme":"\u003ch1 align=\"center\"\u003eZero-Shot Tokenizer Transfer\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg src=https://github.com/bminixhofer/zett/assets/13353204/f64dbdf4-da05-4586-8eb6-bf5a622b2160 width=300px\u003e\n\u003c/P\u003e\n\nThis repository contains the code for the paper [Zero-Shot Tokenizer Transfer](https://arxiv.org/abs/2405.07883). ZeTT frees language models from their tokenizer, allowing you to use any model with any tokenizer, with little or no extra training⚡\n\n## Available pretrained hypernetworks\n\n| Hypernetwork                                                                                                                                                        | ..for Model                                                                                                                       | Comments                   |\n|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|----------------------------|\n| [benjamin/zett-hypernetwork-xlm-roberta-base](https://huggingface.co/benjamin/zett-hypernetwork-xlm-roberta-base)                                                   | [xlm-roberta-base](https://huggingface.co/FacebookAI/xlm-roberta-base)                                                            | multilingual, 26 languages |\n| [benjamin/zett-hypernetwork-Mistral-7B-v0.1](https://huggingface.co/benjamin/zett-hypernetwork-Mistral-7B-v0.1)                                                     | [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)                                                     | English + Code             |\n| [benjamin/zett-hypernetwork-multilingual-Mistral-7B-v0.1](https://huggingface.co/benjamin/zett-hypernetwork-multilingual-Mistral-7B-v0.1)                           | [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)                                                     | multilingual, 26 languages |\n| [benjamin/zett-hypernetwork-TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/benjamin/zett-hypernetwork-TinyLlama-1.1B-intermediate-step-1431k-3T) | [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) | English + Code             |\n| [benjamin/zett-hypernetwork-Meta-Llama-3-8B-experimental](https://huggingface.co/benjamin/zett-hypernetwork-Meta-Llama-3-8B-experimental) | [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) | __experimental__ English + Code, seems to underperform on Code             |\n\n## Using a pretrained hypernetwork\n\n\n### Environment Setup\n\nRequirements are in `requirements.txt`, This, for example, creates a working environment:\n\n```\nconda create -n zett Python=3.11\nconda activate zett\n\npip install -r requirements.txt\npip install -U \"jax[cuda12_pip]==0.4.23\" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html # adjust based on your CUDA version\npip install -e .\n```\n\n### Transferring to a new tokenizer\n\n\u003cdetails open\u003e\n    \u003csummary\u003eLet's transfer XLM-RoBERTa to the GPT2 tokenizer.\u003c/summary\u003e\n\n```bash\ngit clone https://huggingface.co/benjamin/zett-hypernetwork-xlm-roberta-base\n\npython3 scripts/transfer.py \\\n    --target_model=FacebookAI/xlm-roberta-base \\\n    --tokenizer_name=gpt2 \\\n    --output=my-new-fancy-xlm-r \\\n    --model_class=AutoModelForMaskedLM \\\n    --lang_code=en \\\n    --checkpoint_path=zett-hypernetwork-xlm-roberta-base \\\n    --save_pt # otherwise saves only Flax weights\n```\n\nTada!\n\n```python\nfrom transformers import AutoModelForMaskedLM, AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"my-new-fancy-xlm-r\")\nmodel = AutoModelForMaskedLM.from_pretrained(\"my-new-fancy-xlm-r\")\n\nout = model(**tokenizer(\"Hello world!\", return_tensors=\"pt\"))\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e..or Mistral-7B to the GPT-NeoX tokenizer:\u003c/summary\u003e\n\n```bash\ngit clone https://huggingface.co/benjamin/zett-hypernetwork-Mistral-7B-v0.1\n\n# because Flax weights are not merged in the main branch, we need to specify the revision of a PR containing Flax weights\npython3 scripts/transfer.py \\\n    --target_model=mistralai/Mistral-7B-v0.1 \\\n    --revision=refs/pr/95 \\\n    --tokenizer_name=EleutherAI/gpt-neox-20b \\\n    --output=my-new-fancy-mistral \\\n    --model_class=AutoModelForCausalLM \\\n    --checkpoint_path=zett-hypernetwork-Mistral-7B-v0.1 \\\n    --save_pt # otherwise saves only Flax weights\n```\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"my-new-fancy-mistral\")\nmodel = AutoModelForCausalLM.from_pretrained(\"my-new-fancy-mistral\")\n\nout = model(**tokenizer(\"Hello world!\", return_tensors=\"pt\"))\n```\n\u003c/details\u003e\n\nAlthough the codebase is in Jax/Flax, there are Pytorch bindings for the model in `./hf_hypernet`. You can use them as follows:\n\n```python\nfrom transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer\nfrom zett.utils import get_surface_form_matrix\n\nbase_model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\nhypernet = AutoModel.from_pretrained(\"benjamin/zett-hypernetwork-Mistral-7B-v0.1\", trust_remote_code=True)\n\nsource_embeddings = torch.concatenate([\n    base_model.get_input_embeddings().weight.data,\n    base_model.get_output_embeddings().weight.data,\n], axis=1)\n\nhn_tokenizer = AutoTokenizer.from_pretrained(\"benjamin/zett-hypernetwork-Mistral-7B-v0.1\")\n\ntarget_surface_forms = get_surface_form_matrix(\n    [\"Ġhello\", \"Ġworld\"], # byte representation of the tokens to predict\n    maxlen=hypernet.config.hn_surface_maxlen,\n    tokenizer_to_use=hn_tokenizer,\n)[0]\n\n# the last output is the predicted bias in case the model uses a bias (e.g. XLM-R)\npredicted_input_embeddings, predicted_output_embeddings, _ = hypernet(\n    torch.from_numpy(target_surface_forms),\n    source_embeddings=source_embeddings\n)\n\n```\n\nbut `transfer.py` is currently not ported to PyTorch (PRs welcome!).\n\n## Advanced usage\n\n### Training a Hypernetwork\n\nThe script used to train the hypernetwork is `train.py`. \n\nBut first, you'll need to download and prepare the data via `data/prepare.py` and `data/prepare_code.py`.\n\nYou'll also need to install the Rust module in `rust_utils` (used to quickly sample tokenizers) via e.g. `cd rust_utils \u0026\u0026 maturin develop --release`.\n\nOnce finished, you can run training using the configs in `configs/`. For example:\n\n```bash\npython3 train.py configs/zeroshot/v7:tinyllama_en+code:lw=0.5_long.json\n```\n\nto train a hypernetwork for TinyLlama on English and Code.\n\n### Transferring fine-tuned models to a new tokenizer using a base model hypernetwork\n\nUse `scripts/apply_to_ft.py` to transfer the tokenizers of a fine-tuned model, given a base model with already transferred tokenizer. For example:\n\n```bash\npython3 scripts/apply_to_ft.py \\\n    --output=transferred-chat-mistral \\\n    --base_model_path=mistralai/Mistral-7B-v0.1 \\\n    --ft_model_path=mistralai/Mistral-7B-Instruct-v0.1 \\\n    --tokenizer_swapped_base_model_path=path-to-base-model-with-new-tokenizer \\\n    --lambdas 0.5 \\\n```\n\n### Reproducing the experiments from the paper\n\nThere are bash scripts in `experiments/` to allow reproducing the main results from the paper.\n\nEvaluation on code is still missing because we are using a fork of `bigcode-evaluation-harness` to fix some issues we encountered. They will be added soon.\n\n### Unigramifying, using n-shot transferred models, reproducing the tokenizers from the paper, etc.\n\nGuide coming soon... (but feel free to dig into `scripts/` in the meantime)\n\n### \n\n## Disclaimer\n\nI prioritized releasing the code quickly instead of making it perfectly clean. There may still be remnants of my personal environment used to train the models and other non-niceties. I am in the process of cleaning this up. If you run into any problems or have any questions, please open an issue.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbminixhofer%2Fzett","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbminixhofer%2Fzett","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbminixhofer%2Fzett/lists"}