{"id":26103677,"url":"https://github.com/aleph-alpha/trigrams","last_synced_at":"2025-04-12T18:08:38.946Z","repository":{"id":245917304,"uuid":"819527545","full_name":"Aleph-Alpha/trigrams","owner":"Aleph-Alpha","description":null,"archived":false,"fork":false,"pushed_at":"2024-08-29T10:59:46.000Z","size":10735,"stargazers_count":54,"open_issues_count":0,"forks_count":4,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-12T18:07:46.582Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aleph-Alpha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-24T17:27:12.000Z","updated_at":"2025-03-31T01:02:43.000Z","dependencies_parsed_at":"2024-08-29T09:48:21.529Z","dependency_job_id":"59911924-321c-4a05-96ad-4a8be08fd30c","html_url":"https://github.com/Aleph-Alpha/trigrams","commit_stats":null,"previous_names":["aleph-alpha/trigrams"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Ftrigrams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Ftrigrams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Ftrigrams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Ftrigrams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aleph-Alpha","download_url":"https://codeload.github.com/Aleph-Alpha/trigrams/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248610340,"owners_count":21132921,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-09T20:42:01.457Z","updated_at":"2025-04-12T18:08:38.907Z","avatar_url":"https://github.com/Aleph-Alpha.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# T-Free: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory Efficient Embeddings\n\nPaper link: https://arxiv.org/abs/2406.19223\n\n\n![Figure 1: Method comparison classical Tokenizer to T-Free](figures/fig_1.png)\n![Table 1: Near-Duplicate and fertility metrics for several models](figures/tab_1.png)\n\u003cimg src=\"figures/fig_3.png\" width=50%\u003e\n\n\n## Checkpoints\n- [7B trained on 1 epoch fineweb-edu](https://huggingface.co/Aleph-Alpha/tfree-research-vocab-32k-fineweb-steps-370k)\n- [7B trained on 1 epoch fineweb-edu + continued instruct tuning](https://huggingface.co/Aleph-Alpha/tfree-research-vocab-32k-fineweb-steps-370k-instruct-10k)\n\n\n\n## Install\n\n```console\nconda create --name t-free python=3.11 -y\nconda activate t-free\n\nconda install pytorch==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia\n\npip install -e .[test]\n\npre-commit install\n```\n\n## Running some tests\n```\npip install determined==0.26.4\n\npytest tests/test_data\npytest tests/test_tokenizer\npytest tests/test_trainer\npytest tests/test_inference \n```\n\n## Training\n\n### Data preprocessing\nTraining requires data to be in one of two formats: MemoryMap or AlignedMemoryMap. MemoryMaps simply store data in byte format with indices for fast random access.\nAlignedMemoryMaps build a second index above, that aggregate a minimum byte-count per entry. We use this for pre-training to avoid the need of aggregating a dynamic number of random entries to fill the full sequence length, which becomes hard when trying to ensure that each entry is will only be seen once during one epoch.  \n\nAn example of how to download and convert e.g. FinetuneWeb of HuggingFace into a MemoryMap is found in data/download_data.py. \nNote that the data is stored in string format (converted to bytes). Trigramification or classical Tokenization are executed on-demand during training, depending on the respective training configurations.\n\nTo further convert the data into an AlignedMemoryMap execute e.g. data/convert_mmap_to_alignedmmap.py.\nRight now max_bytes is set to 20k, which we found to correlate well to roughly 4k trigram words -- our configured pre-training sequence length.\nNote: If sequence length is exceeded, overhead will simply be cut off. If it is not reached with the index, it will be filled with a random further item.\n\nRight now typical instruction finetunings jsonl's need to casted to MemoryMaps as in data/convert_instruct_jsonl_to_mmap.py.\n\nData in MemoryMap format requires config.data.pretraining = False.\nAlignedMemoryMaps require config.data.pretraining = True.\n\n### Launch training\nWe include example determined configs in determined_configs/ and example training model configs in configs/.\nYou may launch a training through one of the determined experiments, or respectively convert the torch distributed launcher scripts of the following line to your environment:\n```\npython3 -m determined.launch.torch_distributed python3 src/trigram_tokenizer/trainer/train_determined.py --config configs/7b_fineweb.yaml\n```\n\n\n## Inference\nMinimal Example:\n```\n# download checkpoint from huggingface\napt-get install git-lfs\ngit clone https://huggingface.co/Aleph-Alpha/tfree-research-vocab-32k-fineweb-steps-370k\n\n# install this repository\npip install -e .\n\n# adjust checkpoint path in inference.py, then run it\npython inference.py\n```\n\n\nRun one of the prepared inference* scripts.\n\nNote that the inference decode works different to that of other LLMs.\nCurrently we build a dictionary with word-patterns and compute the product with the models output logits to select e.g. the next top-word. This is shown in Fig. 4 below.\n\nAs such you need/ may want to\n- InferencePipe.__init__: pass a path to a collections.Counter pckl file through the \"top_word_dict\" parameter - these will be converted to the patterns of the respective loaded checkpoint config. We counted once the entire fineweb-edu dataset, a top-k subset is found in data/en_fineweb_top_1m_counter.pckl.\n- InferencePipe.__init__: pass an integer through the \"reduce_tokenizer_words_to\" parameter - these cut off the top n words of aboves collection.\n- InferencePipe.generate: pass a \"more_words\" string - a string of words (separated by whitespaces) that will dynamically be added to the dictionary for on demand sampling.\n- InferencePipe.tokenizer.convert_weight_for_word_edge_overweight: will downweight edge-trigrams's as discussed in paper, to reduce artifacts (usually factor \u003c .8).\n- call data/generate_tokenizer.py once with a trained checkpoint. This will preprocess once the patterns for the passed dictionary and store them next to the checkpoint. Subsequent calls of InferencePipe.__init__  then do not require to pass the dict but directly load the stored patterns.  \n\n![Figure 4: T-Free vs Classical Decode](figures/fig_4.png)\n\n## Known Issues/ Ongoing Research\n\n- in trigram_tokenizer.py/decode_trigram we propose two variants to decode words. we are digging into further improvements / are happy for feedback \n\n\n\n## Cite\n\n```\n@article{deiseroth2024t,\n  title={T-FREE: Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings},\n  author={Deiseroth, Bj{\\\"o}rn and Brack, Manuel and Schramowski, Patrick and Kersting, Kristian and Weinbach, Samuel},\n  journal={arXiv preprint arXiv:2406.19223},\n  year={2024}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleph-alpha%2Ftrigrams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faleph-alpha%2Ftrigrams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleph-alpha%2Ftrigrams/lists"}