{"id":49079787,"url":"https://github.com/swiss-ai/parity-aware-bpe","last_synced_at":"2026-04-20T12:19:41.508Z","repository":{"id":322606912,"uuid":"1028346245","full_name":"swiss-ai/parity-aware-bpe","owner":"swiss-ai","description":"Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [arXiv 2025]","archived":false,"fork":false,"pushed_at":"2025-12-01T14:59:35.000Z","size":22,"stargazers_count":15,"open_issues_count":0,"forks_count":3,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-04T03:50:42.222Z","etag":null,"topics":["bpe","llms","multilingual-nlp","multilingual-tokenization","tokenization"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2508.04796","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/swiss-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-29T11:50:30.000Z","updated_at":"2025-12-01T14:59:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"28043138-ba41-4dc5-9e26-dc4d52451f34","html_url":"https://github.com/swiss-ai/parity-aware-bpe","commit_stats":null,"previous_names":["swiss-ai/parity-aware-bpe"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/swiss-ai/parity-aware-bpe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swiss-ai%2Fparity-aware-bpe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swiss-ai%2Fparity-aware-bpe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swiss-ai%2Fparity-aware-bpe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swiss-ai%2Fparity-aware-bpe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/swiss-ai","download_url":"https://codeload.github.com/swiss-ai/parity-aware-bpe/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swiss-ai%2Fparity-aware-bpe/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32046755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T11:35:06.609Z","status":"ssl_error","status_checked_at":"2026-04-20T11:34:48.899Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bpe","llms","multilingual-nlp","multilingual-tokenization","tokenization"],"created_at":"2026-04-20T12:19:38.403Z","updated_at":"2026-04-20T12:19:41.503Z","avatar_url":"https://github.com/swiss-ai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://arxiv.org/abs/2508.04796\"\u003e\n    \u003cimg alt=\"arXiv\" src=\"https://img.shields.io/badge/arXiv-Paper-red\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\n    \u003cimg alt=\"License\" src=\"https://img.shields.io/badge/License-MIT-green.svg\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\nParity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization\n================================== \nThis repository provides an implementation of the **Parity-Aware BPE** algorithm.\nPaper: [\"Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization\"](https://arxiv.org/abs/2508.04796) [ACL 2026]\n\n\nOverview\n------------\n**Parity-aware BPE** learns a tokenization that ensures parity in token lengths across languages on a multi-parallel development set.\nUnlike standard BPE, which optimizes merges based on a single corpus, this approach explicitly considers cross-lingual fairness during the tokenization process.\n\n\nInstallation\n------------\n\nYou can install this package directly from GitHub:\n\n```bash\npip install git+https://github.com/swiss-ai/parity-aware-bpe.git\n```\n\nFor development installation:\n\n```bash\ngit clone https://github.com/swiss-ai/parity-aware-bpe.git\ncd parity-aware-bpe\npip install -e .\n```\n\n\nUsage Instructions\n------------------\n\nThe arguments of `parity-aware-bpe` are as follows:\n\n- `--variant`: Parity-aware BPE variant. Options:\n    - base – standard parity-aware BPE (default)\n    - window – moving-window balancing version \n- `--input`: Space-separated list of training corpora (one per language).\n- `--dev`: Space-separated list of development texts used for parity computation (multi-parallel). The tool assumes that the language of the nth input corpus corresponds to the nth dev corpus (same order as `--input`).\n- `--ratio`: Space-separated list of desired compression ratios (floats), relative to pre-tokenized training set length, per input language. Can be used for parity computation (on training data) in lieu of development set.\n- `--global-merges`: Optionally, one can perform the first M merge operations based on global frequency statistics (equivalent to standard BPE), and only switch to a parity-optimizing mode after (Hybrid parity-aware BPE). This argument controls how many merge operations are performed based on global statistics.\n- `--symbols`: Total number of BPE merges to perform.\n- `--output`: Path to the output file where BPE merge rules will be saved (one per line).\n- `--total-symbols`: Adjusts the number of merges by subtracting character counts (so `--symbols` approximates total symbols needed).\n- `--min-frequency`:  Minimum pair frequency to continue merging (default: `2`).\n- `--window-size`: Context window size for the window-balancing variant (default: `100`).\n- `--alpha`: Parameter controlling the moving-window balancing behavior (default: `2`).\n\n### Example Usage\n\n```bash\nPython3 parity_aware_bpe/parity_aware_learn_bpe.py \\\n        --symbols {num_operations} \\ \n        --variant {\"base\" or \"window\"} \\\n        --input {train_files}  \\\n        --dev {development_files}  \\\n        --output {output_file} \n```\n\nClassical BPE\n------------------\nTo run the classical BPE algorithm you can use `learn_bpe.py`:\n\n```bash\nPython3 parity_aware_bpe/learn_bpe.py \\\n        --symbols {num_operations} \\ \n        --input {train_files}  \\\n        --dev {development_files}  \\\n        --output {output_file} \n```\n\n\nGenerating a Vocabulary\n------------------\nAfter learning the merges, you can build a vocabulary file using the `build_vocab_from_merges` function in `HF_tokenizer.py`.\nTo create a Hugging Face-compatible tokenizer:\n\n```bash\nPython3 parity_aware_bpe/HF_tokenizer.py \\\n        --merges_file_path {merge_file_path} \\\n        --tokenizer_path {tokenizer_save_folder}\n\n```\n\nLoading the tokenizer\n------------------\n```python\nimport os\nfrom transformers import PreTrainedTokenizerFast\nfrom tokenizers.pre_tokenizers import Whitespace, ByteLevel\nfrom tokenizers.models import BPE\nfrom tokenizers import Tokenizer, pre_tokenizers\n\nmerge_file = os.path.join(tokenizer_path, \"merges.txt\")\nvocab_file = os.path.join(tokenizer_path, \"vocab.json\")\ntokenizer = Tokenizer(BPE(vocab=vocab_file, merges=merge_file))\npre_tokenizer = pre_tokenizers.Sequence([Whitespace(), ByteLevel(use_regex=False)]) # You need to use the same pre_tokenizer as the one used in BPE training\ntokenizer.pre_tokenizer = pre_tokenizer\n\nwrapped_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)\n```\n\nIntrinsic Evaluation\n------------------\nFor our intrinsic evaluation, we use `Tok##Suite`  to analyze and compare tokenizers across multiple languages and metrics. You can find the evaluation suite [here](https://github.com/cimeister/tokenizer-analysis-suite).\n\nCitation\n------------------\n\nIf you use this code for your research, please cite our paper:\n\n``` bib\n@article{foroutan-meister-et-al-2025-parity-aware-bpe,\n  title={Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization},\n  author={Foroutan, Negar and Meister, Clara and Paul, Debjit and Niklaus, Joel and Ahmadi, Sina, and Bosselut, Antoine and Sennrich, Rico},\n  url={https://arxiv.org/abs/2508.04796},\n  booktitle={arXiv},\n  year={2025}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswiss-ai%2Fparity-aware-bpe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fswiss-ai%2Fparity-aware-bpe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswiss-ai%2Fparity-aware-bpe/lists"}