{"id":51508546,"url":"https://github.com/blackwell-systems/merge-barriers","last_synced_at":"2026-07-08T03:30:38.663Z","repository":{"id":368278091,"uuid":"1284386959","full_name":"blackwell-systems/merge-barriers","owner":"blackwell-systems","description":"Merge Barriers in BPE Tokenization: How Tokenizer Design Causally Determines Attention Head Specialization. Paper, 23 eval scripts, 86 result files, tokenizer definitions, 4 model checkpoints.","archived":false,"fork":false,"pushed_at":"2026-06-29T22:13:47.000Z","size":11479,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-29T22:15:35.787Z","etag":null,"topics":["architecture-independence","attention-heads","attention-mechanism","bpe","causal-ablation","deep-learning","delimiter-specialization","gpt-neox","gqa","llama","llm","mechanistic-interpretability","merge-barriers","nlp","reproducible-research","sentencepiece","structured-data","subword-tokenization","tokenization","transformer"],"latest_commit_sha":null,"homepage":"https://github.com/blackwell-systems/merge-barriers/blob/main/paper/merge-barriers-v3.pdf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blackwell-systems.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-29T20:15:48.000Z","updated_at":"2026-06-29T22:13:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/blackwell-systems/merge-barriers","commit_stats":null,"previous_names":["blackwell-systems/merge-barriers"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/blackwell-systems/merge-barriers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fmerge-barriers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fmerge-barriers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fmerge-barriers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fmerge-barriers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blackwell-systems","download_url":"https://codeload.github.com/blackwell-systems/merge-barriers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fmerge-barriers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35251015,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-08T02:00:06.796Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["architecture-independence","attention-heads","attention-mechanism","bpe","causal-ablation","deep-learning","delimiter-specialization","gpt-neox","gqa","llama","llm","mechanistic-interpretability","merge-barriers","nlp","reproducible-research","sentencepiece","structured-data","subword-tokenization","tokenization","transformer"],"created_at":"2026-07-08T03:30:38.123Z","updated_at":"2026-07-08T03:30:38.655Z","avatar_url":"https://github.com/blackwell-systems.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tokenizer-Attention Coupling\n\n**How BPE Merge Decisions Permanently Shape Transformer Internal Organization**\n\nDayna Blackwell, Blackwell Systems\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"paper/tokenizer-attention-coupling.pdf\"\u003e\u003cimg src=\"https://img.shields.io/badge/paper-PDF-18befc?style=for-the-badge\" alt=\"Paper\"\u003e\u003c/a\u003e\n  \u003ca href=\"paper/stranded-attention.pdf\"\u003e\u003cimg src=\"https://img.shields.io/badge/short_paper-Stranded_Attention-18befc?style=for-the-badge\" alt=\"Short Paper\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://huggingface.co/blackwell-systems/merge-barriers\"\u003e\u003cimg src=\"https://img.shields.io/badge/checkpoints-HuggingFace-ff9d00?style=for-the-badge\" alt=\"Checkpoints\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://doi.org/10.5281/zenodo.20925910\"\u003e\u003cimg src=\"https://img.shields.io/badge/DOI-10.5281%2Fzenodo.20925910-blue?style=for-the-badge\" alt=\"DOI\"\u003e\u003c/a\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-MIT-333?style=for-the-badge\" alt=\"License\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"charts/stranded-swap-28b.png\" width=\"90%\" alt=\"384/384 heads wake up with clean tokenization\"\u003e\n\u003c/p\u003e\n\n## The Finding\n\nBPE tokenizers merge delimiter characters with adjacent content, permanently constraining the attention capacity of every head in the transformer. When a standard BPE model is given input with clean delimiter boundaries (using its own vocabulary, no retraining), **384/384 heads at 410M and 768/768 at 1.3B show 4x more delimiter attention**. The model has the circuitry for structural processing. The tokenizer prevents it from working.\n\nThis **frustration gap** (41 percentage points of wasted structural capacity) appears by training step 5,000, does not change across 35,000 additional steps, and shows no drift toward dormancy. We call these **stranded heads**: a previously undescribed attention state between productive anchoring and dormancy.\n\nAt scale, stranding becomes actively harmful. Standard BPE at 1.3B develops 124 counterproductive delimiter heads: **removing them improves comprehension by 57%**.\n\n## The Fix\n\n**Merge barriers**: 16 delimiter characters forbidden from BPE merges. No changes to the BPE algorithm, training loop, or model architecture. 16 lines of tokenizer config.\n\n```\n|  @  \u003c  \u003e  \"  '  :  ,  ;  \\t  {  }  [  ]  (  )\n```\n\n## Results\n\n| Finding | Numbers |\n|---------|---------|\n| Structured data PPL improvement | **3-738x** (scaling with model size) |\n| Code comprehension improvement | **1.5x** (per-token advantage maps to merge rates) |\n| Chemistry (SMILES) improvement | **2.2x** (67% more valid molecules) |\n| Natural language cost | **Zero** (19.4 vs 19.5 on Wikipedia) |\n| Stranded heads (tokenizer swap) | **384/384** at 410M, **768/768** at 1.3B |\n| Frustration gap | **41 pp**, flat from step 5K to 40K |\n| P0 drift | **None** (stranding is stable, not a pathway to dormancy) |\n| Model B heads counterproductive | **-22.9%** (NeoX), **-53.8%** (Llama 410M), **-57%** (Llama 1.3B) |\n| Model A ablation degradation | **+59%** (NeoX), **+21.4%** (Llama 410M), **+420.5%** (Llama 1.3B) |\n| Delimiter heads (Model A) | **50** (NeoX), **66** (Llama 410M), **161** (Llama 1.3B) |\n| Cross-format transfer | **8/9** unseen formats at every scale |\n| Architecture independence | GPT-NeoX (MHA) and Llama (GQA) |\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"charts/stranded-timeline-28c.png\" width=\"45%\" alt=\"Frustration gap across training\"\u003e\n  \u003cimg src=\"charts/scaling-paradox-b0.png\" width=\"45%\" alt=\"Scaling paradox\"\u003e\n\u003c/p\u003e\n\n## Key Contributions\n\n1. **Stranded heads**: every head is stranded, not just a few. The tokenizer swap proves it (28b). The training timeline proves it's permanent (28c). The P0 drift analysis proves it's stable, not a pathway to dormancy.\n\n2. **Tokenizer-attention coupling**: BPE merge decisions made during tokenizer training causally constrain which attention heads develop and how much capacity is available for content processing. Confirmed across 2 architectures, 2 scales, 3 domains.\n\n3. **Domain generalization**: the mechanism operates on structured data, code (per-token advantage maps to merge rates), and molecular notation (extends Jindal \u0026 Ju 2026). Effect magnitude correlates with delimiter density.\n\n4. **Scaling paradox**: standard BPE's delimiter heads are counterproductive at every scale. More capacity invested in detecting corrupted boundaries means more capacity wasted.\n\n## Repository Contents\n\n```\npaper/\n  tokenizer-attention-coupling.md    # Main paper source\n  tokenizer-attention-coupling.pdf   # Main paper PDF\n  stranded-attention.md              # Short paper source\n  stranded-attention.pdf             # Short paper PDF\n\neval_*.py                            # 30+ evaluation/ablation scripts\nruns/                                # 100+ result files (JSON + logs)\ncharts/                              # 50+ charts\ndomains/                             # Code and chemistry domain experiments\n  code/results/                      # Code domain results\n  chemistry/results/                 # Chemistry domain results\n\nstructok-64k.json                    # Merge-barrier tokenizer\nGUIDE.md                             # Beginner's guide to the research\nRESEARCH.md                          # Research summary\nEVAL-SCRIPTS.md                      # Script index with descriptions\n```\n\n## Model Checkpoints\n\nAvailable on [HuggingFace](https://huggingface.co/blackwell-systems/merge-barriers):\n\n| Model | Architecture | Tokenizer | Steps |\n|-------|-------------|-----------|-------|\n| NeoX A | GPT-NeoX 410M | structok-64k (merge barriers) | 20,000 |\n| NeoX B | GPT-NeoX 410M | standard-64k (no barriers) | 20,000 |\n| Llama A | Llama 410M (GQA 4:1) | structok-64k (merge barriers) | 40,000 |\n| Llama B | Llama 410M (GQA 4:1) | standard-64k (no barriers) | 40,000 |\n\n1.3B checkpoints available on R2 (contact for access).\n\n## Reproducing Results\n\n```bash\npip install torch transformers tokenizers matplotlib numpy\n```\n\nDownload checkpoints from [HuggingFace](https://huggingface.co/blackwell-systems/merge-barriers), then:\n\n```bash\n# Stranded heads tokenizer swap (Experiment 28b)\npython eval_stranded_swap.py --checkpoint llama-b.pt --tokenizer standard-64k.json\n\n# Training dynamics timeline (Experiment 28c)\npython eval_stranded_timeline.py --checkpoint-dir checkpoints/ --tokenizer standard-64k.json\n\n# P0 drift analysis\npython eval_p0_drift.py --checkpoint-dir checkpoints/ --tokenizer standard-64k.json\n\n# Head ablation (18 phases)\npython eval_ablation_v4_excess.py --checkpoint-a neox-a.pt --tokenizer-a structok-64k.json\n\n# PPL comparison\npython eval_model.py --checkpoint neox-a.pt --tokenizer structok-64k.json --size 410m\n```\n\n## Citation\n\n```bibtex\n@article{blackwell2026tokenizer,\n  title={Tokenizer-Attention Coupling: How BPE Merge Decisions Permanently Shape Transformer Internal Organization},\n  author={Blackwell, Dayna},\n  year={2026},\n  doi={10.5281/zenodo.20925910},\n  url={https://github.com/blackwell-systems/merge-barriers}\n}\n```\n\n## License\n\nMIT\n\n## Related\n\n- [GCF: Graph Compact Format](https://github.com/blackwell-systems/gcf) (the wire format used as comparison format, designed from the tokenizer analysis)\n- [DOI: 10.5281/zenodo.20925910](https://doi.org/10.5281/zenodo.20925910)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackwell-systems%2Fmerge-barriers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblackwell-systems%2Fmerge-barriers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackwell-systems%2Fmerge-barriers/lists"}