https://github.com/blackwell-systems/merge-barriers
Merge Barriers in BPE Tokenization: How Tokenizer Design Causally Determines Attention Head Specialization. Paper, 23 eval scripts, 86 result files, tokenizer definitions, 4 model checkpoints.
https://github.com/blackwell-systems/merge-barriers
architecture-independence attention-heads attention-mechanism bpe causal-ablation deep-learning delimiter-specialization gpt-neox gqa llama llm mechanistic-interpretability merge-barriers nlp reproducible-research sentencepiece structured-data subword-tokenization tokenization transformer
Last synced: 16 days ago
JSON representation
Merge Barriers in BPE Tokenization: How Tokenizer Design Causally Determines Attention Head Specialization. Paper, 23 eval scripts, 86 result files, tokenizer definitions, 4 model checkpoints.
- Host: GitHub
- URL: https://github.com/blackwell-systems/merge-barriers
- Owner: blackwell-systems
- License: other
- Created: 2026-06-29T20:15:48.000Z (25 days ago)
- Default Branch: main
- Last Pushed: 2026-06-29T22:13:47.000Z (25 days ago)
- Last Synced: 2026-06-29T22:15:35.787Z (25 days ago)
- Topics: architecture-independence, attention-heads, attention-mechanism, bpe, causal-ablation, deep-learning, delimiter-specialization, gpt-neox, gqa, llama, llm, mechanistic-interpretability, merge-barriers, nlp, reproducible-research, sentencepiece, structured-data, subword-tokenization, tokenization, transformer
- Language: Python
- Homepage: https://github.com/blackwell-systems/merge-barriers/blob/main/paper/merge-barriers-v3.pdf
- Size: 10.9 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Tokenizer-Attention Coupling
**How BPE Merge Decisions Permanently Shape Transformer Internal Organization**
Dayna Blackwell, Blackwell Systems
## The Finding
BPE tokenizers merge delimiter characters with adjacent content, permanently constraining the attention capacity of every head in the transformer. When a standard BPE model is given input with clean delimiter boundaries (using its own vocabulary, no retraining), **384/384 heads at 410M and 768/768 at 1.3B show 4x more delimiter attention**. The model has the circuitry for structural processing. The tokenizer prevents it from working.
This **frustration gap** (41 percentage points of wasted structural capacity) appears by training step 5,000, does not change across 35,000 additional steps, and shows no drift toward dormancy. We call these **stranded heads**: a previously undescribed attention state between productive anchoring and dormancy.
At scale, stranding becomes actively harmful. Standard BPE at 1.3B develops 124 counterproductive delimiter heads: **removing them improves comprehension by 57%**.
## The Fix
**Merge barriers**: 16 delimiter characters forbidden from BPE merges. No changes to the BPE algorithm, training loop, or model architecture. 16 lines of tokenizer config.
```
| @ < > " ' : , ; \t { } [ ] ( )
```
## Results
| Finding | Numbers |
|---------|---------|
| Structured data PPL improvement | **3-738x** (scaling with model size) |
| Code comprehension improvement | **1.5x** (per-token advantage maps to merge rates) |
| Chemistry (SMILES) improvement | **2.2x** (67% more valid molecules) |
| Natural language cost | **Zero** (19.4 vs 19.5 on Wikipedia) |
| Stranded heads (tokenizer swap) | **384/384** at 410M, **768/768** at 1.3B |
| Frustration gap | **41 pp**, flat from step 5K to 40K |
| P0 drift | **None** (stranding is stable, not a pathway to dormancy) |
| Model B heads counterproductive | **-22.9%** (NeoX), **-53.8%** (Llama 410M), **-57%** (Llama 1.3B) |
| Model A ablation degradation | **+59%** (NeoX), **+21.4%** (Llama 410M), **+420.5%** (Llama 1.3B) |
| Delimiter heads (Model A) | **50** (NeoX), **66** (Llama 410M), **161** (Llama 1.3B) |
| Cross-format transfer | **8/9** unseen formats at every scale |
| Architecture independence | GPT-NeoX (MHA) and Llama (GQA) |
## Key Contributions
1. **Stranded heads**: every head is stranded, not just a few. The tokenizer swap proves it (28b). The training timeline proves it's permanent (28c). The P0 drift analysis proves it's stable, not a pathway to dormancy.
2. **Tokenizer-attention coupling**: BPE merge decisions made during tokenizer training causally constrain which attention heads develop and how much capacity is available for content processing. Confirmed across 2 architectures, 2 scales, 3 domains.
3. **Domain generalization**: the mechanism operates on structured data, code (per-token advantage maps to merge rates), and molecular notation (extends Jindal & Ju 2026). Effect magnitude correlates with delimiter density.
4. **Scaling paradox**: standard BPE's delimiter heads are counterproductive at every scale. More capacity invested in detecting corrupted boundaries means more capacity wasted.
## Repository Contents
```
paper/
tokenizer-attention-coupling.md # Main paper source
tokenizer-attention-coupling.pdf # Main paper PDF
stranded-attention.md # Short paper source
stranded-attention.pdf # Short paper PDF
eval_*.py # 30+ evaluation/ablation scripts
runs/ # 100+ result files (JSON + logs)
charts/ # 50+ charts
domains/ # Code and chemistry domain experiments
code/results/ # Code domain results
chemistry/results/ # Chemistry domain results
structok-64k.json # Merge-barrier tokenizer
GUIDE.md # Beginner's guide to the research
RESEARCH.md # Research summary
EVAL-SCRIPTS.md # Script index with descriptions
```
## Model Checkpoints
Available on [HuggingFace](https://huggingface.co/blackwell-systems/merge-barriers):
| Model | Architecture | Tokenizer | Steps |
|-------|-------------|-----------|-------|
| NeoX A | GPT-NeoX 410M | structok-64k (merge barriers) | 20,000 |
| NeoX B | GPT-NeoX 410M | standard-64k (no barriers) | 20,000 |
| Llama A | Llama 410M (GQA 4:1) | structok-64k (merge barriers) | 40,000 |
| Llama B | Llama 410M (GQA 4:1) | standard-64k (no barriers) | 40,000 |
1.3B checkpoints available on R2 (contact for access).
## Reproducing Results
```bash
pip install torch transformers tokenizers matplotlib numpy
```
Download checkpoints from [HuggingFace](https://huggingface.co/blackwell-systems/merge-barriers), then:
```bash
# Stranded heads tokenizer swap (Experiment 28b)
python eval_stranded_swap.py --checkpoint llama-b.pt --tokenizer standard-64k.json
# Training dynamics timeline (Experiment 28c)
python eval_stranded_timeline.py --checkpoint-dir checkpoints/ --tokenizer standard-64k.json
# P0 drift analysis
python eval_p0_drift.py --checkpoint-dir checkpoints/ --tokenizer standard-64k.json
# Head ablation (18 phases)
python eval_ablation_v4_excess.py --checkpoint-a neox-a.pt --tokenizer-a structok-64k.json
# PPL comparison
python eval_model.py --checkpoint neox-a.pt --tokenizer structok-64k.json --size 410m
```
## Citation
```bibtex
@article{blackwell2026tokenizer,
title={Tokenizer-Attention Coupling: How BPE Merge Decisions Permanently Shape Transformer Internal Organization},
author={Blackwell, Dayna},
year={2026},
doi={10.5281/zenodo.20925910},
url={https://github.com/blackwell-systems/merge-barriers}
}
```
## License
MIT
## Related
- [GCF: Graph Compact Format](https://github.com/blackwell-systems/gcf) (the wire format used as comparison format, designed from the tokenizer analysis)
- [DOI: 10.5281/zenodo.20925910](https://doi.org/10.5281/zenodo.20925910)