Projects in Awesome Lists tagged with subword-tokenization
A curated list of projects in awesome lists tagged with subword-tokenization .
https://github.com/blackwell-systems/merge-barriers
Merge Barriers in BPE Tokenization: How Tokenizer Design Causally Determines Attention Head Specialization. Paper, 23 eval scripts, 86 result files, tokenizer definitions, 4 model checkpoints.
architecture-independence attention-heads attention-mechanism bpe causal-ablation deep-learning delimiter-specialization gpt-neox gqa llama llm mechanistic-interpretability merge-barriers nlp reproducible-research sentencepiece structured-data subword-tokenization tokenization transformer
Last synced: 08 Jul 2026
https://github.com/shivendrra/shredword-trainer
BPE & Unigram Vocab Training library
natural-language-processing subword-tokenization tokenization tokenizer
Last synced: 30 Apr 2026
