https://github.com/swiss-ai/parity-aware-bpe

Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [arXiv 2025]

bpe llms multilingual-nlp multilingual-tokenization tokenization

Last synced: 20 Apr 2026