https://github.com/zouharvi/formal-bpe

https://github.com/zouharvi/formal-bpe

bpe byte-pair-encoding tokenization

Last synced: 7 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/zouharvi/formal-bpe
• Owner: zouharvi
• Created: 2023-01-17T17:08:31.000Z (over 3 years ago)
• Default Branch: master
• Last Pushed: 2025-01-30T09:31:28.000Z (over 1 year ago)
• Last Synced: 2025-02-25T17:31:24.592Z (over 1 year ago)
• Topics: bpe, byte-pair-encoding, tokenization
• Language: Python
• Homepage:
• Size: 491 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# A Formal Perspective on Byte-Pair Encoding

[![Paper](https://img.shields.io/badge/📜%20paper-481.svg)](https://aclanthology.org/2023.findings-acl.38/)

 

[![YouTube video](https://img.shields.io/badge/🎥%20YouTube%20video-F00.svg)](https://www.youtube.com/watch?v=yeEZpf4BlDA)

> **Abstract:** Byte-Pair Encoding (BPE) is a popular algorithm used for tokenizing data in NLP, despite being devised initially as a compression method.BPE appears to be a greedy algorithm at face value, but the underlying optimization problem that BPE seeks to solve has not yet been laid down. We formalize BPE as a combinatorial optimization problem. Via submodular functions, we prove that the iterative greedy version is a 1/sigma*(1-e(-sigma))-approximation of an optimal merge sequence, where sigma is the total backward curvature with respect to the optimal merge sequence. Empirically the lower bound of the approximation is approx0.37.We provide a faster implementation of BPE which improves the runtime complexity from O(NM) to O(N log M), where N is the sequence length and M is the merge count. Finally, we optimize the brute-force algorithm for optimal BPE using memoization.

- read the [paper here](https://aclanthology.org/2023.findings-acl.38/),

- watch a [3 min video](https://www.youtube.com/watch?v=aB7oaS0rlvI),

- watch a [13-minute video](https://www.youtube.com/watch?v=yeEZpf4BlDA) of the paper and two others,

- or take a look [at the poster](meta/poster.pdf).

Cite as:

```

@inproceedings{zouhar2023formal, 

    title={A Formal Perspective on Byte-Pair Encoding},

    author={Zouhar, Vilém and Meister, Clara and Gastaldi, Juan Luis and Sachan, Du, Li and Vieira, Tim and Mrinmaya and Cotterell, Ryan},

    booktitle={Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Findings)},

    year={2023},

    url={https://aclanthology.org/2023.findings-acl.38/},

}

```