https://github.com/ZhuoruLin/fast-wordpiece

Implementation of fast wordpiece tokenization algorithm from https://aclanthology.org/2021.emnlp-main.160.pdf in rust
https://github.com/ZhuoruLin/fast-wordpiece

Last synced: 8 months ago
JSON representation

Implementation of fast wordpiece tokenization algorithm from https://aclanthology.org/2021.emnlp-main.160.pdf in rust

• Host: GitHub
• URL: https://github.com/ZhuoruLin/fast-wordpiece
• Owner: ZhuoruLin
• Created: 2024-03-22T15:45:02.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-03-22T19:03:12.000Z (over 1 year ago)
• Last Synced: 2024-03-22T20:24:24.359Z (over 1 year ago)
• Language: Rust
• Size: 12.7 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:

Awesome Lists containing this project

• awesome-tokenizers - ZhuoruLin/fast-wordpiece