https://github.com/shivendrra/shredword-trainer
BPE & Unigram Vocab Training library
https://github.com/shivendrra/shredword-trainer
natural-language-processing subword-tokenization tokenization tokenizer
Last synced: 3 months ago
JSON representation
BPE & Unigram Vocab Training library
- Host: GitHub
- URL: https://github.com/shivendrra/shredword-trainer
- Owner: shivendrra
- License: mit
- Created: 2024-08-29T05:34:00.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2025-12-18T20:16:01.000Z (7 months ago)
- Last Synced: 2025-12-21T16:37:34.739Z (7 months ago)
- Topics: natural-language-processing, subword-tokenization, tokenization, tokenizer
- Language: C++
- Homepage: https://pypi.org/project/shredword-trainer/
- Size: 18.2 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE