https://github.com/lwch/tokenizer

bpe tokenizer
https://github.com/lwch/tokenizer

Last synced: 2 months ago
JSON representation

bpe tokenizer

• Host: GitHub
• URL: https://github.com/lwch/tokenizer
• Owner: lwch
• License: mit
• Created: 2024-01-25T07:40:40.000Z (over 1 year ago)
• Default Branch: master
• Last Pushed: 2024-03-20T02:41:38.000Z (about 1 year ago)
• Last Synced: 2025-01-03T17:26:00.218Z (4 months ago)
• Language: Go
• Size: 58.6 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# tokenizer

bpe tokenizer