https://github.com/mideind/neural-tokenizers

Repository contains a python package for using segmentation methods built on top of (small) transformer language models.
https://github.com/mideind/neural-tokenizers

Last synced: about 1 year ago
JSON representation

Repository contains a python package for using segmentation methods built on top of (small) transformer language models.

• Host: GitHub
• URL: https://github.com/mideind/neural-tokenizers
• Owner: mideind
• License: gpl-3.0
• Created: 2025-04-23T12:59:36.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2025-05-23T12:07:45.000Z (about 1 year ago)
• Last Synced: 2025-05-23T13:45:01.923Z (about 1 year ago)
• Language: Python
• Size: 364 KB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# neural-tokenizers

Repository contains a python package for using segmentation methods built on top of (small) transformer language models.