https://github.com/bigcode-project/bigcode-tokenizer

https://github.com/bigcode-project/bigcode-tokenizer

Last synced: about 2 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/bigcode-project/bigcode-tokenizer
• Owner: bigcode-project
• License: apache-2.0
• Created: 2023-03-06T08:17:52.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2023-10-24T09:24:28.000Z (over 2 years ago)
• Last Synced: 2025-09-09T16:08:51.627Z (5 months ago)
• Language: Jupyter Notebook
• Size: 128 KB
• Stars: 15
• Watchers: 3
• Forks: 3
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# bigcode-tokenizer

Tokenizer training and analysis