Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/vercetti322/tokenizer
This is a tool which parses raw strings of natural language and generate context-aware tokens which are used for LLM training!
https://github.com/vercetti322/tokenizer
Last synced: 6 days ago
JSON representation
This is a tool which parses raw strings of natural language and generate context-aware tokens which are used for LLM training!
- Host: GitHub
- URL: https://github.com/vercetti322/tokenizer
- Owner: vercetti322
- License: mit
- Created: 2024-04-07T15:11:36.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2024-04-07T16:36:14.000Z (8 months ago)
- Last Synced: 2024-04-07T17:36:51.455Z (8 months ago)
- Language: Jupyter Notebook
- Size: 3.91 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE