https://github.com/lenml/llama-tokenizer-playground
@lenml/llama2-tokenizer playground
https://github.com/lenml/llama-tokenizer-playground
Last synced: about 17 hours ago
JSON representation
@lenml/llama2-tokenizer playground
- Host: GitHub
- URL: https://github.com/lenml/llama-tokenizer-playground
- Owner: lenML
- License: mit
- Created: 2024-01-08T16:21:40.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-07-20T07:54:15.000Z (almost 2 years ago)
- Last Synced: 2025-02-28T23:44:17.088Z (over 1 year ago)
- Language: TypeScript
- Homepage: https://lenml.github.io/llama-tokenizer-playground/
- Size: 7.79 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# :warning: IMPORTANT UPDATE :warning:
Our library `@lenml/llama2-tokenizer` has been deprecated. We are excited to introduce our new library `@lenml/tokenizers` as its replacement, offering a broader set of features and an enhanced experience.
## Why switch to `@lenml/tokenizers`?
- **Fully Compatible with `transformers.js` Interfaces**: Seamlessly supports all interfaces defined in `transformers.js`, making migration and integration effortless.
- **Support for a Wide Range of Models**: Regardless of which model you need, our new library supports it, ensuring broader coverage.
- **Rich Feature Implementation**: Includes a complete implementation of chat templates and normalizers to better serve your text processing and tokenization needs.
check out [lenML/tokenizers](https://github.com/lenML/tokenizers).
# llama-tokenizer-playground
here is demo for [llama2-tokenizer.js package](https://github.com/lenML/llama2-tokenizer.js)
deploy on [github pages](https://lenml.github.io/llama-tokenizer-playground/)
# TODOs
- [x] dark theme
- [ ] tokenizer comparer
- [ ] fix react-highlight-within-textarea `text style` (when paste) issue