https://github.com/jakecyr/omni-tokenizer
Node based tokenizers for open source LLMs hosted on HuggingFace.
https://github.com/jakecyr/omni-tokenizer
gemma mistral mixtral nodejs tokenizer
Last synced: 4 months ago
JSON representation
Node based tokenizers for open source LLMs hosted on HuggingFace.
- Host: GitHub
- URL: https://github.com/jakecyr/omni-tokenizer
- Owner: jakecyr
- Created: 2024-03-22T03:12:59.000Z (about 2 years ago)
- Default Branch: master
- Last Pushed: 2024-03-22T19:55:07.000Z (about 2 years ago)
- Last Synced: 2025-09-30T16:35:17.778Z (9 months ago)
- Topics: gemma, mistral, mixtral, nodejs, tokenizer
- Language: TypeScript
- Homepage:
- Size: 184 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Omni Tokenizer
A Node based tokenizer for open source models hosted on HuggingFace.
Specify the model repo and the tokenizer configuration files will be automatically downloaded from HuggingFace and used to encode provided text.
**This has only been tested on Mixtral, Mistral, and Gemma models!! Please submit issues if you encounter problems with other models or feel free to contribute.**
## Installation
```bash
npm install omni-tokenizer
```
## Usage
```ts
const { Tokenizer } = require('../src/tokenizer');
const tokenizer = new Tokenizer('mistralai/Mixtral-8x7B-Instruct-v0.1');
// Load tokenizer files from HuggingFace.
await tokenizer.load();
const text = 'hey there how are you doing';
const encoding: number[] = await tokenizer.encode(text);
console.log('Token Count', encoding.length);
const decoded: string = await tokenizer.decode(encoding);
console.log(decoded); // 'hey there how are you doing'
```