https://github.com/jakecyr/omni-tokenizer
Node based tokenizers for open source LLMs hosted on HuggingFace.
https://github.com/jakecyr/omni-tokenizer
gemma mistral mixtral nodejs tokenizer
Last synced: about 1 month ago
JSON representation
Node based tokenizers for open source LLMs hosted on HuggingFace.
- Host: GitHub
- URL: https://github.com/jakecyr/omni-tokenizer
- Owner: jakecyr
- Created: 2024-03-22T03:12:59.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2024-03-22T19:55:07.000Z (over 1 year ago)
- Last Synced: 2024-04-24T13:29:50.154Z (over 1 year ago)
- Topics: gemma, mistral, mixtral, nodejs, tokenizer
- Language: TypeScript
- Homepage:
- Size: 184 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Omni Tokenizer
A Node based tokenizer for open source models hosted on HuggingFace.
Specify the model repo and the tokenizer configuration files will be automatically downloaded from HuggingFace and used to encode provided text.
**This has only been tested on Mixtral, Mistral, and Gemma models!! Please submit issues if you encounter problems with other models or feel free to contribute.**
## Installation
```bash
npm install omni-tokenizer
```
## Usage
```ts
const { Tokenizer } = require('../src/tokenizer');
const tokenizer = new Tokenizer('mistralai/Mixtral-8x7B-Instruct-v0.1');
// Load tokenizer files from HuggingFace.
await tokenizer.load();
const text = 'hey there how are you doing';
const encoding: number[] = await tokenizer.encode(text);
console.log('Token Count', encoding.length);
const decoded: string = await tokenizer.decode(encoding);
console.log(decoded); // 'hey there how are you doing'
```