https://github.com/micycle1/count-tokens
Ultra-fast, client-side token counter for large text blobs
https://github.com/micycle1/count-tokens
bpe-tokenizer tokenisation tokenization tokenizer
Last synced: 8 months ago
JSON representation
Ultra-fast, client-side token counter for large text blobs
- Host: GitHub
- URL: https://github.com/micycle1/count-tokens
- Owner: micycle1
- Created: 2025-09-25T11:38:09.000Z (8 months ago)
- Default Branch: master
- Last Pushed: 2025-09-25T12:56:52.000Z (8 months ago)
- Last Synced: 2025-09-25T13:31:00.898Z (8 months ago)
- Topics: bpe-tokenizer, tokenisation, tokenization, tokenizer
- Language: JavaScript
- Homepage: https://micycle1.github.io/count-tokens/
- Size: 21.5 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# count-tokens
Ultra-fast, client-side token counter for large text blobs using OpenAI-style BPEs.
## What this is
- Static site (GitHub Pages) that counts tokens entirely in your browser
- Built with [bpe-openai](https://crates.io/crates/bpe-openai) compiled to WebAssembly
- Optimised for huge pastes (even repo-level): avoids per-token highlighting that makes other sites hang
- Zero backend; no data leaves your machine
## Supported encodings
- cl100k_base
- o200k_base
- voyage3_base