{"id":31782706,"url":"https://github.com/daaain/online-llm-tokenizer","last_synced_at":"2026-07-11T01:31:40.940Z","repository":{"id":317936821,"uuid":"1069435467","full_name":"daaain/online-llm-tokenizer","owner":"daaain","description":null,"archived":false,"fork":false,"pushed_at":"2025-10-04T12:33:41.000Z","size":228,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-30T22:42:16.105Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/daaain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-03T23:55:39.000Z","updated_at":"2025-10-16T16:08:15.000Z","dependencies_parsed_at":"2025-10-09T15:03:25.062Z","dependency_job_id":null,"html_url":"https://github.com/daaain/online-llm-tokenizer","commit_stats":null,"previous_names":["daaain/online-llm-tokenizer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/daaain/online-llm-tokenizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daaain%2Fonline-llm-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daaain%2Fonline-llm-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daaain%2Fonline-llm-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daaain%2Fonline-llm-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/daaain","download_url":"https://codeload.github.com/daaain/online-llm-tokenizer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daaain%2Fonline-llm-tokenizer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35348376,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-10T09:52:41.210Z","updated_at":"2026-07-11T01:31:40.920Z","avatar_url":"https://github.com/daaain.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Online LLM Tokenizer\n\nA pure JavaScript tokenizer running in your browser that can load `tokenizer.json` and `tokenizer_config.json` from any repository on HuggingFace. You can use it to count tokens and compare how different large language model vocabularies work. It's also useful for debugging prompt templates.\n\n## Features\n\n- **No server required**: Pure client-side tokenization using [transformers.js](https://huggingface.co/docs/transformers.js)\n- **Compare models**: Load multiple tokenizers simultaneously to see how different models tokenize the same text\n- **Visual token display**: Each token is displayed with its original text and token ID using colour-coded backgrounds\n- **Share configurations**: Generate shareable URLs with your text and model selection\n- **Persistent settings**: Model lists are saved in localStorage across browser sessions\n- **HuggingFace integration**: Load any tokenizer directly from HuggingFace Hub\n- **Dark mode support**: Automatic theme detection with appropriate colour schemes\n- **Offline capability**: Once models are loaded, tokenization works completely offline\n\n## Usage\n\n### Adding Models\n\nCopy model names from HuggingFace (e.g., from the title of model pages like \"microsoft/Phi-3-mini-4k-instruct\") and paste into the input field. Press Enter or click \"Add tokenizer from HuggingFace\".\n\n### Deleting Models\n\nClick the red \"🗑️ Delete\" button next to any model. You'll get a confirmation prompt and cannot delete the last model.\n\n### Sharing Configurations\n\nClick the \"📋 Share\" button to copy a URL containing your current text and model selection. Share this URL with others to let them see the same tokenization.\n\n### URL Parameters\n\nYou can link directly to specific configurations using URL parameters:\n```\n?text=your%20text\u0026models=model1,model2,model3\n```\n\n### Token Display\n\n- Each word/subword piece shows the original text above and the token number below\n- Different background colours help distinguish adjacent tokens (cycling through 10 colours)\n- Newlines are preserved in the display\n\n## Implementation Details\n\n- **Parallel model loading**: All tokenizers load simultaneously using `Promise.all()` instead of sequentially, to improve startup time\n- **Progressive rendering**: Models appear and update individually as they finish loading, providing immediate feedback\n- **Debounced input processing**: Text changes are debounced by 300ms to prevent excessive re-tokenization during typing\n- **Ruby annotations**: Tokens are displayed using HTML `\u003cruby\u003e` elements with text above and token numbers below\n- **Space preservation**: Automatically detects and removes tokenizer space-stripping to accurately show whitespace tokens\n- **Memory management**: Models are cached in memory and only loaded once, with cleanup on deletion\n\n## Development\n\nThe project consists of three main files:\n\n- `index.html` - Main HTML structure and UI\n- `tokenizer.css` - Styling including dark mode support\n- `tokenizer.js` - Core tokenization logic using transformers.js\n\n### Local Development\n\nSimply open `index.html` in a modern web browser. No build step required.\n\n### Dependencies\n\n- [transformers.js](https://huggingface.co/docs/transformers.js) - Loaded as an ES module for client-side tokenization\n\n## Browser Compatibility\n\nWorks in all modern browsers that support:\n- ES6 modules\n- Async/await\n- LocalStorage\n- Clipboard API (for share functionality)\n\n## Why So Many Xenova Models?\n\nIf you're wondering why there are so many models under Xenova, it's because they work for HuggingFace and re-upload just the tokenizers, so it's possible to load them without agreeing to model licences.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## Licence\n\nSee [LICENCE](LICENCE) file for details.\n\n## Live Demo\n\nTry it at: [danieldemmel.me/tokenizer](https://www.danieldemmel.me/tokenizer.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaaain%2Fonline-llm-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdaaain%2Fonline-llm-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaaain%2Fonline-llm-tokenizer/lists"}