{"id":20645973,"url":"https://github.com/jakecyr/omni-tokenizer","last_synced_at":"2026-02-12T05:04:23.266Z","repository":{"id":229105256,"uuid":"775785089","full_name":"jakecyr/omni-tokenizer","owner":"jakecyr","description":"Node based tokenizers for open source LLMs hosted on HuggingFace.","archived":false,"fork":false,"pushed_at":"2024-03-22T19:55:07.000Z","size":188,"stargazers_count":2,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-09-30T16:35:17.778Z","etag":null,"topics":["gemma","mistral","mixtral","nodejs","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jakecyr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-22T03:12:59.000Z","updated_at":"2024-03-22T13:38:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"b9b17ef4-031a-436d-8708-13aa415382f5","html_url":"https://github.com/jakecyr/omni-tokenizer","commit_stats":null,"previous_names":["jakecyr/omni-tokenizer"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/jakecyr/omni-tokenizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakecyr%2Fomni-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakecyr%2Fomni-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakecyr%2Fomni-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakecyr%2Fomni-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jakecyr","download_url":"https://codeload.github.com/jakecyr/omni-tokenizer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakecyr%2Fomni-tokenizer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29359360,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-12T01:03:07.613Z","status":"online","status_checked_at":"2026-02-12T02:00:06.911Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gemma","mistral","mixtral","nodejs","tokenizer"],"created_at":"2024-11-16T16:22:52.619Z","updated_at":"2026-02-12T05:04:23.248Z","avatar_url":"https://github.com/jakecyr.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Omni Tokenizer\n\nA Node based tokenizer for open source models hosted on HuggingFace.\n\nSpecify the model repo and the tokenizer configuration files will be automatically downloaded from HuggingFace and used to encode provided text.\n\n**This has only been tested on Mixtral, Mistral, and Gemma models!! Please submit issues if you encounter problems with other models or feel free to contribute.**\n\n## Installation\n\n```bash\nnpm install omni-tokenizer\n```\n\n## Usage\n\n```ts\nconst { Tokenizer } = require('../src/tokenizer');\n\nconst tokenizer = new Tokenizer('mistralai/Mixtral-8x7B-Instruct-v0.1');\n\n// Load tokenizer files from HuggingFace.\nawait tokenizer.load();\n\nconst text = 'hey there how are you doing';\nconst encoding: number[] = await tokenizer.encode(text);\n\nconsole.log('Token Count', encoding.length);\n\nconst decoded: string = await tokenizer.decode(encoding);\n\nconsole.log(decoded); // 'hey there how are you doing'\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjakecyr%2Fomni-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjakecyr%2Fomni-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjakecyr%2Fomni-tokenizer/lists"}