{"id":20464275,"url":"https://github.com/endlessreform/token-counter","last_synced_at":"2025-07-18T20:37:07.404Z","repository":{"id":246850417,"uuid":"824319934","full_name":"EndlessReform/token-counter","owner":"EndlessReform","description":"`wc` for tokens, using HuggingFace Tokenizers in Rust","archived":false,"fork":false,"pushed_at":"2024-07-04T21:38:27.000Z","size":13,"stargazers_count":7,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-14T21:59:23.703Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EndlessReform.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-04T21:25:26.000Z","updated_at":"2025-07-07T08:08:40.000Z","dependencies_parsed_at":"2024-07-05T14:06:19.885Z","dependency_job_id":null,"html_url":"https://github.com/EndlessReform/token-counter","commit_stats":null,"previous_names":["endlessreform/token-counter"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/EndlessReform/token-counter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndlessReform%2Ftoken-counter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndlessReform%2Ftoken-counter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndlessReform%2Ftoken-counter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndlessReform%2Ftoken-counter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EndlessReform","download_url":"https://codeload.github.com/EndlessReform/token-counter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndlessReform%2Ftoken-counter/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265829212,"owners_count":23835090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T13:14:31.497Z","updated_at":"2025-07-18T20:37:07.373Z","avatar_url":"https://github.com/EndlessReform.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tc - Token Count\n\n`tc` is a CLI tool for counting tokens in text files, as a lightweight wrapper around the HuggingFace [Tokenizers](https://docs.rs/tokenizers/latest/tokenizers/) crate. It's like the Unix `wc` command, but for tokens instead of words.\n\n## Features\n\n- Count tokens in files or from stdin\n- Support for multiple files and glob patterns\n- Uses any tokenizer in HuggingFace Tokenizers\n\n## Installation\n\n```\ncargo install token-counter\n```\n\n### Usage\n\nUsing default tokenizer ([cl100k](https://huggingface.co/DWDMaiMai/tiktoken_cl100k_base), the tokenizer for GPT-3.5 and GPT-4):\n\n```\ntc file1.md file2.md\n```\n\nUsing globs:\n\n```\ntc *.md\n```\n\nArguments:\n\n- `-m`, `--model`: HuggingFace ID of the model for tokenizer (ex. `google-bert/bert-base-uncased`)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fendlessreform%2Ftoken-counter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fendlessreform%2Ftoken-counter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fendlessreform%2Ftoken-counter/lists"}