{"id":48894636,"url":"https://github.com/phlx0/tokenmiser","last_synced_at":"2026-04-16T10:06:19.857Z","repository":{"id":350102565,"uuid":"1205256935","full_name":"phlx0/tokenmiser","owner":"phlx0","description":"Cut Claude Code token usage 50–80%. Auto-generates a codebase index so Claude    finds files instantly instead of reading everything.","archived":false,"fork":false,"pushed_at":"2026-04-08T21:52:28.000Z","size":94,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-08T23:26:11.713Z","etag":null,"topics":["ai-tools","anthropic","claude","claude-code","cli","codebase-index","context-window","developer-tools","productivity","tokens"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/phlx0.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-08T19:41:48.000Z","updated_at":"2026-04-08T21:54:05.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/phlx0/tokenmiser","commit_stats":null,"previous_names":["phlx0/tokenmiser"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/phlx0/tokenmiser","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phlx0%2Ftokenmiser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phlx0%2Ftokenmiser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phlx0%2Ftokenmiser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phlx0%2Ftokenmiser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/phlx0","download_url":"https://codeload.github.com/phlx0/tokenmiser/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phlx0%2Ftokenmiser/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31880933,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T09:23:21.276Z","status":"ssl_error","status_checked_at":"2026-04-16T09:23:15.028Z","response_time":69,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-tools","anthropic","claude","claude-code","cli","codebase-index","context-window","developer-tools","productivity","tokens"],"created_at":"2026-04-16T10:06:19.013Z","updated_at":"2026-04-16T10:06:19.820Z","avatar_url":"https://github.com/phlx0.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tokenmiser\n\n[![npm version](https://img.shields.io/npm/v/tokenmiser.svg)](https://www.npmjs.com/package/tokenmiser)\n[![CI](https://github.com/phlx0/tokenmiser/actions/workflows/ci.yml/badge.svg)](https://github.com/phlx0/tokenmiser/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Node.js \u003e=18](https://img.shields.io/badge/node-%3E%3D18-brightgreen)](https://nodejs.org)\n\n**Cut Claude Code token usage by 50–80% per session.** Generates a compact codebase index, smart ignore rules, and an optimized `CLAUDE.md` — so Claude navigates straight to the right file instead of reading 10 files to find one function.\n\n```\n$ npx tokenmiser init\n\n  tokenmiser — slash Claude Code token usage\n\n  ✔ Scanned 134 files · ~180k tokens  (typescript)\n\n  → .claudeignore         created\n  ✔ CODEBASE_INDEX.md     112 files indexed · ~1.8k tokens\n  → CLAUDE.md             updated — index reference added\n  → Claude Code hooks     configured (.claude/settings.json)\n\n  Files in repo               134\n  Tokens without index        ~180k\n  CODEBASE_INDEX.md           ~1.8k tokens\n  CLAUDE.md overhead          ~420 tokens/session\n  Estimated token savings     ~99%\n```\n\n---\n\n## Why tokens get wasted\n\nClaude Code burns tokens in three predictable ways:\n\n1. **Blind file reading** — Claude doesn't know what's in a file without opening it, so it reads 5–10 files to find the right one. At ~1,500 tokens per file, that's 7k–15k tokens per lookup.\n2. **Irrelevant content** — `node_modules`, build artifacts, lock files, generated code. None of this helps; all of it costs.\n3. **Bloated `CLAUDE.md`** — Loaded at session start. A 3,000-token `CLAUDE.md` burns 3,000 tokens before Claude types a single character.\n\n## How tokenmiser fixes it\n\n### 1. `CODEBASE_INDEX.md` — the core fix\n\nA compact map of your entire codebase that Claude reads once instead of reading files blindly:\n\n```markdown\n# Codebase Index\n\u003e 2026-04-08 · 134 files · ~180k tokens total\n\u003e Always read this file first. Navigate to exact files rather than exploring.\n\n## Source\n\n**src/auth/**\n- `service.ts` — User authentication · AuthService, login, logout, validateToken, refreshToken\n- `middleware.ts` — authenticate, requireRole, optionalAuth\n\n**src/api/**\n- `routes.ts` — UserRouter, ProductRouter, OrderRouter\n- `controllers/userController.ts` — getUser, createUser, updateUser, deleteUser\n```\n\nInstead of reading 10 files to find a function, Claude reads this index (~1,800 tokens) and goes directly to the right file. **One read instead of ten.**\n\n### 2. `.claudeignore` — stop reading junk\n\nAutomatically excludes build artifacts, lock files, coverage reports, generated code, binaries, and media files. Language-aware: detects TypeScript, Python, Go, Rust, Java, Ruby, PHP, and adds the right patterns.\n\n### 3. `CLAUDE.md` optimisation\n\nInjects an instruction that tells Claude to always consult `CODEBASE_INDEX.md` before reading any source file. If no `CLAUDE.md` exists, creates a minimal, well-structured template.\n\n### 4. Live index updates via hooks\n\nInstalls Claude Code hooks that automatically regenerate `CODEBASE_INDEX.md` after every `Write` or `Edit` call — the index stays current without you doing anything.\n\nA `PreToolUse` hook on `Read` **blocks** full reads of files over ~1,500 lines and tells Claude to use `offset`/`limit` for targeted reads instead.\n\n---\n\n## Real numbers\n\nIndex size vs. full repo size, measured on actual projects:\n\n| Project | Language | Files | Full repo | Index only | Index reduction |\n|---|---|---|---|---|---|\n| Go CLI (ghscope) | Go | 60 | ~40.9k tokens | ~758 tokens | **98%** |\n| Next.js website | TypeScript | 37 | ~28.8k tokens | ~541 tokens | **98%** |\n| TUI app (snip) | Python | 51 | ~40.5k tokens | ~566 tokens | **99%** |\n| tokenmiser itself | TypeScript | 36 | ~22.9k tokens | ~452 tokens | **98%** |\n\nThe index reduction is how much smaller `CODEBASE_INDEX.md` is vs. reading every file. **Real per-session savings depend on how many files Claude would have read without the index** — typically 5–10 files per task. In practice that's 50–80% fewer tokens per session, compounding across every task in a conversation.\n\n---\n\n## Install\n\n```bash\n# Run directly (no install needed)\nnpx tokenmiser init\n\n# Or install globally\nnpm install -g tokenmiser\ntokenmiser init\n```\n\n**Requirements:** Node.js 18+\n\n---\n\n## Commands\n\n### `tokenmiser init`\n\nOne-time setup for a project. Run this first.\n\n```bash\ntokenmiser init [options]\n\nOptions:\n  -d, --dir \u003cpath\u003e   Repository root (default: current directory)\n  --hooks            Install Claude Code hooks for live updates (default)\n  --no-hooks         Skip hook installation\n```\n\nGenerates:\n- `.claudeignore` with language-aware patterns\n- `CODEBASE_INDEX.md` with exports and purpose for every source file\n- Injects an index reference into `CLAUDE.md` (or creates one)\n- Optionally installs hooks in `.claude/settings.json`\n\n### `tokenmiser scan`\n\nRead-only analysis. See where your tokens are going.\n\n```bash\ntokenmiser scan [-d \u003cpath\u003e]\n```\n\nShows a token heatmap, directory breakdown, index vs. full-read cost, and CLAUDE.md overhead per session.\n\n### `tokenmiser update`\n\nRegenerates `CODEBASE_INDEX.md` after code changes. Fast — uses a file mtime cache so only changed files are re-processed.\n\n```bash\ntokenmiser update [-d \u003cpath\u003e] [--quiet]\n```\n\nWire it to a git hook or npm script:\n\n```json\n{ \"scripts\": { \"postbuild\": \"tokenmiser update\" } }\n```\n\n---\n\n## What gets installed\n\n```\nyour-project/\n├── CODEBASE_INDEX.md         ← commit this\n├── .claudeignore             ← commit this\n├── CLAUDE.md                 ← updated with index instruction\n├── .claude/\n│   └── settings.json         ← hook config\n└── .tokenmiser/\n    └── cache.json            ← add to .gitignore\n```\n\nAdd `.tokenmiser/` to your `.gitignore`. Commit everything else.\n\n---\n\n## Claude Code hooks\n\n**PostToolUse on Write/Edit** — runs `tokenmiser update --quiet` after every file Claude writes or edits. The index stays current throughout the session automatically.\n\n**PreToolUse on Read** — for files over ~1,500 lines (~21k tokens), blocks the full read and outputs:\n```\n[tokenmiser] Blocked full read of src/generated/schema.ts (~2,847 lines, ~12k tokens).\nCheck CODEBASE_INDEX.md for the file map, then read only the section you need using offset/limit.\nIf you need the full file, re-issue the Read tool call with an explicit offset of 0.\n```\nClaude can still read large files in sections using `offset`/`limit`, or fully by re-issuing with `offset: 0`.\n\n---\n\n## How the index is built\n\nFor each source file, tokenmiser reads the first 32KB and extracts:\n- **Purpose** — first JSDoc comment, docstring, or leading comment\n- **Exports** — functions, classes, interfaces, types, constants (capped at 10 per file)\n\nSupported languages: TypeScript, JavaScript, Python, Go, Rust, Java, Kotlin, Ruby, PHP, C/C++.\n\nOn `tokenmiser update`, a file mtime cache means only changed files are re-processed — fast even on 1,000+ file codebases.\n\n---\n\n## Tips\n\n**Commit `CODEBASE_INDEX.md`** — it doubles as useful documentation and is immediately available to Claude without any scanning.\n\n**Add generated dirs to `.claudeignore`** — if you have a `src/generated/` directory, add it. Generated code is typically the single biggest source of token waste (often 60–80% of all tokens in a project).\n\n**Keep `CLAUDE.md` short** — under 80 lines, under 1,500 tokens. Move detailed docs to separate files that Claude reads on demand.\n\n**Large monorepos** — run `tokenmiser init` in each service directory. Each service gets its own focused index rather than one massive one.\n\n---\n\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md). PRs welcome — especially new language extractors.\n\n## License\n\nMIT © [phlx0](https://github.com/phlx0)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphlx0%2Ftokenmiser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphlx0%2Ftokenmiser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphlx0%2Ftokenmiser/lists"}