{"id":50873944,"url":"https://github.com/yttrium400/reducethemtokens","last_synced_at":"2026-06-15T07:36:16.606Z","repository":{"id":357969548,"uuid":"1238365150","full_name":"yttrium400/reducethemtokens","owner":"yttrium400","description":"Compress any code repo into a compact skeleton to reduce LLM token usage. 90%+ reduction with full structural retention.","archived":false,"fork":false,"pushed_at":"2026-05-15T04:33:53.000Z","size":621,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-15T04:40:31.212Z","etag":null,"topics":["ai-tools","claude","code-analysis","context-window","cursor","developer-tools","llm","python","token-reduction","tree-sitter"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yttrium400.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-14T03:56:06.000Z","updated_at":"2026-05-15T04:33:57.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/yttrium400/reducethemtokens","commit_stats":null,"previous_names":["yttrium400/reducethemtokens"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/yttrium400/reducethemtokens","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yttrium400%2Freducethemtokens","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yttrium400%2Freducethemtokens/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yttrium400%2Freducethemtokens/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yttrium400%2Freducethemtokens/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yttrium400","download_url":"https://codeload.github.com/yttrium400/reducethemtokens/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yttrium400%2Freducethemtokens/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34353193,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-15T02:00:07.085Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-tools","claude","code-analysis","context-window","cursor","developer-tools","llm","python","token-reduction","tree-sitter"],"created_at":"2026-06-15T07:36:11.903Z","updated_at":"2026-06-15T07:36:16.599Z","avatar_url":"https://github.com/yttrium400.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# rtt - reducethemtokens\n\n[![PyPI](https://img.shields.io/pypi/v/reducethemtokens)](https://pypi.org/project/reducethemtokens/)\n[![Python](https://img.shields.io/pypi/pyversions/reducethemtokens)](https://pypi.org/project/reducethemtokens/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n\nGive any LLM a complete map of your codebase in a single, cheap read.\n\n![rtt demo](demo.gif)\n\n`rtt` extracts every file's imports, function signatures, class hierarchies, and method\nlists into a compact plain-text skeleton - typically 90% smaller than the raw source -\nand wires it into your agent's config so the map is available from the first message of\nevery session.\n\n---\n\n## The problem it solves\n\nModern coding agents (Cursor, Claude Code, Copilot) are good at retrieving context for\n*specific, targeted queries*. Ask about one bug, one function, one file - they find it.\n\nBut they struggle with the **orientation problem**: starting a session on an unfamiliar\ncodebase, or asking questions that span the whole structure. Before an agent can retrieve\nthe right context, it needs to understand what exists and where. Without that map, it\neither scans files speculatively (burning tokens) or makes wrong assumptions about\nstructure.\n\n`rtt` solves this by providing that map upfront, once, cheaply. The agent reads the\nskeleton at session start, knows the full API surface, and then opens only the specific\nfiles it actually needs.\n\n**rtt is not a replacement for agent retrieval.** Retrieval is better for targeted,\nimplementation-level tasks. rtt is the orientation layer that makes retrieval more\naccurate by giving the agent the right mental model before it starts searching.\n\n---\n\n## Where agents save tokens\n\nEvery code-change task follows the same four steps:\n\n```\n 1. Task arrives       2. Navigate          3. Read              4. Write\n \"add a rate-limit  →  find the right    →  open that file   →  make the edit\n  endpoint\"            file to edit         in full\n                          ↑\n                    tokens wasted here\n                    without a map\n```\n\n**Step 2 is where agents burn tokens unnecessarily.** Without a structural map, an agent scans speculatively - opening files that turn out to be wrong before landing on the right one. With rtt, step 2 becomes a single skeleton lookup: the agent sees every file's exports, imports, and signatures upfront, identifies the target directly, and skips the exploratory reads.\n\nStep 3 always happens - the agent needs the function body to write a correct edit. rtt does not replace that.\n\n### Measured on a real private codebase\n\nWe ran the same 5 code-change navigation tasks twice on a **246-file TypeScript/Next.js** repo - once with no prior context, once with the rtt skeleton (18,149 tokens) prepended:\n\n| | No skeleton | With skeleton |\n|---|---|---|\n| File reads | 16 | **7** |\n| Total tool calls | 22 | **14** |\n\n**56% fewer file reads. 36% fewer total tool calls.** Several navigation tasks were answered entirely from the skeleton - no file opened at all - while the remaining reads went directly to the right file.\n\n### Why larger repos save more\n\nOn a small 50-file repo, an agent can often guess the right file from its name alone. On a 500-file repo it cannot - the exploratory tax grows with surface area. The skeleton overhead scales linearly with file count, but the number of prevented speculative reads grows faster. A rough model:\n\n| Repo size | Skeleton overhead | Est. reads saved per session | Break-even |\n|---|---|---|---|\n| 50 files | ~2k tokens | 1–2 reads | immediate |\n| 250 files | ~18k tokens | 5–10 reads | first session |\n| 1,000+ files | ~60k tokens | 20+ reads | first session |\n\nEach prevented file read avoids loading that file's full source into context for the rest of the session. On TypeScript/Python files averaging 200–500 lines, that is 1,000–4,000 tokens per read. The skeleton pays for itself once it prevents 4–6 exploratory reads - which typically happens in a single task on any repo over 200 files.\n\n---\n\n## When to use it\n\n**Use rtt when:**\n\n- Starting a session on a codebase the agent hasn't seen before\n- The task involves cross-cutting changes across many files (a refactor, a rename, adding\n  a feature that touches multiple layers)\n- You're using a chat interface (ChatGPT, Claude.ai, direct API) that has no built-in\n  retrieval - every session starts from zero\n- You're building a CI pipeline, a code review bot, or any automated workflow where\n  reproducible, deterministic context matters\n- You want to give an LLM repo context without setting up a vector store or any\n  additional infrastructure\n\n**rtt is less useful when:**\n\n- You're asking about one specific file or function - just open it\n- Your agent already has full retrieval and you're working on targeted, well-scoped tasks\n\n---\n\n## Installation\n\n```\npip install reducethemtokens\n```\n\nRequires Python 3.10+.\n\n---\n\n## Quick start\n\n```\ncd your-repo\nrtt install .\n```\n\nThis writes `.rtt/context.txt` (the skeleton) and adds a short instruction to every\nsupported agent config file - `CLAUDE.md`, `AGENTS.md`, `.cursor/rules/`, and others.\nThe instruction tells the agent to read the skeleton at session start for orientation,\nthen work normally from there.\n\nCommit both files. Every collaborator and every future session gets the map automatically.\n\n```\n# After code changes - regenerate the skeleton\nrtt update .\n\n# See how many tokens the skeleton saves vs raw source\nrtt compare .\n```\n\n**Sample skeleton output for one file:**\n\n```\n# rtt/formatter.py [python]\nimports: datetime, json, dataclasses.asdict, rtt.RepoIndex, rtt.FileIndex, rtt.Symbol\ndef format_text(repo: RepoIndex) -\u003e str\ndef format_json(repo: RepoIndex) -\u003e str\ndef format_text_with_header(repo: RepoIndex, token_count: int) -\u003e str\ndef format_file_text(file_index: FileIndex) -\u003e str\ndef format_markdown(repo: RepoIndex) -\u003e str\n```\n\n---\n\n## Benchmark - Django (3,020 files)\n\n| Metric | Value |\n|---|---|\n| Raw codebase | 6,464,961 tokens |\n| rtt skeleton | 585,421 tokens |\n| Reduction | **90.9%** |\n| Audit coverage (Python) | **99.9%** (34,454 / 34,480 symbols) |\n| Audit coverage (JavaScript) | **97.9%** (46 / 47 symbols) |\n\n---\n\n## Commands\n\n### `rtt install`\n\nIndex the repo, write the skeleton to `.rtt/context.txt`, and inject orientation\ninstructions into every supported agent config file. Also installs a git pre-commit hook\nthat regenerates the skeleton automatically on every commit.\n\n```\nrtt install .\nrtt install . --platform claude    # single agent only\nrtt install . --force              # overwrite existing rtt sections\nrtt install . --no-tests           # exclude test/spec/fixture files\nrtt install . --max-tokens 100000  # trim to fit a context window budget\nrtt install . --include 'src/**'   # only index specific directories\nrtt install . --exclude 'vendor/**'\n```\n\nSupported agents:\n\n| Agent | Config file |\n|---|---|\n| Claude Code | `CLAUDE.md` |\n| Cursor | `.cursor/rules/rtt.mdc` |\n| Windsurf | `.windsurfrules` |\n| Codex / OpenAI | `AGENTS.md` |\n| GitHub Copilot | `.github/copilot-instructions.md` |\n| Kiro | `.kiro/steering/rtt.md` |\n| Gemini CLI | `GEMINI.md` |\n| Aider | `.aider/prompts/conventions.md` |\n| Zed | `.rules` |\n| Continue.dev | `.continue/rules/rtt.md` |\n\nThe instruction added to each config file tells the agent to read `.rtt/context.txt`\nonce at session start for orientation, then work normally. It does not restrict the\nagent from reading source files or using its own retrieval.\n\n### `rtt update`\n\nRegenerate `.rtt/context.txt` after code changes. Does not touch agent config files.\nThe git hook installed by `rtt install` runs this automatically on every commit.\n\n```\nrtt update .\nrtt update . --diff        # show what symbols changed\nrtt update . --no-tests    # same flags as install are accepted\n```\n\n### `rtt uninstall`\n\nRemove rtt instructions from all agent config files.\n\n```\nrtt uninstall .\nrtt uninstall . --platform cursor\nrtt uninstall . --clean    # also delete .rtt/context.txt\n```\n\n### `rtt index`\n\nGenerate the skeleton and print to stdout, or write to a file. Useful for piping\ninto other tools or building custom workflows.\n\n```\nrtt index .\nrtt index /path/to/repo --output context.txt\nrtt index . --no-tests\nrtt index . --include 'src/**' --include 'lib/**'\nrtt index . --exclude 'vendor/**' --exclude 'generated/**'\nrtt index . --max-tokens 50000\n```\n\n### `rtt compare`\n\nShow token reduction statistics with a per-file breakdown.\n\n```\nrtt compare .\nrtt compare . --diff HEAD~3..HEAD    # token delta for a git range\n```\n\n### `rtt audit`\n\nVerify extraction accuracy: symbols found vs expected, and signature correctness.\n\n```\nrtt audit .\n```\n\n### `rtt vs`\n\nCompare token footprint against another repo-indexing tool (currently supports graphify).\n\n```\npip install graphifyy\nrtt vs .\n```\n\n### `rtt view`\n\nRender the skeleton as markdown and open in a pager.\n\n```\nrtt view .\nrtt view . --output overview.md\n```\n\n---\n\n## Keeping the skeleton current\n\n`rtt install` sets up a git pre-commit hook that runs `rtt update` automatically\non every commit. For most solo workflows that is enough.\n\n**For teams**, the hook only runs on machines where rtt is installed. A new\ncontributor who clones the repo without installing rtt will not regenerate the\nskeleton. Two approaches:\n\nAdd a CI step that regenerates and commits the skeleton on every merge to main:\n\n```yaml\n# .github/workflows/rtt.yml\nname: Update rtt index\non:\n  push:\n    branches: [main]\njobs:\n  rtt:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - run: pip install reducethemtokens\n      - run: rtt update .\n      - uses: stefanzweifel/git-auto-commit-action@v5\n        with:\n          commit_message: \"chore: update rtt index\"\n          file_pattern: \".rtt/context.txt\"\n```\n\nOr document it in your contributing guide:\n\n```\n# after pulling changes\nrtt update .\ngit add .rtt/context.txt\n```\n\nThe first line of `.rtt/context.txt` includes the generation timestamp and file\ncount, so agents can detect a stale index without reading the whole file.\n\n**Large repos and context window limits**\n\nTest files are usually the biggest contributor to skeleton size. The simplest\nreduction for most projects is `--no-tests`:\n\n```\nrtt install . --no-tests    # drops test/, spec/, fixture/ files\n```\n\nOn Django (3,020 files), this alone cuts the skeleton from 585k tokens to 193k.\n\nIf the skeleton is still too large, use `--max-tokens` to trim it to fit.\nrtt keeps non-test files with the most symbols and drops the rest:\n\n```\nrtt install . --max-tokens 100000    # fits in most 128k-window models\nrtt install . --max-tokens 50000     # conservative\n```\n\nA rough guide by repo size:\n\n| Repo scale | Approach |\n|---|---|\n| \u003c 500 files | no flag needed |\n| 500–2,000 files | `--no-tests` |\n| 2,000+ files (e.g. Django) | `--no-tests` + `--max-tokens 100000` |\n\nOr limit to specific directories:\n\n```\nrtt install . --include 'src/**' --include 'lib/**'\nrtt install . --exclude 'vendor/**' --exclude 'generated/**'\n```\n\n---\n\n## Python API\n\n```python\nimport rtt\n\nrepo = rtt.index(\"/path/to/repo\")\nrepo = rtt.index(\"/path/to/repo\", no_tests=True)\nrepo = rtt.index(\"/path/to/repo\", max_tokens=100000)\nrepo = rtt.index(\"/path/to/repo\", include=[\"src/**\"], exclude=[\"vendor/**\"])\n\nprint(repo.token_count)    # int\nprint(repo.text)           # full skeleton as a string\n\nfor file in repo.files:\n    print(file.path, file.language)\n    print(file.imports)    # e.g. [\"pathlib.Path\", \"typing.Optional\"]\n    for sym in file.symbols:\n        print(sym.name, sym.kind, sym.signature)\n        for child in sym.children:\n            print(\" \", child.signature)\n\nreport = rtt.compare(\"/path/to/repo\")\nprint(f\"{report.reduction_pct:.1f}% reduction\")\nprint(f\"{report.raw_tokens:,} → {report.compressed_tokens:,} tokens\")\n```\n\n---\n\n## Supported languages\n\nPython, JavaScript, TypeScript, Go, Rust, Java, C, C++, Ruby, Swift, Kotlin, C#, Lua, Dart, Scala.\n\n---\n\n## How it works\n\n`rtt` parses each file with [tree-sitter](https://tree-sitter.github.io/tree-sitter/)\nand walks the AST to collect top-level definitions: functions, classes, methods, and\nimports. Function bodies are discarded. The output is one line per symbol, indented to\nshow class membership, with imports resolved to specific symbols\n(`from pathlib import Path` → `pathlib.Path`).\n\nResults are cached by file content hash. Subsequent runs on large repos are fast.\n\n---\n\n## Development\n\n```\ngit clone https://github.com/yttrium400/reducethemtokens\ncd reducethemtokens\npip install -e \".[dev]\"\npytest tests/\n```\n\n91 tests. No network calls required.\n\n---\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyttrium400%2Freducethemtokens","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyttrium400%2Freducethemtokens","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyttrium400%2Freducethemtokens/lists"}