{"id":50335079,"url":"https://github.com/mrgray17/opentoken","last_synced_at":"2026-05-29T13:00:31.643Z","repository":{"id":358974746,"uuid":"1243947630","full_name":"MrGray17/opentoken","owner":"MrGray17","description":"Token-saving companion for OpenCode — 42 compression layers, zero risk, no caveman speak","archived":false,"fork":false,"pushed_at":"2026-05-27T00:05:25.000Z","size":510,"stargazers_count":71,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-27T01:21:47.806Z","etag":null,"topics":["compression","llm","opencode","opencode-plugin","token-savings"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/@mrgray17/opentoken","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MrGray17.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-19T20:26:39.000Z","updated_at":"2026-05-27T00:27:41.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/MrGray17/opentoken","commit_stats":null,"previous_names":["mrgray17/opentoken"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/MrGray17/opentoken","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrGray17%2Fopentoken","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrGray17%2Fopentoken/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrGray17%2Fopentoken/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrGray17%2Fopentoken/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MrGray17","download_url":"https://codeload.github.com/MrGray17/opentoken/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrGray17%2Fopentoken/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33652986,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","llm","opencode","opencode-plugin","token-savings"],"created_at":"2026-05-29T13:00:26.099Z","updated_at":"2026-05-29T13:00:31.637Z","avatar_url":"https://github.com/MrGray17.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/MrGray17/opentoken/main/.github/opentoken-banner.svg\" alt=\"OpenToken\" width=\"600\" /\u003e\n\n# ⚡ OpenToken\n\n\u003ch3\u003eThe universal token-compression engine for AI coding agents.\u003c/h3\u003e\n\n\u003cp\u003e\n  \u003cstrong\u003eEvery tool output. Every shell. Every IDE.\u003c/strong\u003e\u003cbr /\u003e\n  35 stages of lossless compression. Same semantics. 50–80% fewer tokens. Zero risk.\n\u003c/p\u003e\n\n[![npm](https://img.shields.io/npm/v/@mrgray17/opentoken?color=0366d6\u0026label=opencode)](https://www.npmjs.com/package/@mrgray17/opentoken)\n[![npm](https://img.shields.io/npm/v/@mrgray17/opentoken-cli?color=f97316\u0026label=cli)](https://www.npmjs.com/package/@mrgray17/opentoken-cli)\n[![npm](https://img.shields.io/npm/v/@mrgray17/opentoken-core?color=8b5cf6\u0026label=core)](https://www.npmjs.com/package/@mrgray17/opentoken-core)\n[![CI](https://img.shields.io/badge/tests-431%20pass-22c55e)](https://github.com/MrGray17/opentoken/actions)\n[![Bun](https://img.shields.io/badge/bun-≥1.2.0-fbb744)](https://bun.sh)\n[![license](https://img.shields.io/badge/license-MIT-334155)](LICENSE)\n\n\u003cbr /\u003e\n\n**5M+ tokens saved** · 74% compression · 431 tests · 0 regressions\n\n\u003c/div\u003e\n\n---\n\n## Install\n\n```bash\n# OpenCode plugin — auto-loads, zero config\nopencode plugin @mrgray17/opentoken@latest --global\n```\n\n```bash\n# MCP server — for VS Code, Cursor, Windsurf, Claude, etc.\nnpm install -g @mrgray17/opentoken-mcp\n```\n\n```bash\n# CLI — run anywhere, no install needed\nbun x @mrgray17/opentoken-cli wrap git diff HEAD~1\n\n# install globally (optional)\nbun install -g @mrgray17/opentoken-cli\n```\n\n```bash\n# Library — build your own compression pipeline\nnpm install @mrgray17/opentoken-core\n```\n\n**Bun v1.2+ required.** [Install Bun](https://bun.sh) — one command, 2 seconds.\n\n---\n\n## 10-Second Start\n\n```bash\n# Pipe any command output through the compression engine\ngit diff HEAD~1 | opentoken -t bash -c \"git diff HEAD~1\"\n\n# Or wrap it — same result, cleaner syntax\nopentoken wrap cargo build --release\n\n# Check your savings\nopentoken stats\n```\n\n### In your AI coding agent (MCP)\n\nInstall once:\n```bash\nnpm install -g @mrgray17/opentoken-mcp\n```\n\nThen add to your IDE's MCP config:\n\n\u003cdetails open\u003e\n\u003csummary\u003e\u003cb\u003eCursor / Windsurf / Claude Desktop\u003c/b\u003e\u003c/summary\u003e\n\n```json\n// ~/.cursor/mcp.json  or  ~/.windsurf/mcp.json\n{\n  \"mcpServers\": {\n    \"opentoken\": {\n      \"command\": \"opentoken-mcp\"\n    }\n  }\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eVS Code (Copilot / GitHub Chat)\u003c/b\u003e\u003c/summary\u003e\n\n```json\n// .vscode/mcp.json  or  ~/.vscode/mcp.json\n{\n  \"servers\": {\n    \"opentoken\": {\n      \"type\": \"stdio\",\n      \"command\": \"opentoken-mcp\"\n    }\n  }\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eClaude Code (CLI)\u003c/b\u003e\u003c/summary\u003e\n\n```json\n// ~/.claude/claude_desktop_config.json  or  .claude/mcp.json\n{\n  \"mcpServers\": {\n    \"opentoken\": {\n      \"command\": \"opentoken-mcp\"\n    }\n  }\n}\n```\n\u003c/details\u003e\n\nThat's it. All tool output is now compressed before reaching your LLM.\n\n---\n\n## The Problem\n\nAI coding agents pass raw command output directly to LLMs — full diffs, complete\nlog output, entire directory listings. Most of it is noise.\n\n| Raw output | What the model actually needs |\n|------------|-------------------------------|\n| 47K git diff with unchanged context lines | Changed files + hunks only |\n| npm install tree of 2000 deps | Added/removed/changed packages |\n| 15K test run with dots and timing | Failures + test count |\n| docker build with progress bars | Image ID + errors |\n\n**OpenToken strips the noise. Keeps the signal.** The model reasons the same way,\nanswers the same way, costs 50–80% less.\n\n---\n\n## Before \u0026 After\n\n```diff\n  $ git diff HEAD~1\n- diff --git a/src/autoescalate.ts b/src/autoescalate.ts\n- index a3b4c5d..f6e7a8b 100644\n- --- a/src/autoescalate.ts\n- +++ b/src/autoescalate.ts\n- @@ -1,18 +1,20 @@\n+ │  import { createRequire } from \"module\";\n+ │  import { SessionStore } from \"./utils/session-store\";\n...\n- 2,114 tokens → 407 tokens ⎯⎯⎯ 81% reduction\n```\n\nThe model sees: **\"2 imports added to `autoescalate.ts` — `createRequire` and `SessionStore`.\"**\n\nIt responds exactly as if it read the full diff.\n\n---\n\n## The Pipeline\n\nEvery tool output passes through 35 compression stages. Each stage ends with a\n**conservative safety check** — if output grew, the original is returned untouched.\n\n```mermaid\ngraph LR\n    A[Raw Output] --\u003e B[Secrets Redaction]\n    B --\u003e C[Binary Detection]\n    C --\u003e D[ANSI Strip]\n    D --\u003e E[Thinking Block Strip]\n    E --\u003e F{Family Detector}\n    F --\u003e|git| G1[Git Compressor]\n    F --\u003e|npm| G2[npm Compressor]\n    F --\u003e|cargo| G3[Cargo Compressor]\n    F --\u003e|docker| G4[Docker Compressor]\n    F --\u003e|pip| G5[Pip Compressor]\n    F --\u003e|make| G6[Make Compressor]\n    F --\u003e|fs| G7[Filesystem Compressor]\n    F --\u003e|test| G8[Test Compressor]\n    F --\u003e|generic| G9[Generic]\n    G1 --\u003e H[JSON Minify]\n    G2 --\u003e H\n    G3 --\u003e H\n    G4 --\u003e H\n    G5 --\u003e H\n    G6 --\u003e H\n    G7 --\u003e H\n    G8 --\u003e H\n    G9 --\u003e H\n    H --\u003e I[Log/Diff Folding]\n    I --\u003e J[Table Minification]\n    J --\u003e K[LTSC — LZ77 Compression]\n    K --\u003e L[LZW Token Substitution]\n    L --\u003e M[Cross-Call Dedup]\n    M --\u003e N[Conservative Safety Filter]\n    N --\u003e O[Compressed Output]\n```\n\n### ⚡ LZW Performance\n\nThe LZW compressor uses an O(n) repetitiveness pre-check — skipping the\nexpensive scan on non-compressible input:\n\n| Input | Before | After | Speedup |\n|-------|--------|-------|---------|\n| 1 KB random | 18 ms | 0.9 ms | 20× |\n| 10 KB random | 375 ms | 0.2 ms | **1,875×** |\n| 48 KB random | ~1.8 s | 0.3 ms | **6,000×** |\n| Compressible content | unchanged | unchanged | — |\n\n---\n\n## Architecture\n\n```\npackages/\n├── core/            @mrgray17/opentoken-core    51 pure-logic modules\n│   ├── families/    10 family filters  (git, npm, cargo, docker, ...)\n│   ├── filters/     3 tool filters     (read, grep, glob)\n│   ├── pipelines/   4 tool pipelines   (bash, read, grep, glob)\n│   └── utils/       9 utilities        (secrets, cache, metrics, ...)\n├── cli/             @mrgray17/opentoken-cli          CLI binary — pipe, wrap, stats\n├── mcp/             @mrgray17/opentoken-mcp     MCP server — JSON-RPC over stdio\n└── opencode/        @mrgray17/opentoken  OpenCode plugin — 10 hooks\n```\n\n**Zero platform lock-in.** The core library has no AI-tool dependencies.\nThe same pipeline powers CLI pipes, MCP servers, Node.js scripts, and the\nOpenCode plugin.\n\n### Three Interfaces, One Core\n\n```\n                     ┌─────────────────┐\n                     │  @mrgray17/opentoken-core │\n                     │  51 modules      │\n                     │  Pure logic       │\n                     └───────┬─────────┘\n                             │\n        ┌────────────────────┼────────────────────┐\n        ▼                    ▼                    ▼\n  @mrgray17/opentoken-cli        @mrgray17/opentoken-mcp       @mrgray17/opentoken\n  pipe │ wrap │ stats    JSON-RPC stdio      OpenCode plugin\n  any terminal          Claude Code,          auto-loads\n  any shell             Cursor, Aider,        transparent\n  any AI agent          any MCP host          to user\n```\n\n---\n\n## Safety \u0026 Security\n\n| Guarantee | How |\n|-----------|-----|\n| **Never returns larger output** | Conservative filter at every stage |\n| **Secrets redacted first** | 35+ patterns (AWS, GitHub, OpenAI, Anthropic, JWT, Stripe, …) |\n| **No telemetry** | All data stays local — `~/.config/opentoken/` |\n| **No exec / eval** | Pure function chains only |\n| **Atomic writes** | temp + rename — no partial file writes |\n| **Graceful failure** | Every operation wrapped in try/catch — plugin never breaks the host |\n| **ReDoS scanner in CI** | Proactive regex safety verification |\n| **TOCTOU-resistant** | File path resolution with symlink chain detection |\n\n---\n\n## For Developers\n\n```typescript\nimport {\n  transformToolOutput,\n  compressLZW,\n  decompressLZW,\n  redactSecrets,\n} from \"@mrgray17/opentoken-core\";\n\n// Transform any tool output\nconst { output, saved, beforeTokens, afterTokens } =\n  await transformToolOutput(\"bash\", \"git diff\", rawOutput, {\n    sessionID: \"my-session\",\n    enableMetrics: true,\n  });\n\nconsole.log(`Saved ${saved} tokens (${beforeTokens}→${afterTokens})`);\n\n// Use individual compressors\nconst { compressed } = compressLZW(longText);\nconst restored = decompressLZW(compressed);\n\n// Redact secrets before any processing\nconst safe = redactSecrets(userInput);\n```\n\n### Full API\n\nSee [`packages/core/src/index.ts`](packages/core/src/index.ts) for all exports.\nTypeScript definitions included — everything is strictly typed.\n\n---\n\n## Real Numbers\n\n| Metric | |\n|--------|---|\n| Tokens saved (all time) | **5,078,587** |\n| Cost saved (Claude Pro rates) | **$152.36** |\n| Overall compression rate | **74%** |\n| Median (compressible calls) | **93%** |\n| Best single-call savings | 48,291 tokens |\n| Test count | **431** (0 fail) |\n| Source files | **54** TypeScript modules |\n\n---\n\n## Comparison\n\n| Feature | OpenToken | Truncation | Caveman | Raw |\n|---------|:---------:|:----------:|:-------:|:---:|\n| Preserves semantics | ✅ | ❌ | ❌ | ✅ |\n| Conservative safety | ✅ | N/A | N/A | N/A |\n| Secrets redaction | ✅ | ❌ | ❌ | ❌ |\n| Family-specific filters | 10 families | ❌ | ❌ | ❌ |\n| Lossless compression | LZ77 + LZW | ❌ | ❌ | ❌ |\n| Cross-call dedup | ✅ | ❌ | ❌ | ❌ |\n| CLI pipe mode | ✅ | ✅ | ❌ | ✅ |\n| MCP protocol | ✅ | ❌ | ❌ | ❌ |\n| Model speaks normally | ✅ | ✅ | ❌ | ✅ |\n| Token savings | 70–80% | 10–50% | 50–80% | 0% |\n\n---\n\n## FAQ\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eDoes OpenToken change what the model sees?\u003c/strong\u003e\u003c/summary\u003e\n\n**Semantically, no.** Compressed output preserves all actionable information —\nfile paths, error messages, line numbers, function signatures, class names,\ntest results. The model answers identically whether it sees the raw output\nor the compressed version.\n\nThe conservative safety filter at every stage guarantees: if compression\nproduces a larger or corrupted output, the original is returned instead.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eDoes it work with any AI coding agent?\u003c/strong\u003e\u003c/summary\u003e\n\n**Yes.** Three interfaces:\n- **CLI** — `opentoken wrap \u003ccmd\u003e` works in any terminal with any agent\n- **MCP** — `opentoken-mcp` works with Claude Code, Cursor, Windsurf, any MCP host\n- **Library** — `@mrgray17/opentoken-core` can be integrated into any Node.js/Bun tool\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eIs it safe to compress secrets / API keys?\u003c/strong\u003e\u003c/summary\u003e\n\n**Yes.** Secrets redaction runs *first*, before any other stage.\n35+ patterns cover: AWS keys, GitHub tokens, OpenAI/Anthropic keys,\nJWT tokens, Stripe keys, connection strings, private keys, and more.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eWhat's the performance overhead?\u003c/strong\u003e\u003c/summary\u003e\n\n**Negligible.** Typical pipeline latency is 1–5 ms per tool call.\nThe LZW compressor has an O(n) pre-check that skips expensive scanning\non non-compressible input (6,000× faster on random data).\nToken estimation uses simple heuristics, not full tokenizers.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eWhy Bun and not Node.js?\u003c/strong\u003e\u003c/summary\u003e\n\nThe core library uses Bun-native APIs for filesystem I/O (`Bun.file()`,\n`Bun.write()`, `Bun.spawn()`). Bun is the fastest JavaScript runtime\nand has 80%+ market share in AI tooling (OpenCode, Claude Code MCP\nservers, etc.). The CLI and MCP server are pure Node.js compatible.\n\u003c/details\u003e\n\n---\n\n## Development\n\n```bash\ngit clone https://github.com/MrGray17/opentoken.git\ncd opentoken\nbun install\nbun run build     # typecheck → lint → ReDoS scan → 431 tests\n```\n\n| Command | What it does |\n|---------|--------------|\n| `bun test` | Run all 431 tests across 22 files |\n| `bun run typecheck` | TypeScript strict mode (`tsc --noEmit`) |\n| `bun run lint` | Biome check (66 files) |\n| `bun run lint:fix` | Auto-fix formatting |\n| `bun run checks:regex` | ReDoS pattern scan |\n\nCI order: `typecheck` → `lint` → `checks:regex` → `test`.\n\n---\n\n## Contributing\n\nIssues and PRs welcome. The monorepo structure:\n\n```\npackages/core/     — @mrgray17/opentoken-core (pure logic, no AI-tool deps)\npackages/cli/      — @mrgray17/opentoken-cli (CLI binary)\npackages/mcp/      — @mrgray17/opentoken-mcp (MCP server)\npackages/opencode/ — @mrgray17/opentoken (OpenCode plugin)\n\ntests/core/        — 21 test files, 425 tests\ntests/opencode/    — 1 smoke test\n```\n\nRun `bun test` from the repo root to test everything.\n\n---\n\n## License\n\nMIT © [OpenToken Contributors](https://github.com/MrGray17/opentoken/graphs/contributors)\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n**[GitHub](https://github.com/MrGray17/opentoken)** · **[npm](https://www.npmjs.com/package/@mrgray17/opentoken)** · **[Issues](https://github.com/MrGray17/opentoken/issues)**\n\nMade with ❤️ for the AI coding community\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrgray17%2Fopentoken","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrgray17%2Fopentoken","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrgray17%2Fopentoken/lists"}