https://github.com/mrgray17/opentoken
Token-saving companion for OpenCode — 42 compression layers, zero risk, no caveman speak
https://github.com/mrgray17/opentoken
compression llm opencode opencode-plugin token-savings
Last synced: about 20 hours ago
JSON representation
Token-saving companion for OpenCode — 42 compression layers, zero risk, no caveman speak
- Host: GitHub
- URL: https://github.com/mrgray17/opentoken
- Owner: MrGray17
- License: mit
- Created: 2026-05-19T20:26:39.000Z (11 days ago)
- Default Branch: main
- Last Pushed: 2026-05-27T00:05:25.000Z (3 days ago)
- Last Synced: 2026-05-27T01:21:47.806Z (3 days ago)
- Topics: compression, llm, opencode, opencode-plugin, token-savings
- Language: TypeScript
- Homepage: https://www.npmjs.com/package/@mrgray17/opentoken
- Size: 498 KB
- Stars: 71
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# ⚡ OpenToken
The universal token-compression engine for AI coding agents.
Every tool output. Every shell. Every IDE.
35 stages of lossless compression. Same semantics. 50–80% fewer tokens. Zero risk.
[](https://www.npmjs.com/package/@mrgray17/opentoken)
[](https://www.npmjs.com/package/@mrgray17/opentoken-cli)
[](https://www.npmjs.com/package/@mrgray17/opentoken-core)
[](https://github.com/MrGray17/opentoken/actions)
[](https://bun.sh)
[](LICENSE)
**5M+ tokens saved** · 74% compression · 431 tests · 0 regressions
---
## Install
```bash
# OpenCode plugin — auto-loads, zero config
opencode plugin @mrgray17/opentoken@latest --global
```
```bash
# MCP server — for VS Code, Cursor, Windsurf, Claude, etc.
npm install -g @mrgray17/opentoken-mcp
```
```bash
# CLI — run anywhere, no install needed
bun x @mrgray17/opentoken-cli wrap git diff HEAD~1
# install globally (optional)
bun install -g @mrgray17/opentoken-cli
```
```bash
# Library — build your own compression pipeline
npm install @mrgray17/opentoken-core
```
**Bun v1.2+ required.** [Install Bun](https://bun.sh) — one command, 2 seconds.
---
## 10-Second Start
```bash
# Pipe any command output through the compression engine
git diff HEAD~1 | opentoken -t bash -c "git diff HEAD~1"
# Or wrap it — same result, cleaner syntax
opentoken wrap cargo build --release
# Check your savings
opentoken stats
```
### In your AI coding agent (MCP)
Install once:
```bash
npm install -g @mrgray17/opentoken-mcp
```
Then add to your IDE's MCP config:
Cursor / Windsurf / Claude Desktop
```json
// ~/.cursor/mcp.json or ~/.windsurf/mcp.json
{
"mcpServers": {
"opentoken": {
"command": "opentoken-mcp"
}
}
}
```
VS Code (Copilot / GitHub Chat)
```json
// .vscode/mcp.json or ~/.vscode/mcp.json
{
"servers": {
"opentoken": {
"type": "stdio",
"command": "opentoken-mcp"
}
}
}
```
Claude Code (CLI)
```json
// ~/.claude/claude_desktop_config.json or .claude/mcp.json
{
"mcpServers": {
"opentoken": {
"command": "opentoken-mcp"
}
}
}
```
That's it. All tool output is now compressed before reaching your LLM.
---
## The Problem
AI coding agents pass raw command output directly to LLMs — full diffs, complete
log output, entire directory listings. Most of it is noise.
| Raw output | What the model actually needs |
|------------|-------------------------------|
| 47K git diff with unchanged context lines | Changed files + hunks only |
| npm install tree of 2000 deps | Added/removed/changed packages |
| 15K test run with dots and timing | Failures + test count |
| docker build with progress bars | Image ID + errors |
**OpenToken strips the noise. Keeps the signal.** The model reasons the same way,
answers the same way, costs 50–80% less.
---
## Before & After
```diff
$ git diff HEAD~1
- diff --git a/src/autoescalate.ts b/src/autoescalate.ts
- index a3b4c5d..f6e7a8b 100644
- --- a/src/autoescalate.ts
- +++ b/src/autoescalate.ts
- @@ -1,18 +1,20 @@
+ │ import { createRequire } from "module";
+ │ import { SessionStore } from "./utils/session-store";
...
- 2,114 tokens → 407 tokens ⎯⎯⎯ 81% reduction
```
The model sees: **"2 imports added to `autoescalate.ts` — `createRequire` and `SessionStore`."**
It responds exactly as if it read the full diff.
---
## The Pipeline
Every tool output passes through 35 compression stages. Each stage ends with a
**conservative safety check** — if output grew, the original is returned untouched.
```mermaid
graph LR
A[Raw Output] --> B[Secrets Redaction]
B --> C[Binary Detection]
C --> D[ANSI Strip]
D --> E[Thinking Block Strip]
E --> F{Family Detector}
F -->|git| G1[Git Compressor]
F -->|npm| G2[npm Compressor]
F -->|cargo| G3[Cargo Compressor]
F -->|docker| G4[Docker Compressor]
F -->|pip| G5[Pip Compressor]
F -->|make| G6[Make Compressor]
F -->|fs| G7[Filesystem Compressor]
F -->|test| G8[Test Compressor]
F -->|generic| G9[Generic]
G1 --> H[JSON Minify]
G2 --> H
G3 --> H
G4 --> H
G5 --> H
G6 --> H
G7 --> H
G8 --> H
G9 --> H
H --> I[Log/Diff Folding]
I --> J[Table Minification]
J --> K[LTSC — LZ77 Compression]
K --> L[LZW Token Substitution]
L --> M[Cross-Call Dedup]
M --> N[Conservative Safety Filter]
N --> O[Compressed Output]
```
### ⚡ LZW Performance
The LZW compressor uses an O(n) repetitiveness pre-check — skipping the
expensive scan on non-compressible input:
| Input | Before | After | Speedup |
|-------|--------|-------|---------|
| 1 KB random | 18 ms | 0.9 ms | 20× |
| 10 KB random | 375 ms | 0.2 ms | **1,875×** |
| 48 KB random | ~1.8 s | 0.3 ms | **6,000×** |
| Compressible content | unchanged | unchanged | — |
---
## Architecture
```
packages/
├── core/ @mrgray17/opentoken-core 51 pure-logic modules
│ ├── families/ 10 family filters (git, npm, cargo, docker, ...)
│ ├── filters/ 3 tool filters (read, grep, glob)
│ ├── pipelines/ 4 tool pipelines (bash, read, grep, glob)
│ └── utils/ 9 utilities (secrets, cache, metrics, ...)
├── cli/ @mrgray17/opentoken-cli CLI binary — pipe, wrap, stats
├── mcp/ @mrgray17/opentoken-mcp MCP server — JSON-RPC over stdio
└── opencode/ @mrgray17/opentoken OpenCode plugin — 10 hooks
```
**Zero platform lock-in.** The core library has no AI-tool dependencies.
The same pipeline powers CLI pipes, MCP servers, Node.js scripts, and the
OpenCode plugin.
### Three Interfaces, One Core
```
┌─────────────────┐
│ @mrgray17/opentoken-core │
│ 51 modules │
│ Pure logic │
└───────┬─────────┘
│
┌────────────────────┼────────────────────┐
▼ ▼ ▼
@mrgray17/opentoken-cli @mrgray17/opentoken-mcp @mrgray17/opentoken
pipe │ wrap │ stats JSON-RPC stdio OpenCode plugin
any terminal Claude Code, auto-loads
any shell Cursor, Aider, transparent
any AI agent any MCP host to user
```
---
## Safety & Security
| Guarantee | How |
|-----------|-----|
| **Never returns larger output** | Conservative filter at every stage |
| **Secrets redacted first** | 35+ patterns (AWS, GitHub, OpenAI, Anthropic, JWT, Stripe, …) |
| **No telemetry** | All data stays local — `~/.config/opentoken/` |
| **No exec / eval** | Pure function chains only |
| **Atomic writes** | temp + rename — no partial file writes |
| **Graceful failure** | Every operation wrapped in try/catch — plugin never breaks the host |
| **ReDoS scanner in CI** | Proactive regex safety verification |
| **TOCTOU-resistant** | File path resolution with symlink chain detection |
---
## For Developers
```typescript
import {
transformToolOutput,
compressLZW,
decompressLZW,
redactSecrets,
} from "@mrgray17/opentoken-core";
// Transform any tool output
const { output, saved, beforeTokens, afterTokens } =
await transformToolOutput("bash", "git diff", rawOutput, {
sessionID: "my-session",
enableMetrics: true,
});
console.log(`Saved ${saved} tokens (${beforeTokens}→${afterTokens})`);
// Use individual compressors
const { compressed } = compressLZW(longText);
const restored = decompressLZW(compressed);
// Redact secrets before any processing
const safe = redactSecrets(userInput);
```
### Full API
See [`packages/core/src/index.ts`](packages/core/src/index.ts) for all exports.
TypeScript definitions included — everything is strictly typed.
---
## Real Numbers
| Metric | |
|--------|---|
| Tokens saved (all time) | **5,078,587** |
| Cost saved (Claude Pro rates) | **$152.36** |
| Overall compression rate | **74%** |
| Median (compressible calls) | **93%** |
| Best single-call savings | 48,291 tokens |
| Test count | **431** (0 fail) |
| Source files | **54** TypeScript modules |
---
## Comparison
| Feature | OpenToken | Truncation | Caveman | Raw |
|---------|:---------:|:----------:|:-------:|:---:|
| Preserves semantics | ✅ | ❌ | ❌ | ✅ |
| Conservative safety | ✅ | N/A | N/A | N/A |
| Secrets redaction | ✅ | ❌ | ❌ | ❌ |
| Family-specific filters | 10 families | ❌ | ❌ | ❌ |
| Lossless compression | LZ77 + LZW | ❌ | ❌ | ❌ |
| Cross-call dedup | ✅ | ❌ | ❌ | ❌ |
| CLI pipe mode | ✅ | ✅ | ❌ | ✅ |
| MCP protocol | ✅ | ❌ | ❌ | ❌ |
| Model speaks normally | ✅ | ✅ | ❌ | ✅ |
| Token savings | 70–80% | 10–50% | 50–80% | 0% |
---
## FAQ
Does OpenToken change what the model sees?
**Semantically, no.** Compressed output preserves all actionable information —
file paths, error messages, line numbers, function signatures, class names,
test results. The model answers identically whether it sees the raw output
or the compressed version.
The conservative safety filter at every stage guarantees: if compression
produces a larger or corrupted output, the original is returned instead.
Does it work with any AI coding agent?
**Yes.** Three interfaces:
- **CLI** — `opentoken wrap ` works in any terminal with any agent
- **MCP** — `opentoken-mcp` works with Claude Code, Cursor, Windsurf, any MCP host
- **Library** — `@mrgray17/opentoken-core` can be integrated into any Node.js/Bun tool
Is it safe to compress secrets / API keys?
**Yes.** Secrets redaction runs *first*, before any other stage.
35+ patterns cover: AWS keys, GitHub tokens, OpenAI/Anthropic keys,
JWT tokens, Stripe keys, connection strings, private keys, and more.
What's the performance overhead?
**Negligible.** Typical pipeline latency is 1–5 ms per tool call.
The LZW compressor has an O(n) pre-check that skips expensive scanning
on non-compressible input (6,000× faster on random data).
Token estimation uses simple heuristics, not full tokenizers.
Why Bun and not Node.js?
The core library uses Bun-native APIs for filesystem I/O (`Bun.file()`,
`Bun.write()`, `Bun.spawn()`). Bun is the fastest JavaScript runtime
and has 80%+ market share in AI tooling (OpenCode, Claude Code MCP
servers, etc.). The CLI and MCP server are pure Node.js compatible.
---
## Development
```bash
git clone https://github.com/MrGray17/opentoken.git
cd opentoken
bun install
bun run build # typecheck → lint → ReDoS scan → 431 tests
```
| Command | What it does |
|---------|--------------|
| `bun test` | Run all 431 tests across 22 files |
| `bun run typecheck` | TypeScript strict mode (`tsc --noEmit`) |
| `bun run lint` | Biome check (66 files) |
| `bun run lint:fix` | Auto-fix formatting |
| `bun run checks:regex` | ReDoS pattern scan |
CI order: `typecheck` → `lint` → `checks:regex` → `test`.
---
## Contributing
Issues and PRs welcome. The monorepo structure:
```
packages/core/ — @mrgray17/opentoken-core (pure logic, no AI-tool deps)
packages/cli/ — @mrgray17/opentoken-cli (CLI binary)
packages/mcp/ — @mrgray17/opentoken-mcp (MCP server)
packages/opencode/ — @mrgray17/opentoken (OpenCode plugin)
tests/core/ — 21 test files, 425 tests
tests/opencode/ — 1 smoke test
```
Run `bun test` from the repo root to test everything.
---
## License
MIT © [OpenToken Contributors](https://github.com/MrGray17/opentoken/graphs/contributors)
---
**[GitHub](https://github.com/MrGray17/opentoken)** · **[npm](https://www.npmjs.com/package/@mrgray17/opentoken)** · **[Issues](https://github.com/MrGray17/opentoken/issues)**
Made with ❤️ for the AI coding community