{"id":51407922,"url":"https://github.com/jpoindexter/winnow","last_synced_at":"2026-07-04T12:30:20.673Z","repository":{"id":365685680,"uuid":"1273229791","full_name":"jpoindexter/winnow","owner":"jpoindexter","description":"Local-first context compression for AI agents — content-aware, reversible, zero runtime deps. Cuts agent token usage 40-95% while keeping the signal.","archived":false,"fork":false,"pushed_at":"2026-06-18T12:24:46.000Z","size":103,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-18T13:08:51.880Z","etag":null,"topics":["ai-agents","context-compression","llm","llmops","mcp","tokens"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jpoindexter.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-18T10:21:35.000Z","updated_at":"2026-06-18T12:25:10.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jpoindexter/winnow","commit_stats":null,"previous_names":["jpoindexter/winnow"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jpoindexter/winnow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpoindexter%2Fwinnow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpoindexter%2Fwinnow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpoindexter%2Fwinnow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpoindexter%2Fwinnow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jpoindexter","download_url":"https://codeload.github.com/jpoindexter/winnow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpoindexter%2Fwinnow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35122497,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-04T02:00:05.987Z","response_time":113,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","context-compression","llm","llmops","mcp","tokens"],"created_at":"2026-07-04T12:30:19.193Z","updated_at":"2026-07-04T12:30:20.646Z","avatar_url":"https://github.com/jpoindexter.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# winnow\n\n[![ci](https://github.com/jpoindexter/winnow/actions/workflows/ci.yml/badge.svg)](https://github.com/jpoindexter/winnow/actions/workflows/ci.yml)\n[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)\n![node](https://img.shields.io/badge/node-%E2%89%A518-brightgreen)\n![runtime deps](https://img.shields.io/badge/runtime%20deps-0-brightgreen)\n\n**Local-first context compression for AI agents.** Keep the signal, winnow the chaff.\n\nAgents burn tokens on fat tool outputs — JSON dumps, logs, file reads, RAG chunks, conversation history. `winnow` compresses that text *before* it reaches the model, cutting tokens by **40–95%** while keeping what matters. It's content-aware, **reversible** (originals are recoverable on demand), and the core has **zero runtime dependencies**. Everything runs on your machine — no proxy, no API key, no egress.\n\n```\nyour agent / app  →  winnow (local)  →  LLM provider\n```\n\n## Why\n\nCompression that silently drops the wrong line is worse than no compression. `winnow` is built around three ideas:\n\n1. **Content-aware, lossy-but-reversible.** Different compressors for JSON, logs, code, and binary. Every original is stashed locally under a content id, so the model can retrieve the full text the moment it needs detail. Lossy inline, lossless on demand.\n2. **Delivery is backbone-gated.** *How* a large result is delivered changes accuracy as much as how well it's compressed. Strong models get a short preview + a retrievable pointer; small/distilled models get a larger inline window and are never handed a pointer they won't follow.\n3. **Cache-aligned.** A volatile segment (a timestamp, \"current\" state) early in your prompt invalidates the provider's KV cache every turn. `winnow` aligns a tiered prompt so the stable prefix leads and the cache survives.\n\n## Install\n\nInstall from GitHub (not on the npm registry — the `winnow` name there is an unrelated package):\n\n```bash\nnpm install github:jpoindexter/winnow\n```\n\nPin to a commit for reproducible builds:\n\n```bash\nnpm install \"github:jpoindexter/winnow#\u003ccommit-sha\u003e\"\n```\n\nNode ≥ 18, ESM. Core has no runtime deps. Code (AST) compression uses an optional `typescript` peer.\n\n## Quickstart\n\n```ts\nimport { compress, retrieve, stats } from \"winnow\";\n\nconst huge = JSON.stringify(await fetchManyRows()); // e.g. 200 similar objects\n\nconst r = await compress(huge);\nconsole.log(r.text);          // head+tail sample, middle elided, + a retrieval footer\nconsole.log(r.compressed);    // true\nconsole.log(stats(huge, r.text)); // { tokensBefore, tokensAfter, tokensSaved, ratio }\n\n// later, if the model needs the full thing:\nconst original = await retrieve(r.originalId!);\n```\n\nCompress a whole chat array:\n\n```ts\nimport { compressMessages } from \"winnow\";\nconst slim = await compressMessages(messages); // compresses each message's content\n```\n\n## Exactly what it does\n\nNo hand-waving — here is the literal transformation each compressor applies, on real input, with real savings. (Token counts use the default `length/4` heuristic; inject a real tokenizer for exact figures.)\n\n**JSON — keep the edges, elide the middle (recoverable).** A 200-object dump:\n\n```text\nBEFORE  [ {\"id\":0,\"name\":\"item-0\",\"active\":true,\"score\":0}, {\"id\":1,…}, … ×200 ]\nAFTER   head (3) + tail (1) objects kept verbatim; the middle becomes an `__elided__` marker\n4252 → 112 tokens   (97% saved)\n```\n\nThe model sees the schema and a sample; `compress()` stashes the full array so `retrieve(id)` returns it intact. An agent reading 200 rows needs the shape and an example — and asks for row 137 when it actually needs it.\n\n**Logs — collapse repeats to a count.** 40 identical lines:\n\n```text\nBEFORE  2026-06-19T21:04:11Z INFO  cache hit for key=session:abc123 (ttl 300s)\n        …the same line ×40\nAFTER   2026-06-19T21:04:11Z INFO  cache hit for key=session:abc123 (ttl 300s) (×40)\n710 → 19 tokens   (97% saved)\n```\n\n\"It happened 40 times\" is the signal; 40 byte-identical copies are the chaff.\n\n**Repeated blocks — reference, don't repeat (reversible).** A boilerplate paragraph repeated down a page (`¶` = blank line):\n\n```text\nBEFORE  # Report ¶ \u003ccookie notice\u003e ¶ Section 1 ¶ \u003ccookie notice\u003e ¶ Section 2 ¶ \u003ccookie notice\u003e\nAFTER   # Report ¶ \u003ccookie notice\u003e ¶ Section 1 ¶ ⟦↺#0⟧ ¶ Section 2 ¶ ⟦↺#0⟧\n115 → 51 tokens   (56% saved)\n```\n\nThe first occurrence stays inline; later identical blocks become `⟦↺#k⟧` pointing at it, and `rehydrateBlocks` restores the exact original. Repeated nav/footer/disclaimer blocks are the biggest single waste in scraped web and RAG content.\n\n**On real agent tool output.** `compressText` detects the type and routes; `dedupeBlocks` mops up the repeated blocks the router leaves inline. Measured on representative results:\n\n| tool output | tokens | saved |\n|---|---|---|\n| web page (repeated cookie + footer) | 260 | **47%** |\n| 8 search results (shared sponsored block) | 344 | **58%** |\n| 6 repeated stack traces | 195 | **79%** |\n| plain prose, no repetition | 16 | **0% — returned untouched** |\n\nSavings are content-dependent, and that's the honest point: repetition-heavy output (most web / log / RAG content) compresses hard; genuinely unique prose doesn't, and winnow hands it back unchanged rather than mangling it. Everything elided is recoverable — **lossy inline, lossless on demand.**\n\n## Benchmark — measured, not claimed\n\n`winnow bench` runs a fidelity harness: for each case it records token savings and checks whether the \"needle\" (the fact a model would need) **survives compression inline**. Anything elided is still recoverable from the store, so *recoverable* fidelity is 100% by construction — this measures the harder number, what survives **without** a retrieval round-trip.\n\n```\nwinnow fidelity — 7 cases\n  json-head    json  save  86%  inline ✓\n  json-tail    json  save  86%  inline ✓\n  json-middle  json  save  86%  inline · (recoverable)\n  wide-table   json  save  97%  inline · (recoverable)\n  log-error    logs  save  99%  inline ✓\n  log-dupes    logs  save  99%  inline ✓\n  text-prose   text  save   0%  inline ✓\n\navg savings: 79%   inline needle survival: 71%   CNG: -0.362\nby position: head 100% · tail 100% · middle 0% · anywhere 100%\nrecoverable fidelity: 100% (every elided original is retrievable from the store)\n```\n\nThe honest tradeoff is visible: a needle buried deep in the middle of a 200-row array is elided inline — and recoverable in one `retrieve` call. Logs and head/tail JSON keep their signal at a fraction of the tokens. (`CNG`, cost-normalized gain, is negative on the default run because it scores *inline* fidelity only and the default mode is lossy-but-recoverable — it's the conservative number, not a quality loss, since every elided original is retrievable.)\n\n## API\n\n| Export | What it does |\n|---|---|\n| `compress(text, opts?)` | Reversible compress of one block; returns `{ text, compressed, originalId, tokensBefore, tokensAfter }`. |\n| `compressMessages(messages, opts?)` | Compress each `{ content }` in a chat array. |\n| `retrieve(id, dir?)` | Read a stored original back by id. |\n| `stats(before, after)` | Token savings + ratio. |\n| `compressText(text, opts?)` | Pure router (no I/O, no stashing). `opts.tabular` → lossless TOON. |\n| `crushJson` / `squashLogs` / `compressCode` | Individual compressors. |\n| `encodeTable` / `decodeTable` / `toonCompress` | **TOON** — lossless object-array ↔ table (keeps every row). |\n| `dedupeBlocks` / `rehydrateBlocks` / `dedupeMessages` | Collapse repeated blocks/messages anywhere; reversible. |\n| `compactHistory(messages, opts?)` | Anchored history compaction (injected summarizer, extractive fallback). |\n| `pruneText(text, opts?)` | LLMLingua-style score-and-drop; **inject your own scorer**, heuristic fallback. |\n| `makeCounter(encode?)` / `countTokens` | Token counting — exact with an injected encoder. |\n| `tuneOptions(cases?, grid?, weight?)` | Pick compression options that maximize measured survival × savings. |\n| `offload(text, opts?)` | Size-based offload with the backbone-gated delivery policy. |\n| `resolveDelivery` / `classifyBackbone` | The delivery policy primitives. |\n| `alignSegments(segments)` | Cache-align a tiered prompt; returns the prompt, stable-prefix `cacheKey`, and breakpoint. |\n\n`CompressOptions`: `minTokens` (default 400), `headItems` (3), `tailItems` (1), `maxStringLength` (200).\n\n### Cache alignment\n\n```ts\nimport { alignSegments, cacheHolds } from \"winnow\";\n\nconst aligned = alignSegments([\n  { id: \"system\", text: SYSTEM, stable: true },\n  { id: \"tools\",  text: TOOLS,  stable: true },\n  { id: \"clock\",  text: now(),  stable: false }, // moved after the stable prefix\n]);\naligned.prompt;     // stable segments first → cacheable prefix\naligned.cacheKey;   // equal across turns ⇒ the KV cache can hit\ncacheHolds(lastKey, aligned); // did the cached prefix survive this turn?\n```\n\n## CLI\n\nRun the CLI straight from GitHub, no install:\n\n```bash\nnpx github:jpoindexter/winnow bench\n```\n\nFor a persistent `winnow` command, clone the repo and link it:\n\n```bash\ngit clone https://github.com/jpoindexter/winnow \u0026\u0026 cd winnow \u0026\u0026 npm install \u0026\u0026 npm link\n```\n\n(`npm install -g` from a git URL is unreliable on some npm versions — use `npx` or `npm link`.)\n\n```bash\nwinnow bench                 # fidelity benchmark (savings + needle survival)\ncat big.json | winnow compress   # compress stdin → stdout (stats on stderr)\nwinnow retrieve \u003cid\u003e         # print a stored original\nwinnow mcp                   # start the MCP server (stdio)\n```\n\n## MCP server\n\nExpose `winnow` to any MCP client (editors, agent runtimes) as three tools — `winnow_compress`, `winnow_retrieve`, `winnow_stats`:\n\n```bash\nwinnow mcp\n```\n\n```jsonc\n// in your client's MCP config\n{ \"mcpServers\": { \"winnow\": { \"command\": \"winnow\", \"args\": [\"mcp\"] } } }\n```\n\n## Design notes\n\n- **Lossy inline, lossless on demand.** Compression always shrinks; the original is one `retrieve` away. The compressor never keeps a result that didn't actually shrink.\n- **Read-fidelity is a contract.** Precision matters most for code and exact reads — code compression keeps every signature/type/import and only elides bodies (recoverable), so the model still sees the shape.\n- **Local-first.** Originals live in `.winnow/ccr/` (override with `WINNOW_DIR`). Nothing leaves your machine.\n- **Token counts** default to a `length/4` heuristic; swap in a real tokenizer where exact numbers matter.\n\n## License\n\nMIT © Jason Poindexter\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjpoindexter%2Fwinnow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjpoindexter%2Fwinnow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjpoindexter%2Fwinnow/lists"}