{"id":47603211,"url":"https://github.com/open-compress/claw-compactor","last_synced_at":"2026-04-01T18:59:13.043Z","repository":{"id":337942611,"uuid":"1154042797","full_name":"open-compress/claw-compactor","owner":"open-compress","description":"🦞 LLM Token Compression \u0026 Reduction Tool — Cut AI agent token costs by up to 97%. 6-layer deterministic context compression for AI agent workspaces. No LLM required. Prompt compression, context window optimization \u0026 cost reduction for any LLM pipeline.","archived":false,"fork":false,"pushed_at":"2026-03-18T06:58:37.000Z","size":1563,"stargazers_count":1616,"open_issues_count":2,"forks_count":138,"subscribers_count":94,"default_branch":"main","last_synced_at":"2026-03-18T07:48:40.744Z","etag":null,"topics":["ai-agent-tools","ai-cost-saving","ai-infrastructure","claw-compactor","context-compression","context-pruning","context-window-optimization","developer-tools","llm-compression","llm-context-compression","llm-cost-reduction","llm-token-compression","llm-tools","openclaw","prompt-compression","python-tools","token-compression","token-optimization","token-reduction","token-saving"],"latest_commit_sha":null,"homepage":"https://www.opencompress.ai/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/open-compress.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"aeromomo","custom":["https://openclaw.ai"]}},"created_at":"2026-02-10T00:27:08.000Z","updated_at":"2026-03-18T07:00:33.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/open-compress/claw-compactor","commit_stats":null,"previous_names":["aeromomo/claw-compactor","open-compress/claw-compactor"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/open-compress/claw-compactor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compress%2Fclaw-compactor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compress%2Fclaw-compactor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compress%2Fclaw-compactor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compress%2Fclaw-compactor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/open-compress","download_url":"https://codeload.github.com/open-compress/claw-compactor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compress%2Fclaw-compactor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31290993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agent-tools","ai-cost-saving","ai-infrastructure","claw-compactor","context-compression","context-pruning","context-window-optimization","developer-tools","llm-compression","llm-context-compression","llm-cost-reduction","llm-token-compression","llm-tools","openclaw","prompt-compression","python-tools","token-compression","token-optimization","token-reduction","token-saving"],"created_at":"2026-04-01T18:59:11.556Z","updated_at":"2026-04-01T18:59:13.032Z","avatar_url":"https://github.com/open-compress.png","language":"Python","readme":"\u003c!--\n\u003cscript type=\"application/ld+json\"\u003e\n{\n  \"@context\": \"https://schema.org\",\n  \"@type\": \"SoftwareApplication\",\n  \"name\": \"Claw Compactor\",\n  \"description\": \"14-stage Fusion Pipeline for LLM token compression with reversible compression, AST-aware code analysis, and intelligent content routing\",\n  \"applicationCategory\": \"DeveloperApplication\",\n  \"operatingSystem\": \"Cross-platform\",\n  \"softwareVersion\": \"7.0.0\",\n  \"license\": \"https://opensource.org/licenses/MIT\",\n  \"url\": \"https://github.com/open-compress/claw-compactor\",\n  \"downloadUrl\": \"https://github.com/open-compress/claw-compactor\",\n  \"author\": {\n    \"@type\": \"Organization\",\n    \"name\": \"OpenClaw\",\n    \"url\": \"https://openclaw.ai\"\n  },\n  \"offers\": {\n    \"@type\": \"Offer\",\n    \"price\": \"0\",\n    \"priceCurrency\": \"USD\"\n  },\n  \"keywords\": \"token compression, LLM, AI agent, fusion pipeline, reversible compression, AST code analysis, context window optimization\"\n}\n\u003c/script\u003e\n--\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n# Claw Compactor\n\n### 14-Stage Fusion Pipeline for LLM Token Compression\n\n![Claw Compactor Banner](assets/banner.png)\n\n[![CI](https://github.com/open-compress/claw-compactor/actions/workflows/ci.yml/badge.svg)](https://github.com/open-compress/claw-compactor/actions)\n[![codecov](https://codecov.io/gh/open-compress/claw-compactor/graph/badge.svg)](https://codecov.io/gh/open-compress/claw-compactor)\n[![Python](https://img.shields.io/badge/python-3.9%2B-blue)](https://python.org)\n[![License](https://img.shields.io/badge/license-MIT-purple)](LICENSE)\n[![PyPI](https://img.shields.io/pypi/v/claw-compactor?color=blue\u0026label=PyPI)](https://pypi.org/project/claw-compactor/)\n[![Downloads](https://img.shields.io/pypi/dm/claw-compactor?color=green\u0026label=downloads)](https://pypi.org/project/claw-compactor/)\n[![Stars](https://img.shields.io/github/stars/open-compress/claw-compactor?style=social)](https://github.com/open-compress/claw-compactor)\n\n**15–82% compression depending on content \u0026middot; Zero LLM inference cost \u0026middot; Reversible \u0026middot; 1600+ tests**\n\n[Documentation](https://open-compress.github.io/claw-compactor) \u0026middot; [Architecture](ARCHITECTURE.md) \u0026middot; [Benchmarks](#benchmarks) \u0026middot; [Quick Start](#quick-start) \u0026middot; [API](#api)\n\n\u003c/div\u003e\n\n---\n\n## What is Claw Compactor?\n\nClaw Compactor is an open-source **LLM token compression engine** built around a 14-stage **Fusion Pipeline**. Each stage is a specialized compressor — from AST-aware code analysis to JSON statistical sampling to simhash-based deduplication — chained through an immutable data flow architecture where each stage's output feeds the next.\n\n### Demo\n\n```\n$ claw-compactor benchmark ./my-workspace\n\n  Claw Compactor v7.0 — Fusion Pipeline Benchmark\n  ─────────────────────────────────────────────────\n\n  Scanning workspace... 47 files, 234,891 tokens\n\n  Stage Results:\n  ┌──────────────────┬──────────┬───────────┬──────────┐\n  │ Stage            │ Applied  │ Reduction │ Time     │\n  ├──────────────────┼──────────┼───────────┼──────────┤\n  │ Cortex           │ 47/47    │ —         │ 12ms     │\n  │ Photon           │ 3/47     │ 2.1%      │ 4ms      │\n  │ RLE              │ 41/47    │ 8.3%      │ 6ms      │\n  │ SemanticDedup    │ 47/47    │ 12.7%     │ 18ms     │\n  │ Ionizer          │ 8/47     │ 71.2%     │ 9ms      │\n  │ Neurosyntax      │ 23/47    │ 18.4%     │ 31ms     │\n  │ TokenOpt         │ 47/47    │ 4.1%      │ 3ms      │\n  │ Abbrev           │ 12/47    │ 6.8%      │ 5ms      │\n  └──────────────────┴──────────┴───────────┴──────────┘\n\n  Summary:\n    Before:  234,891 tokens ($2.35 at GPT-4 rates)\n    After:   108,250 tokens ($1.08)\n    Saved:   126,641 tokens (53.9%) — $1.27/run\n    Time:    88ms total\n\n  Estimated monthly savings at 100 runs/day: $3,810\n```\n\n---\n\n## How It Compares\n\n| Feature | Claw Compactor | LLMLingua-2 | SelectiveContext | gzip + base64 |\n|:--------|:-:|:-:|:-:|:-:|\n| Compression rate | 15–82% | 30–70% | 10–40% | 60–80% |\n| ROUGE-L @ 0.3 | **0.653** | 0.346 | ~0.4 | N/A |\n| ROUGE-L @ 0.5 | **0.723** | 0.570 | ~0.6 | N/A |\n| LLM inference cost | **$0** | ~$0.02/call | **$0** | **$0** |\n| Latency | **\u003c50ms** | ~300ms | ~200ms | \u003c10ms |\n| Reversible | **Yes** | No | No | Yes (manual) |\n| Content-aware routing | **14 stages** | 1 (perplexity) | 1 (self-info) | None |\n| AST-aware code handling | **Yes** (tree-sitter) | No | No | No |\n| JSON schema sampling | **Yes** | No | No | No |\n| Log/diff/search stages | **Yes** | No | No | No |\n| Required dependencies | **0** | torch, transformers | torch | zlib |\n| LLM-readable output | **Yes** | Partial | Partial | **No** |\n\n**Why Claw Compactor wins:** LLMLingua-2 drops tokens by perplexity score — effective for natural language, but destroys code identifiers, JSON keys, and log patterns. Claw Compactor uses content-type-aware stages that understand the structure of what they're compressing.\n\n---\n\n```\nInput\n  |\n  v\n┌─────────────────────────────────────────────────────────────────────────┐\n│                         FUSION PIPELINE                                 │\n│                                                                         │\n│  QuantumLock ─\u003e Cortex ─\u003e Photon ─\u003e RLE ─\u003e SemanticDedup ─\u003e Ionizer    │\n│       |            |         |        |          |              |        │\n│   KV-cache    auto-detect  base64   path     simhash       JSON         │\n│   alignment   16 languages  strip  shorten   dedup        sampling      │\n│                                                                         │\n│  ─\u003e LogCrunch ─\u003e SearchCrunch ─\u003e DiffCrunch ─\u003e StructuralCollapse      │\n│        |              |              |                |                  │\n│    log folding    result dedup   context fold    import merge            │\n│                                                                         │\n│  ─\u003e Neurosyntax ─\u003e Nexus ─\u003e TokenOpt ─\u003e Abbrev ─────────\u003e Output       │\n│        |             |          |           |                            │\n│    AST compress   ML token   format     NL shorten                      │\n│    (tree-sitter)  classify   optimize   (text only)                     │\n│                                                                         │\n│  [ RewindStore ] ── hash-addressed LRU for reversible retrieval         │\n└─────────────────────────────────────────────────────────────────────────┘\n```\n\nKey design principles:\n\n- **Immutable data flow** — `FusionContext` is a frozen dataclass. Every stage produces a new `FusionResult`; nothing is mutated in-place.\n- **Gate-before-compress** — Each stage has `should_apply()` that inspects context type, language, and role before doing any work. Stages that don't apply are skipped at zero cost.\n- **Content-aware routing** — Cortex auto-detects content type (code, JSON, logs, diffs, search results) and language (Python, Go, Rust, TypeScript, etc.), then downstream stages make type-aware compression decisions.\n- **Reversible compression** — Ionizer stores originals in a hash-addressed `RewindStore`. The LLM can call a tool to retrieve any compressed section by its marker ID.\n\n---\n\n## Benchmarks\n\n### Real-World Compression (FusionEngine v7 vs Legacy Regex)\n\n| Content Type | Legacy | FusionEngine | Improvement |\n|:-------------|-------:|-------------:|:-----------:|\n| Python source | 7.3% | **25.0%** | 3.4x |\n| JSON (100 items) | 12.6% | **81.9%** | 6.5x |\n| Build logs | 5.5% | **24.1%** | 4.4x |\n| Agent conversation | 5.7% | **31.0%** | 5.4x |\n| Git diff | 6.2% | **15.0%** | 2.4x |\n| Search results | 5.3% | **40.7%** | 7.7x |\n| **Weighted average** | **9.2%** | **36.3%** | **3.9x** |\n\n### SWE-bench Real Tasks\n\nTested on real SWE-bench instances with actual repository code:\n\n| Instance | Size | Compression |\n|:---------|-----:|------------:|\n| django__django-11620 | 4.5K | **14.5%** |\n| sympy__sympy-14396 | 5.5K | **19.1%** |\n| scikit-learn-25747 | 11.8K | **15.9%** |\n| scikit-learn-13554 | 73K | **11.8%** |\n| scikit-learn-25308 | 81K | **14.4%** |\n\n### vs LLMLingua-2 (ROUGE-L Fidelity)\n\n| Compression Rate | Claw Compactor | LLMLingua-2 | Delta |\n|:-----------------|---------------:|------------:|------:|\n| 0.3 (aggressive) | **0.653** | 0.346 | +88.2% |\n| 0.5 (balanced) | **0.723** | 0.570 | +26.8% |\n\nClaw Compactor preserves more semantic content at the same compression ratio, with zero LLM inference cost.\n\n---\n\n## Quick Start\n\n### Install from PyPI\n\n```bash\npip install claw-compactor\n```\n\n### Or clone from source\n\n```bash\ngit clone https://github.com/open-compress/claw-compactor.git\ncd claw-compactor\npip install -e .\n```\n\n### Run\n\n```bash\n# Benchmark your workspace (non-destructive)\nclaw-compactor benchmark /path/to/workspace\n\n# Full compression pipeline\nclaw-compactor compress /path/to/workspace\n```\n\n**Requirements:** Python 3.9+. Optional: `pip install claw-compactor[accurate]` for exact token counts via tiktoken.\n\n---\n\n## API\n\n### FusionEngine — Single Text\n\n```python\nfrom scripts.lib.fusion.engine import FusionEngine\n\nengine = FusionEngine()\n\nresult = engine.compress(\n    text=\"def hello():\\n    # greeting function\\n    print('hello')\",\n    content_type=\"code\",    # or let Cortex auto-detect\n    language=\"python\",      # optional hint\n)\n\nprint(result[\"compressed\"])     # compressed output\nprint(result[\"stats\"])          # per-stage timing + token counts\nprint(result[\"markers\"])        # Rewind markers for reversibility\n```\n\n### FusionEngine — Chat Messages\n\n```python\nmessages = [\n    {\"role\": \"system\", \"content\": \"You are a coding assistant...\"},\n    {\"role\": \"user\", \"content\": \"Fix the auth bug in login.py\"},\n    {\"role\": \"assistant\", \"content\": \"I found the issue. Here's the fix:\\n```python\\n...\"},\n    {\"role\": \"tool\", \"content\": '{\"results\": [{\"file\": \"login.py\", ...}, ...]}'},\n]\n\nresult = engine.compress_messages(messages)\n\n# Cross-message dedup runs first, then per-message pipeline\nprint(result[\"stats\"][\"reduction_pct\"])   # aggregate compression %\nprint(result[\"per_message\"])              # per-message breakdown\n```\n\n### Rewind — Reversible Retrieval\n\n```python\nengine = FusionEngine(enable_rewind=True)\nresult = engine.compress(large_json, content_type=\"json\")\n\n# LLM sees compressed output with markers like [rewind:abc123...]\n# When the LLM needs the original, it calls the Rewind tool:\noriginal = engine.rewind_store.retrieve(\"abc123def456...\")\n```\n\n### Custom Stage\n\n```python\nfrom scripts.lib.fusion.base import FusionStage, FusionContext, FusionResult\n\nclass MyStage(FusionStage):\n    name = \"my_compressor\"\n    order = 22  # runs between StructuralCollapse (20) and Neurosyntax (25)\n\n    def should_apply(self, ctx: FusionContext) -\u003e bool:\n        return ctx.content_type == \"log\"\n\n    def apply(self, ctx: FusionContext) -\u003e FusionResult:\n        compressed = my_compression_logic(ctx.content)\n        return FusionResult(\n            content=compressed,\n            original_tokens=estimate_tokens(ctx.content),\n            compressed_tokens=estimate_tokens(compressed),\n        )\n\n# Add to pipeline\npipeline = engine.pipeline.add(MyStage())\n```\n\n---\n\n## The 14 Stages\n\n| # | Stage | Order | Purpose | Applies To |\n|:-:|:------|:-----:|:--------|:-----------|\n| 1 | **QuantumLock** | 3 | Isolates dynamic content in system prompts to maximize KV-cache hit rate | system messages |\n| 2 | **Cortex** | 5 | Auto-detects content type and programming language (16 languages) | untyped content |\n| 3 | **Photon** | 8 | Detects and compresses base64-encoded images | all |\n| 4 | **RLE** | 10 | Path shorthand (`$WS`), IP prefix compression, enum compaction | all |\n| 5 | **SemanticDedup** | 12 | SimHash fingerprint deduplication across content blocks | all |\n| 6 | **Ionizer** | 15 | JSON array statistical sampling with schema discovery + error preservation | json |\n| 7 | **LogCrunch** | 16 | Folds repeated log lines with occurrence counts | log |\n| 8 | **SearchCrunch** | 17 | Deduplicates search/grep results | search |\n| 9 | **DiffCrunch** | 18 | Folds unchanged context lines in git diffs | diff |\n| 10 | **StructuralCollapse** | 20 | Merges import blocks, collapses repeated assertions/patterns | code |\n| 11 | **Neurosyntax** | 25 | AST-aware code compression via tree-sitter (safe regex fallback). Never shortens identifiers. | code |\n| 12 | **Nexus** | 35 | ML token-level compression (stopword removal fallback without model) | text |\n| 13 | **TokenOpt** | 40 | Tokenizer format optimization — strips bold/italic markers, normalizes whitespace | all |\n| 14 | **Abbrev** | 45 | Natural language abbreviation. Only fires on text — never touches code, JSON, or structured data. | text |\n\nEach stage is independent and stateless. Stages communicate only through the immutable `FusionContext` that flows forward through the pipeline.\n\n---\n\n## Workspace Commands\n\n```bash\npython3 scripts/mem_compress.py \u003cworkspace\u003e \u003ccommand\u003e [options]\n```\n\n| Command | Description |\n|:--------|:-----------|\n| `full` | Run complete compression pipeline |\n| `benchmark` | Dry-run compression report |\n| `compress` | Rule-based compression only |\n| `dict` | Dictionary encoding with auto-learned codebook |\n| `observe` | Session transcript JSONL to structured observations |\n| `tiers` | Generate L0/L1/L2 tiered summaries |\n| `dedup` | Cross-file duplicate detection |\n| `estimate` | Token count report |\n| `audit` | Workspace health check |\n| `optimize` | Tokenizer-level format optimization |\n| `auto` | Watch mode — compress on file changes |\n\nOptions: `--json`, `--dry-run`, `--since YYYY-MM-DD`, `--quiet`\n\n---\n\n## Architecture\n\nSee [ARCHITECTURE.md](ARCHITECTURE.md) for the full technical deep-dive:\n- Immutable data flow design\n- Stage execution model and gating\n- Rewind reversible compression protocol\n- Cross-message semantic deduplication\n- How to extend the pipeline\n\n```\n12,000+ lines Python  ·  1,600+ tests  ·  14 fusion stages  ·  0 external ML dependencies\n```\n\n---\n\n## Installation\n\n```bash\n# Clone\ngit clone https://github.com/open-compress/claw-compactor.git\ncd claw-compactor\n\n# Optional: exact token counting\npip install tiktoken\n\n# Optional: AST-aware code compression (Neurosyntax)\npip install tree-sitter-language-pack\n\n# Development\npip install -e \".[dev,accurate]\"\n```\n\n**Zero required dependencies.** tiktoken and tree-sitter are optional enhancements — the pipeline runs with built-in heuristic fallbacks for both.\n\n---\n\n## Who Uses This\n\n| Project | How |\n|:--------|:----|\n| [OpenClaw](https://openclaw.ai) | Built-in skill for all OpenClaw AI agents — compresses workspace context before every LLM call |\n| [OpenCompress](https://opencompress.ai) | Production compression engine powering the OpenCompress API |\n\nUsing Claw Compactor? [Open a PR](https://github.com/open-compress/claw-compactor/pulls) to add yourself here.\n\n---\n\n## Project Stats\n\n| Metric | Value |\n|:-------|:------|\n| Tests | 1,600+ passed |\n| Python source | 12,000+ lines |\n| Fusion stages | 14 |\n| Languages detected | 16 |\n| Required dependencies | 0 |\n| Compression (code) | 15–25% |\n| Compression (JSON peak) | 81.9% |\n| ROUGE-L @ 0.3 rate | 0.653 |\n| License | MIT |\n\n---\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on:\n- Setting up the development environment\n- Adding new Fusion stages\n- Running the test suite\n- Submitting PRs\n\n---\n\n## Related\n\n- [OpenClaw](https://openclaw.ai) — AI agent platform\n- [ClawhubAI](https://clawhub.com) — Agent skills marketplace\n- [OpenClaw Discord](https://discord.com/invite/clawd) — Community\n- [OpenClaw Docs](https://docs.openclaw.ai) — Documentation\n- [Full Documentation](https://open-compress.github.io/claw-compactor) — GitHub Pages docs\n\n---\n\n`token-compression` `llm-tools` `fusion-pipeline` `reversible-compression` `ast-code-analysis` `context-compression` `ai-agent` `openclaw` `python` `developer-tools`\n\n## License\n\n[MIT](LICENSE)\n","funding_links":["https://github.com/sponsors/aeromomo","https://openclaw.ai"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-compress%2Fclaw-compactor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopen-compress%2Fclaw-compactor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-compress%2Fclaw-compactor/lists"}