{"id":45090754,"url":"https://github.com/mpecan/tokf","last_synced_at":"2026-04-18T11:06:52.918Z","repository":{"id":339398796,"uuid":"1160693006","full_name":"mpecan/tokf","owner":"mpecan","description":"Config-driven CLI tool that compresses command output before it reaches an LLM context","archived":false,"fork":false,"pushed_at":"2026-04-11T12:06:30.000Z","size":2078,"stargazers_count":144,"open_issues_count":18,"forks_count":13,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-11T13:14:52.150Z","etag":null,"topics":["ai-tools","claude-code","cli","command-line","context-window","developer-tools","homebrew","llm","output-filter","rust","token-optimization","toml"],"latest_commit_sha":null,"homepage":"https://tokf.net","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mpecan.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-18T09:00:44.000Z","updated_at":"2026-04-11T11:30:33.000Z","dependencies_parsed_at":"2026-02-24T11:07:07.655Z","dependency_job_id":null,"html_url":"https://github.com/mpecan/tokf","commit_stats":null,"previous_names":["mpecan/tokf"],"tags_count":186,"template":false,"template_full_name":null,"purl":"pkg:github/mpecan/tokf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpecan%2Ftokf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpecan%2Ftokf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpecan%2Ftokf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpecan%2Ftokf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mpecan","download_url":"https://codeload.github.com/mpecan/tokf/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpecan%2Ftokf/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31966218,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-tools","claude-code","cli","command-line","context-window","developer-tools","homebrew","llm","output-filter","rust","token-optimization","toml"],"created_at":"2026-02-19T17:02:58.282Z","updated_at":"2026-04-18T11:06:52.904Z","avatar_url":"https://github.com/mpecan.png","language":"Rust","readme":"# tokf\n\n[![CI](https://github.com/mpecan/tokf/actions/workflows/ci.yml/badge.svg)](https://github.com/mpecan/tokf/actions/workflows/ci.yml)\n[![crates.io](https://img.shields.io/crates/v/tokf)](https://crates.io/crates/tokf)\n[![crates.io downloads](https://img.shields.io/crates/d/tokf)](https://crates.io/crates/tokf)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n**[tokf.net](https://tokf.net)** — reduce LLM context consumption from CLI commands by 60–90%.\n\nCommands like `git push`, `cargo test`, and `docker build` produce verbose output packed with progress bars, compile noise, and boilerplate. tokf intercepts that output, applies a TOML filter, and emits only what matters — so your AI agent sees a clean signal instead of hundreds of wasted tokens.\n\n---\n\n## Before / After\n\n**`cargo test` — 61 lines → 1 line:**\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eWithout tokf\u003c/th\u003e\n\u003cth\u003eWith tokf\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```\n   Compiling tokf v0.2.0 (/home/user/tokf)\n   Compiling proc-macro2 v1.0.92\n   Compiling unicode-ident v1.0.14\n   Compiling quote v1.0.38\n   Compiling syn v2.0.96\n   Compiling serde_derive v1.0.217\n   Compiling serde v1.0.217\n   ...\nrunning 47 tests\ntest config::tests::test_load ... ok\ntest filter::tests::test_skip ... ok\ntest filter::tests::test_keep ... ok\ntest filter::tests::test_extract ... ok\n...\ntest result: ok. 47 passed; 0 failed; 0 ignored\n  finished in 2.31s\n```\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n```\n✓ 47 passed (2.31s)\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n**`git push` — 8 lines → 1 line:**\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eWithout tokf\u003c/th\u003e\n\u003cth\u003eWith tokf\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```\nEnumerating objects: 5, done.\nCounting objects: 100% (5/5), done.\nDelta compression using up to 10 threads\nCompressing objects: 100% (3/3), done.\nWriting objects: 100% (3/3), 312 bytes | 312.00 KiB/s, done.\nTotal 3 (delta 2), reused 0 (delta 0), pack-reused 0\nremote: Resolving deltas: 100% (2/2), completed with 2 local objects.\nTo github.com:user/repo.git\n   a1b2c3d..e4f5a6b  main -\u003e main\n```\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n```\nok ✓ main\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n---\n\n\n## Quickstart\n\n```sh\nbrew install mpecan/tokf/tokf   # or: cargo install tokf\ntokf setup                      # detect your AI tools and install hooks\n```\n\nThat's it. Every command your AI agent runs is now automatically filtered.\n\nRun `tokf gain` to see how many tokens you've saved, or `tokf setup --refresh` to re-run detection.\n\n---\n\n\n## Installation\n\n### Homebrew (macOS and Linux)\n\n```sh\nbrew install mpecan/tokf/tokf\n```\n\n### cargo\n\n```sh\ncargo install tokf\n```\n\n### Build from source\n\n```sh\ngit clone https://github.com/mpecan/tokf\ncd tokf\ncargo build --release\n# binary at target/release/tokf\n```\n\n---\n\n## How it works\n\n```\ntokf run git push origin main\n```\n\ntokf looks up a filter for `git push`, runs the command, and applies the filter. The filter logic lives in plain TOML files — no recompilation required. Anyone can author, share, or override a filter.\n\n---\n\n## Set up automatic filtering\n\nIf you use an AI coding tool, install the hook so every command is filtered automatically — no `tokf run` prefix needed:\n\n```sh\n# Claude Code (recommended: --global so it works in every project)\ntokf hook install --global\n\n# OpenCode\ntokf hook install --tool opencode --global\n\n# OpenAI Codex CLI\ntokf hook install --tool codex --global\n```\n\nDrop `--global` to install for the current project only. See [Claude Code hook](#claude-code-hook) for details on each tool, the `--path` flag, and optional extras like the filter-authoring skill.\n\n---\n\n## Usage\n\n### Run a command with filtering\n\n```sh\ntokf run git push origin main\ntokf run cargo test\ntokf run docker build .\n```\n\n### Apply a filter to a fixture\n\n```sh\ntokf apply filters/git/push.toml tests/fixtures/git_push_success.txt --exit-code 0\n```\n\n### Verify filter test suites\n\n```sh\ntokf verify                    # run all test suites\ntokf verify git/push           # run a specific suite\ntokf verify --list             # list available suites and case counts\ntokf verify --json             # output results as JSON\ntokf verify --require-all      # fail if any filter has no test suite\ntokf verify --list --require-all  # show coverage per filter\ntokf verify --scope project    # only project-local filters (.tokf/filters/)\ntokf verify --scope global     # only user-level filters (~/.config/tokf/filters/)\ntokf verify --scope stdlib     # only built-in stdlib (filters/ in CWD)\ntokf verify --safety           # run safety checks (prompt injection, shell injection, hidden unicode)\ntokf verify git/push --safety  # safety check a specific filter\n```\n\n### Task runner filtering\n\ntokf automatically wraps `make` and `just` so that each recipe line is individually filtered:\n\n```sh\nmake check    # each recipe line (cargo test, cargo clippy, ...) is filtered\njust test     # same — each recipe runs through tokf\n```\n\nSee [Rewrite configuration](#rewrite-configuration-rewritestoml) for details and customization.\n\n### Explore available filters\n\n```sh\ntokf ls                    # list all filters\ntokf which \"cargo test\"    # which filter would match\ntokf show git/push         # print the TOML source\n```\n\n### Customize a built-in filter\n\n```sh\ntokf eject cargo/build            # copy to .tokf/filters/ (project-local)\ntokf eject cargo/build --global   # copy to ~/.config/tokf/filters/ (user-level)\n```\n\nThis copies the filter TOML and its test suite to your config directory, where it shadows the built-in. Edit the ejected copy freely — tokf's priority system ensures your version is used instead of the original.\n\n### Flags\n\n| Flag | Description |\n|---|---|\n| `--timing` | Print how long filtering took |\n| `--verbose` | Show which filter was matched (also explains skipped rewrites) |\n| `--no-filter` | Pass output through without filtering |\n| `--no-cache` | Bypass the filter discovery cache |\n| `--no-mask-exit-code` | Disable exit-code masking. By default tokf exits 0 and prepends `Error: Exit code N` on failure |\n| `--preserve-color` | Preserve ANSI color codes in filtered output (env: `TOKF_PRESERVE_COLOR=1`). See [Color passthrough](#color-passthrough) below |\n| `--baseline-pipe` | Pipe command for fair baseline accounting (injected by rewrite) |\n| `--prefer-less` | Compare filtered vs piped output and use whichever is smaller (requires `--baseline-pipe`) |\n\n### Color passthrough\n\nBy default, filters with `strip_ansi = true` permanently remove ANSI escape codes. The `--preserve-color` flag changes this: tokf strips ANSI **internally** for pattern matching (skip, keep, dedup) but restores the original colored lines in the final output. When `--preserve-color` is active it overrides `strip_ansi = true` in the filter config.\n\ntokf does **not** force commands to emit color — you must ensure the child command outputs ANSI codes (e.g. via `FORCE_COLOR=1` or `--color=always`):\n\n```sh\n# Node.js / Vitest / Jest\nFORCE_COLOR=1 tokf run --preserve-color npm test\n\n# Cargo\ntokf run --preserve-color cargo test -- --color=always\n\n# Or set the env var once for all invocations\nexport TOKF_PRESERVE_COLOR=1\nFORCE_COLOR=1 tokf run npm test\n```\n\n**Limitations:** color passthrough applies to the skip/keep/dedup pipeline (stages 2–2.5). The `match_output`, `parse`, and `lua_script` stages operate on clean text and are unaffected by this flag. `[[replace]]` rules run on the raw text before the color split, so when `--preserve-color` is enabled their patterns may need to account for ANSI escape codes, similar to branch-level `skip` patterns, which also match against the restored colored text.\n\n---\n\n## Built-in filter library\n\n| Filter | Command |\n|---|---|\n| `git/add` | `git add` |\n| `git/commit` | `git commit` |\n| `git/diff` | `git diff` — overrides to `git diff --stat` for compact output. Pass `-p`/`--patch`/`--no-stat`/`-U\u003cn\u003e`/`--name-only`/`--name-status`/`--numstat`/`--shortstat`/`--raw` to skip the override and get the requested format instead |\n| `git/log` | `git log` — overrides to `git log --oneline --no-decorate -n 20`. Pass `-p`/`--patch`/`--format`/`--pretty`/`--graph`/`--stat`/`--shortstat`/`--dirstat`/`--name-only`/`--name-status`/`-L` to skip the override. Empty results emit a one-line hint pointing at common causes (untracked pathspec, missing `--all`, missing `--follow`) instead of nothing — this stops agents looping through flag variations trying to escape a non-existent filter |\n| `git/push` | `git push` |\n| `git/show` | `git show` |\n| `git/status` | `git status` — runs `git status --porcelain=v1 -b -uall --find-renames`; shows branch + upstream sync state (`[synced]`, `[ahead N]`, `[behind N]`, `(no upstream)`) and one porcelain line per changed file (`M  src/main.rs`, `?? scratch.rs`, `R  old.rs -\u003e new.rs`). `-uall` lists every untracked file individually instead of collapsing newly-created directories. When 3+ files share a directory prefix the listing is restructured into a directory tree (see [`[tree]`](writing-filters.md#tree-restructuring)), writing each shared prefix once. Measured 24.4% averaged token reduction across the bundled test fixtures |\n| `cargo/build` | `cargo build` |\n| `cargo/check` | `cargo check` |\n| `cargo/clippy` | `cargo clippy` |\n| `cargo/fmt` | `cargo fmt` |\n| `cargo/install` | `cargo install *` |\n| `cargo/test` | `cargo test` |\n| `docker/*` | `docker build`, `docker compose`, `docker images`, `docker ps` |\n| `npm/run` | `npm run *` |\n| `npm/test` | `npm test`, `pnpm test`, `yarn test` (with vitest/jest variants) |\n| `pnpm/*` | `pnpm add`, `pnpm install` |\n| `go/*` | `go build`, `go vet` |\n| `gradle/*` | `gradle build`, `gradle test`, `gradle dependencies` |\n| `gh/*` | `gh pr list`, `gh pr view`, `gh pr checks`, `gh issue list`, `gh issue view` |\n| `kubectl/*` | `kubectl get pods` |\n| `next/*` | `next build` |\n| `prisma/*` | `prisma generate` |\n| `pytest` | Python test runner |\n| `tsc` | TypeScript compiler |\n| `ls` | `ls` |\n\n---\n\n\n## Generic Commands\n\nWhen no dedicated filter exists for a command, three built-in subcommands provide useful compression for arbitrary output:\n\n| Command | Purpose | Default context |\n|---------|---------|----------------|\n| `tokf err \u003ccmd\u003e` | Extract errors and warnings | 3 lines |\n| `tokf test \u003ccmd\u003e` | Extract test failures | 5 lines |\n| `tokf summary \u003ccmd\u003e` | Heuristic summary | 30 lines max |\n\n### `tokf err` — Error extraction\n\nScans output for error/warning patterns across common toolchains (Rust, Python, Node, Go, Java) and shows only the relevant lines with surrounding context.\n\n```sh\n# Show only errors from a build\ntokf err cargo build\n\n# Adjust context lines around each error\ntokf err -C 5 cargo build\n\n# Works with any command\ntokf err python train.py\ntokf err npm run build\n```\n\n**Patterns matched:** `error:`, `warning:`, `FAILED`, `Traceback`, `panic`, `npm ERR!`, `fatal:`, Python/Java exception types, and more.\n\n**Behaviour:**\n- Empty output: `[tokf err] no errors detected (empty output)`\n- Short output (\u003c 10 lines): shows `[tokf err]` header with full output\n- No errors + exit 0: prints `[tokf err] no errors detected`\n- No errors + exit ≠ 0: includes full output (something failed but no recognized pattern)\n\n### `tokf test` — Test failure extraction\n\nExtracts test failure details and always includes summary/result lines.\n\n```sh\n# Show only test failures\ntokf test cargo test\ntokf test go test ./...\ntokf test npm test\ntokf test pytest\n```\n\n**Patterns matched:** `FAIL`, `FAILED`, `panicked`, assertion mismatches, Jest `✕` markers, Go `--- FAIL:`, and more.\n\n**Summary lines always included:** `test result:`, `Tests:`, `passed`, `failed` counts, pytest/RSpec summary lines.\n\n**Behaviour:**\n- Output \u003c 10 lines: passed through unchanged\n- All pass + exit 0: prints `[tokf test] all tests passed`\n- Failures detected: shows failure lines with context + summary\n\n### `tokf summary` — Heuristic summary\n\nProduces a budget-constrained summary by identifying header, footer/summary, and repetitive middle sections.\n\n```sh\n# Summarize a long build log\ntokf summary cargo build\n\n# Limit to 15 lines\ntokf summary --max-lines 15 make all\n```\n\n**Algorithm:**\n1. Header (first 5 lines) and footer/summary (last lines matching keywords like \"total\", \"finished\", \"result\")\n2. Middle section is sampled; highly repetitive content shows a count + samples\n3. Extracted statistics (pass/fail counts, timing) appended as `[tokf summary]` line\n\n### Common flags\n\nAll three commands support:\n\n| Flag | Description |\n|------|-------------|\n| `--baseline-pipe \u003ccmd\u003e` | Fair baseline accounting (as with `tokf run`) |\n| `--no-mask-exit-code` | Propagate the real exit code instead of masking to 0 |\n| `--timing` | Show how long filtering took |\n\n### Using generic commands with rewrites\n\nGeneric commands can be integrated with the [rewrite system](rewrites-config.md) so they trigger automatically through the hook. Add rules to `.tokf/rewrites.toml` for commands that don't have dedicated filters:\n\n```toml\n# Route build commands without filters through tokf err\n[[rewrite]]\nmatch = \"^mix compile\"\nreplace = \"tokf err {0}\"\n\n[[rewrite]]\nmatch = \"^cmake --build\"\nreplace = \"tokf err {0}\"\n\n# Route test runners without filters through tokf test\n[[rewrite]]\nmatch = \"^mix test\"\nreplace = \"tokf test {0}\"\n\n[[rewrite]]\nmatch = \"^ctest\"\nreplace = \"tokf test {0}\"\n\n# Summarize long-running commands\n[[rewrite]]\nmatch = \"^terraform plan\"\nreplace = \"tokf summary {0}\"\n```\n\n**Important:** User rewrite rules are checked *before* filter matching. Don't add rules for commands that already have dedicated filters (like `cargo build`, `npm test`) — the dedicated filter will produce better output than the generic command.\n\nTo check whether a command already has a filter: `tokf which \"cargo build\"`.\n\n### Tracking and history\n\nGeneric commands record to the same tracking database and history as `tokf run`, using filter names `_builtin/err`, `_builtin/test`, and `_builtin/summary`. Use `tokf raw last` to see the full uncompressed output.\n\n---\n\n\nFilters are TOML files placed in `.tokf/filters/` (project-local) or `~/.config/tokf/filters/` (user-level). Project-local filters take priority over user-level, which take priority over the built-in library.\n\n## Minimal example\n\n```toml\ncommand = \"my-tool\"\n\n[on_success]\noutput = \"ok ✓\"\n\n[on_failure]\ntail = 10\n```\n\n## Command matching\n\ntokf matches commands against filter patterns using two built-in behaviours:\n\n**Basename matching** — the first word of a pattern is compared by basename, so a filter with `command = \"git push\"` will also match `/usr/bin/git push` or `./git push`.  This works automatically; no special pattern syntax is required.\n\n**Transparent global flags** — flag-like tokens between the command name and a subcommand keyword are skipped during matching.  A filter for `git log` will match all of:\n\n```\ngit log\ngit -C /path log\ngit --no-pager -C /path log --oneline\n/usr/bin/git --no-pager -C /path log\n```\n\nThe skipped flags are preserved in the command that actually runs — they are only bypassed during the pattern match.\n\n\u003e **Note on `run` override and transparent flags:** If a filter sets a `run` field, transparent global flags are *not* included in `{args}`.  Only the arguments that appear after the matched pattern words are available as `{args}`.\n\n## Common fields\n\n```toml\ncommand = \"git push\"          # command pattern to match (supports wildcards and arrays)\nrun = \"git push {args}\"       # override command to actually execute\ndescription = \"Compact git push output\"  # human-readable description (shown in `tokf ls`)\n\nskip = [\"^Enumerating\", \"^Counting\"]  # drop lines matching these regexes\nkeep = [\"^error\"]                      # keep only lines matching (inverse of skip)\n\n# Per-line regex replacement — applied before skip/keep, in order.\n# Capture groups use {1}, {2}, … . Invalid patterns are silently skipped.\n[[replace]]\npattern = '^(\\S+)\\s+\\S+\\s+(\\S+)\\s+(\\S+)'\noutput = \"{1}: {2} → {3}\"\n\ndedup = true                  # collapse consecutive identical lines\ndedup_window = 10             # optional: compare within a N-line sliding window\n\nstrip_ansi = true             # strip ANSI escape sequences before processing\ntrim_lines = true             # trim leading/trailing whitespace from each line\nstrip_empty_lines = true      # remove all blank lines from the final output\ncollapse_empty_lines = true   # collapse consecutive blank lines into one\ntruncate_lines_at = 120       # truncate lines longer than N chars (with trailing …)\n\ntail = 30                     # keep last N lines regardless of exit code (branch tail overrides)\non_empty = \"git push: ok\"     # message when filter produces empty output (all lines stripped)\n\nshow_history_hint = true      # append a hint line (`tokf raw \u003cid\u003e`) pointing to the full output in history\ninject_path = true            # inject shims into PATH so sub-processes (e.g. git hooks) are filtered\n\npassthrough_args = [\"--watch\", \"--web\", \"-w\"]  # skip filter when user passes these flags\n\n# Lua escape hatch — for logic TOML can't express (see Lua Escape Hatch section)\n[lua_script]\nlang = \"luau\"\nsource = 'return output:upper()'    # inline script\n# file = \"transform.luau\"           # or reference a local file (auto-inlined on publish)\n\nmatch_output = [              # whole-output substring checks, short-circuit the pipeline\n  { contains = \"rejected\", output = \"push rejected\" },\n]\n\n[on_success]                  # branch for exit code 0\noutput = \"ok ✓ {2}\"          # template; {output} = pre-filtered output\n\n[on_failure]                  # branch for non-zero exit\ntail = 10                     # keep the last N lines (overrides top-level tail)\n```\n\n## Passthrough args\n\nSome filters inject flags like `--json` or `--format` via the `run` field. When users pass conflicting flags (e.g. `--watch`), the combined command fails. The `passthrough_args` field declares flag prefixes that trigger passthrough mode — tokf skips the filter entirely and runs the original command as-is.\n\n```toml\ncommand = \"gh pr checks *\"\nrun = \"gh pr checks {args} --json name,state,workflow\"\npassthrough_args = [\"--watch\", \"--web\", \"-w\"]\n```\n\n**Matching semantics**: each user arg is checked with `starts_with` against each prefix. This handles `--format=table` matching `--format`, while `-w` does **not** match `--watch` (correct — they are different flags). Short-flag prefixes like `-o` also match concatenated forms like `-oyaml` (common in tools like `kubectl`). Empty-string prefixes are ignored. When any arg matches, no `run` override is applied and no filter pipeline runs.\n\n**Variant interaction**: passthrough is checked on the resolved filter config after file-based and args-based variant detection. If a parent filter delegates to a variant (via file detection or `args_pattern`), the variant's own `passthrough_args` apply — not the parent's. Output-pattern variants (post-execution) are not resolved when passthrough is active.\n\nUse `--verbose` to see when passthrough activates:\n\n```\n$ tokf run gh pr checks 142 --watch --verbose\n[tokf] passthrough: user args match passthrough_args, skipping filter\n```\n\n## Template pipes\n\nOutput templates support pipe chains: `{var | pipe | pipe: \"arg\"}`.\n\n| Pipe | Input → Output | Description |\n|---|---|---|\n| `join: \"sep\"` | Collection → Str | Join items with separator |\n| `each: \"tmpl\"` | Collection → Collection | Map each item through a sub-template |\n| `truncate: N` | Str → Str | Truncate to N characters, appending `…` |\n| `lines` | Str → Collection | Split on newlines |\n| `keep: \"re\"` | Collection → Collection | Retain items matching the regex |\n| `where: \"re\"` | Collection → Collection | Alias for `keep:` |\n\nExample — filter a multi-line output variable to only error lines:\n\n```toml\n[on_failure]\noutput = \"{output | lines | keep: \\\"^error\\\" | join: \\\"\\\\n\\\"}\"\n```\n\nExample — for each collected block, show only `\u003e` (pointer) and `E` (assertion) lines:\n\n```toml\n[on_failure]\noutput = \"{failure_lines | each: \\\"{value | lines | keep: \\\\\\\"^[\u003eE] \\\\\\\"}\\\" | join: \\\"\\\\n\\\"}\"\n```\n\n## Sections\n\nSections collect lines into named buckets using a state-machine model. They are processed on the raw output (before skip/keep filtering) so structural markers like blank lines are available.\n\n```toml\n[[section]]\nname = \"failures\"\nenter = \"^failures:$\"        # regex that starts collecting\nexit = \"^failures:$\"         # regex that stops collecting (second occurrence)\nsplit_on = \"^\\\\s*$\"          # split collected lines into blocks at blank lines\ncollect_as = \"failure_blocks\" # name used in templates: {failure_blocks}\n\n[[section]]\nname = \"summary\"\nmatch = \"^test result:\"      # stateless: collect any matching line\ncollect_as = \"summary_lines\"\n```\n\n**Stateful sections** (with `enter`/`exit`) toggle on/off as the state machine hits the enter/exit patterns. **Stateless sections** (with `match` only) collect every matching line regardless of state.\n\nSection data is available in templates:\n- `{failure_blocks}` — the collected items\n- `{failure_blocks.count}` — number of items (blocks if `split_on` is set, otherwise lines)\n- `{failure_blocks | each: \"...\" | join: \"\\\\n\"}` — iterate over items\n\n## Aggregates\n\nAggregates extract numeric values from section items and produce named variables for templates.\n\n**Single aggregate** (backwards compatible):\n\n```toml\n[on_success]\noutput = \"{passed} passed ({suites} suites)\"\n\n[on_success.aggregate]\nfrom = \"summary_lines\"\npattern = 'ok\\. (\\d+) passed'\nsum = \"passed\"\ncount_as = \"suites\"\n```\n\n**Multiple aggregates** — use `[[on_success.aggregates]]` (plural) to define several rules:\n\n```toml\n[on_success]\noutput = \"✓ {passed} passed, {failed} failed, {ignored} ignored ({suites} suites)\"\n\n[[on_success.aggregates]]\nfrom = \"summary_lines\"\npattern = 'ok\\. (\\d+) passed'\nsum = \"passed\"\ncount_as = \"suites\"\n\n[[on_success.aggregates]]\nfrom = \"summary_lines\"\npattern = '(\\d+) failed'\nsum = \"failed\"\n\n[[on_success.aggregates]]\nfrom = \"summary_lines\"\npattern = '(\\d+) ignored'\nsum = \"ignored\"\n```\n\nEach rule scans the named section's items. `sum` accumulates the first capture group as a number. `count_as` counts the number of matching lines. Both singular `aggregate` and plural `aggregates` can be used together — they are merged at runtime.\n\n## Chunk processing\n\nChunks split raw output into repeating structural blocks, extract structured data per-block, and produce named collections for template rendering. Use chunks when you need per-block breakdown (e.g., per-crate test results in a Cargo workspace).\n\n\u003e **Note:** Like sections, chunks operate on the raw (unfiltered) command output. Skip/keep patterns do not affect chunk processing. This ensures structural markers are available for splitting.\n\n```toml\n[[chunk]]\nsplit_on = \"^\\\\s*Running \"   # regex that marks the start of each chunk\ninclude_split_line = true     # include the splitting line in the chunk (default: true)\ncollect_as = \"suites_detail\"  # name for the structured collection\ngroup_by = \"crate_name\"       # merge chunks sharing this field value\n\n[chunk.extract]\npattern = 'deps/([\\w_-]+)-'  # extract a field from the split (header) line\nas = \"crate_name\"\n\n[[chunk.aggregate]]\npattern = '(\\d+) passed'     # aggregates run within each chunk's own lines\nsum = \"passed\"\n\n[[chunk.aggregate]]\npattern = '(\\d+) failed'\nsum = \"failed\"\n\n[[chunk.aggregate]]\npattern = '^test result:'\ncount_as = \"suite_count\"\n```\n\n**Fields**:\n\n| Field | Description |\n|---|---|\n| `split_on` | Regex marking the start of each chunk |\n| `include_split_line` | Whether the splitting line is part of the chunk (default: `true`) |\n| `collect_as` | Name for the resulting structured collection |\n| `extract` | Extract a named field from the header line (`pattern` + `as`) |\n| `body_extract` | Extract fields from body lines (`pattern` + `as`, first match wins) |\n| `aggregate` | Per-chunk aggregation rules (run within each chunk's own lines) |\n| `group_by` | Merge chunks sharing the same field value, summing numeric fields |\n| `children_as` | When set with `group_by`, preserve original items as a nested collection under this name |\n| `carry_forward` | On `extract` or `body_extract`: inherit value from the previous chunk when the pattern doesn't match |\n\nThe resulting structured collection is available in templates as `{suites_detail}` and supports field access in `each` pipes.\n\n### Structured collections in templates\n\nWhen a chunk produces a structured collection, each item has named fields. Use `each` to iterate with field access:\n\n```toml\n[on_success]\noutput = \"\"\"✓ cargo test: {passed} passed ({suites} suites)\n{suites_detail | each: \"  {crate_name}: {passed} passed ({suite_count} suites)\" | join: \"\\\\n\"}\"\"\"\n```\n\nInside the `each` template, all named fields from the chunk item are available as variables (`{crate_name}`, `{passed}`, `{suite_count}`), plus `{index}` (1-based) and `{value}` (debug representation).\n\n`{suites_detail.count}` returns the number of items in the collection.\n\n### Carry-forward fields\n\nWhen a chunk's `extract` or `body_extract` rule has `carry_forward = true`, chunks that don't match the pattern inherit the value from the most recent chunk that did. This is useful when boundary markers (like `Running unittests`) identify a group, and subsequent chunks (like integration test suites) should inherit that identity.\n\n```toml\n[chunk.extract]\npattern = 'unittests.+deps/([\\w_-]+)-'\nas = \"crate_name\"\ncarry_forward = true\n```\n\n### Tree-structured groups (children_as)\n\nWhen `children_as` is set alongside `group_by`, the grouped collection preserves each group's original items as a nested collection. Inside an `each` template, the children are accessible by the `children_as` name and support their own `each`/`join` pipes:\n\n```toml\n[[chunk]]\nsplit_on = \"^\\\\s*Running \"\ncollect_as = \"suites_detail\"\ngroup_by = \"crate_name\"\nchildren_as = \"children\"\n\n[on_success]\noutput = \"\"\"✓ {passed} passed ({suites} suites)\n{suites_detail | each: \"  {crate_name}: {passed} passed\\n{children | each: \\\"    {suite_name}: {passed}\\\" | join: \\\"\\\\n\\\"}\" | join: \"\\\\n\"}\"\"\"\n```\n\nThis produces tree output like:\n\n```\n✓ 565 passed (2 suites)\n  tokf: 565 passed\n    unittests src/lib.rs: 550\n    tests/cli_basic.rs: 15\n```\n\n## JSON extraction\n\nWhen commands produce JSON output (e.g. `kubectl get pods -o json`, `gh api`, `docker inspect`), use the `[json]` block to extract values via `JSONPath` (RFC 9535) instead of line-based parsing.\n\n```toml\ncommand = \"kubectl get pods -o json\"\n\n[json]\n\n# Array of objects → structured collection (usable with |each: pipe)\n# Auto-generates {pods_count} with the number of matched items.\n[[json.extract]]\npath = \"$.items[*]\"\nas = \"pods\"\n\n# Sub-field extraction from each matched object (dot-path, not JSONPath)\n[[json.extract.fields]]\nfield = \"metadata.name\"\nas = \"name\"\n\n[[json.extract.fields]]\nfield = \"status.phase\"\nas = \"phase\"\n\n[on_success]\noutput = \"Pods ({pods_count}):\\n{pods | each: \\\"  {name}: {phase}\\\" | join: \\\"\\\\n\\\"}\"\n```\n\n**Result mapping**:\n\n| JSONPath result | Behavior |\n|---|---|\n| Single scalar (string/number/bool/null) | `vars[\"as_name\"] = string_value` |\n| Array of scalars | `ChunkData::Flat` with `{value}` key per item; auto-generates `{as_name_count}` |\n| Array of objects (with `fields`) | `ChunkData::Flat` with named field keys; auto-generates `{as_name_count}` |\n| Array of objects (without `fields`) | All top-level scalar fields auto-flattened; auto-generates `{as_name_count}` |\n\n**Pipeline position**: JSON extraction runs after `lua_script` (step 2c) and replaces `parse`/`sections`/`chunks` — when `[json]` is configured, those line-based structural steps are skipped. The extracted vars and chunks flow into branch selection (`on_success`/`on_failure`) and template rendering.\n\n**Dot-path syntax** for `[[json.extract.fields]]`: uses simple dot-separated paths (not JSONPath). Supports array indices: `containers.0.name` traverses `obj[\"containers\"][0][\"name\"]`.\n\n**Error handling**: if the input is not valid JSON, extraction is skipped and tokf falls back to raw output (templates are not rendered). Invalid JSONPath or dot-path expressions are silently skipped.\n\n## Tree restructuring\n\nWhen a filter emits a list of file paths, common directory prefixes are repeated on every line. The `[tree]` section restructures the output into a directory tree, writing each shared prefix once. Reusable across any path-shaped filter (`git status`, `git diff --name-only`, etc.).\n\n```toml\ncommand = \"git status\"\n\n[tree]\n# Regex with two capture groups: (1) decoration to keep on the leaf\n# (e.g. \"M  \", \"?? \"), (2) the path itself.\npattern = '^(.. )(.+)$'\n\n# Lines that don't match (e.g. \"main [synced]\" branch headers) are kept\n# verbatim: unmatched lines before the first matched path stay in place\n# above the tree, and any later unmatched lines are emitted after the\n# tree. Set to false to drop them.\npassthrough_unmatched = true\n\n# Engagement gates — when not satisfied, the original flat output is\n# returned unchanged. Tuned per filter.\nmin_files = 3            # require at least N matched paths\nmin_shared_depth = 1     # require at least N common directory levels\n\n# Visual style. \"indent\" is the cheapest in token count (plain 2-space\n# indent, no connectors). \"unicode\" uses ├─ │ └─ box-drawing characters\n# (prettier but each char is 3 bytes in UTF-8 — measurably more expensive\n# on deep trees). \"ascii\" uses |- | `-.\nstyle = \"indent\"\n\n# Collapse single-child internal directories. Without this, narrow-deep\n# paths like a/b/c/d/foo.rs render as four separate dir nodes.\ncollapse_single_child = true\n\n# Sort children alphabetically. Off by default — source order is stable\n# and predictable for LLMs.\nsort = false\n```\n\n### Example\n\nBefore (`git status` raw porcelain):\n\n```text\n## main...origin/main\nM  crates/tokf-cli/src/config/cache.rs\nM  crates/tokf-cli/src/config/types.rs\nM  crates/tokf-cli/src/main.rs\nM  crates/tokf-cli/filters/git/diff.toml\nM  crates/tokf-cli/filters/git/status.toml\n?? crates/tokf-filter/src/filter/tree.rs\n```\n\nAfter (with `[tree]` enabled, indent style):\n\n```text\nmain [synced]\ncrates/\n  tokf-cli/\n    src/\n      config/\n        M  cache.rs\n        M  types.rs\n      M  main.rs\n    filters/git/\n      M  diff.toml\n      M  status.toml\n  ?? tokf-filter/src/filter/tree.rs\n```\n\nThe shared `crates/tokf-cli/` prefix is written once. The single-child chain `tokf-filter/src/filter/` collapses into one leaf. The model sees at a glance which directories cluster work.\n\n### Pipeline position\n\nThe tree transform runs **after** `dedup` and **before** `on_success.output` / `max_lines`. Specifically: stage 2.6 in `apply_internal`, between dedup (2.5) and the lua/json/parse/section pipeline (2b–4).\n\n### Constraints\n\n- **Color restoration is bypassed** when `[tree]` is active. Tree-rendered lines are synthesized from path components, so per-line ANSI color spans from the original output don't survive structural rearrangement. If you need both colored output and tree structuring, you'll have to pick one.\n- **Engagement is opt-in.** Without a `[tree]` section, filters behave exactly as before — no magic detection.\n- **Engagement gates fail closed.** If `min_files` or `min_shared_depth` aren't met, the original flat lines pass through unchanged. There's no half-rendered intermediate state.\n- **Rename arrows** like `R  old.rs -\u003e new.rs` are handled: the path is split on ` -\u003e ` and the suffix stays attached to the leaf. The trie key is the *old* path.\n- **`[parse]` takes precedence.** A filter that declares both `[parse]` and `[tree]` will run parse and skip tree entirely. The two solve different problems (tree restructures path-list output, parse structures arbitrary text) and don't compose, so the precedence is fixed at parse-wins.\n\n## Filter variants\n\nSome commands are wrappers around different underlying tools (e.g. `npm test` may run Jest, Vitest, or Mocha). A parent filter can declare `[[variant]]` entries that delegate to specialized child filters based on project context:\n\n```toml\ncommand = [\"npm test\", \"pnpm test\", \"yarn test\"]\n\nstrip_ansi = true\nskip = [\"^\u003e \", \"^\\\\s*npm (warn|notice|WARN|verbose|info|timing|error|ERR)\"]\n\n[on_success]\noutput = \"{output}\"\n\n[on_failure]\ntail = 20\n\n[[variant]]\nname = \"vitest\"\ndetect.files = [\"vitest.config.ts\", \"vitest.config.js\", \"vitest.config.mts\"]\nfilter = \"npm/test-vitest\"\n\n[[variant]]\nname = \"jest\"\ndetect.files = [\"jest.config.js\", \"jest.config.ts\", \"jest.config.json\"]\nfilter = \"npm/test-jest\"\n```\n\nDetection has three modes, checked in order:\n\n1. **File detection** (Phase A, before execution) — checks if config files exist in the current directory. First match wins.\n2. **Args pattern** (Phase A.5, before execution) — regex-matches the remaining command-line arguments (joined with spaces). Fires after file detection but before the `passthrough_args` check, so a matched variant's own `passthrough_args` apply instead of the parent's.\n3. **Output pattern** (Phase B, after execution) — regex-matches command output. Used as a fallback when no file or args pattern matched.\n\nArgs-pattern example — route `git diff --name-only` to a tree-structured child filter:\n\n```toml\n[[variant]]\nname = \"name-list\"\ndetect.args_pattern = '--(name-only|name-status)'\nfilter = \"git/diff-name-list\"\n```\n\nWhen no variant matches, the parent filter's own fields (`skip`, `on_success`, etc.) apply as the fallback.\n\nThe `filter` field references another filter by its discovery name (relative path without `.toml`). Use `tokf which \"npm test\" -v` to see variant resolution.\n\n\u003e **TOML ordering**: `[[variant]]` entries must appear **after** all top-level fields (`skip`, `[on_success]`, etc.) because TOML array-of-tables sections capture subsequent keys.\n\n## Filter resolution\n\n1. `.tokf/filters/` in the current directory (repo-local overrides)\n2. `~/.config/tokf/filters/` (user-level overrides)\n3. Built-in library (embedded in the binary)\n\nFirst match wins. Use `tokf which \"git push\"` to see which filter would activate.\n\n## Writing test cases\n\nFilter tests live in a `\u003cstem\u003e_test/` directory adjacent to the filter TOML:\n\n```\nfilters/\n  git/\n    push.toml          \u003c- filter config\n    push_test/         \u003c- test suite\n      success.toml\n      rejected.toml\n```\n\nEach test case is a TOML file specifying a fixture (inline or file path), expected exit code, and one or more `[[expect]]` assertions:\n\n```toml\nname = \"rejected push shows pull hint\"\nfixture = \"tests/fixtures/git_push_rejected.txt\"\nexit_code = 1\n\n[[expect]]\nequals = \"✗ push rejected (try pulling first)\"\n```\n\nFor quick inline fixtures without a file:\n\n```toml\nname = \"clean tree shows nothing to commit\"\ninline = \"## main...origin/main\\n\"\nexit_code = 0\n\n[[expect]]\ncontains = \"clean\"\n```\n\n**Assertion types**:\n\n| Field | Description |\n|---|---|\n| `equals` | Output exactly equals this string |\n| `contains` | Output contains this substring |\n| `not_contains` | Output does not contain this substring |\n| `starts_with` | Output starts with this string |\n| `ends_with` | Output ends with this string |\n| `line_count` | Output has exactly N non-empty lines |\n| `matches` | Output matches this regex |\n| `not_matches` | Output does not match this regex |\n\nExit codes from `tokf verify`: `0` = all pass, `1` = assertion failure, `2` = config/IO error or uncovered filters (`--require-all`).\n\n### Safety checks\n\nAdd `--safety` to detect potential security issues in your filter:\n\n```sh\ntokf verify --safety\ntokf verify my-filter --safety --json\n```\n\nSafety checks scan for:\n\n- **Prompt injection** — templates containing patterns like \"ignore previous instructions\", \"you are now\", \"system prompt\", etc. Both static config text and filtered output are checked (NFKC-normalized to handle compatibility/fullwidth forms; cross-script homoglyphs are not fully covered).\n- **Shell injection** — `run`, `step[].run`, and rewrite replacement strings containing shell metacharacters (`$(...)`, backticks, `;`, `\u0026\u0026`, pipes, redirections). Known-safe templates like `tokf run {0}` are allowlisted.\n- **Hidden Unicode** — zero-width spaces, RTL overrides, and other invisible characters that could smuggle content.\n\nSafety warnings do **not** block publishing — filters with issues are published with `safety_passed = false` and the registry shows a warning badge. Use `--safety` locally to catch issues before publishing.\n\n---\n\n\n## Configuration files\n\ntokf uses TOML files for configuration. Files can live at two levels:\n\n| File | Global path | Project-local path | Purpose |\n|------|-------------|---------------------|---------|\n| `config.toml` | `~/.config/tokf/config.toml` | `.tokf/config.toml` | History retention, sync settings, telemetry |\n| `rewrites.toml` | `~/.config/tokf/rewrites.toml` | `.tokf/rewrites.toml` | Shell rewrite rules |\n| `auth.toml` | `~/.config/tokf/auth.toml` | — | Registry authentication (managed by `tokf auth`) |\n| `machine.toml` | `~/.config/tokf/machine.toml` | — | Machine UUID for remote sync |\n\n\u003e Paths shown are for Linux/macOS. On macOS, the global directory is `~/Library/Application Support/tokf`. On Windows, it is `%APPDATA%/tokf`.\n\nSet `TOKF_HOME` to redirect **all** user-level paths to a single directory (like `CARGO_HOME` or `RUSTUP_HOME`):\n\n```sh\nexport TOKF_HOME=/opt/tokf   # all config, data, and cache under /opt/tokf\n```\n\nRun `tokf info` to see which paths are active.\n\n---\n\n## config.toml\n\n### `[history]`\n\nControls how many filtered outputs are retained in the local history database.\n\n```toml\n[history]\nretention = 10   # number of history entries to keep (default: 10)\n```\n\n### `[sync]`\n\nSettings for remote filter-usage sync (requires `tokf auth login`).\n\n```toml\n[sync]\nauto_sync_threshold = 100   # sync after this many unsynced records (default: 100)\nupload_usage_stats = true   # upload anonymous usage statistics (default: not set)\n```\n\n### `[shims]`\n\nControls PATH-based shim injection for sub-process filtering. When filters use `inject_path = true`, tokf generates shim scripts and prepends them to `PATH` so that sub-processes (e.g. commands inside git hooks) are automatically filtered.\n\n```toml\n[shims]\nenabled = true   # generate and use shims for inject_path filters (default: true)\n```\n\nSet to `false` to disable shim generation and PATH injection globally. This overrides any per-filter `inject_path = true` setting — no shims will be generated or used.\n\n```sh\ntokf config set shims.enabled false\n```\n\nShim scripts are stored in `~/.cache/tokf/shims/` (or `$TOKF_HOME/shims/`). Disabling shims via `config set` immediately removes any existing shim scripts.\n\n\u003e **Note:** `shims.enabled` is read from the global config only — project-local overrides are not checked, to avoid filesystem scanning on every command invocation.\n\n### `[telemetry]`\n\nExport metrics via OpenTelemetry OTLP. Disabled by default.\n\n```toml\n[telemetry]\nenabled = true\nendpoint = \"http://localhost:4318\"   # OTLP collector endpoint\nprotocol = \"http\"                    # \"http\" (default) or \"grpc\"\nservice_name = \"tokf\"                # service.name resource attribute\n\n[telemetry.headers]\nx-api-key = \"your-secret\"\n```\n\nDefault endpoints by protocol: HTTP uses port `4318`, gRPC uses port `4317`.\n\nEnvironment variables override config-file values for telemetry — see the [Environment variables](#environment-variables) section below.\n\n### Priority order\n\nFor all `config.toml` settings:\n\n1. **Project-local** `.tokf/config.toml` (highest priority)\n2. **Global** `~/.config/tokf/config.toml`\n3. **Built-in defaults**\n\n### Managing config via CLI\n\n```sh\ntokf config show              # show all effective config with source paths\ntokf config show --json       # machine-readable JSON output\ntokf config get \u003ckey\u003e         # print a single value (for scripting)\ntokf config set \u003ckey\u003e \u003cvalue\u003e # set a value in the global config\ntokf config set --local \u003ckey\u003e \u003cvalue\u003e  # set in project-local .tokf/config.toml\ntokf config print             # print raw config file contents\ntokf config path              # show config file paths with existence status\n```\n\nAvailable keys: `history.retention`, `shims.enabled`, `sync.auto_sync_threshold`, `sync.upload_stats`.\n\n---\n\n## rewrites.toml\n\nRewrite rules let tokf intercept and transform commands before execution — wrapping task runners, stripping pipes, and injecting baselines. See the [Rewrite Configuration](#rewrite-configuration-rewritestoml) section for the full reference.\n\n---\n\n## Environment variables\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| **Paths** | | |\n| `TOKF_HOME` | Redirect all user-level tokf paths (config, data, cache) to a single directory | Platform config dir |\n| `TOKF_DB_PATH` | Override the tracking database path only (takes precedence over `TOKF_HOME`) | Platform data dir |\n| **Runtime** | | |\n| `TOKF_DEBUG` | Enable debug output (set to `1` or `true`) | unset |\n| `TOKF_NO_FILTER` | Skip filtering in shell mode (set to `1`, `true`, or `yes`) | unset |\n| `TOKF_VERBOSE` | Print filter resolution details in shell mode | unset |\n| `TOKF_PRESERVE_COLOR` | Preserve ANSI color codes in filtered output | unset |\n| `TOKF_HTTP_TIMEOUT` | HTTP request timeout in seconds (for remote operations) | `5` |\n| `NO_COLOR` | Disable colored output in `tokf gain` (per [no-color.org](https://no-color.org/)) | unset |\n| **Telemetry** | | |\n| `TOKF_TELEMETRY_ENABLED` | Enable telemetry export (`true`, `1`, or `yes`) — overrides config file | unset |\n| `TOKF_OTEL_PIPELINE` | Pipeline label attached to telemetry metrics | unset |\n| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector endpoint — overrides config file | `http://localhost:4318` |\n| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http` or `grpc` — overrides config file | `http` |\n| `OTEL_EXPORTER_OTLP_HEADERS` | Comma-separated `key=value` header pairs — overrides config file | empty |\n| `OTEL_RESOURCE_ATTRIBUTES` | Comma-separated `key=value` resource attributes; `service.name` is extracted | empty |\n| **Registry (CI)** | | |\n| `TOKF_REGISTRY_URL` | Registry base URL for `tokf regenerate-examples` | — |\n| `TOKF_SERVICE_TOKEN` | Service token for registry authentication | — |\n\n---\n\n## CLI flags\n\n### Global flags\n\nAvailable on all `tokf run` invocations and most subcommands:\n\n| Flag | Description |\n|------|-------------|\n| `--timing` | Print how long filtering took |\n| `--verbose` | Show filter resolution details |\n| `--no-filter` | Pass output through without filtering |\n| `--no-cache` | Bypass the binary filter discovery cache |\n| `--no-mask-exit-code` | Propagate real exit code instead of masking to 0 |\n| `--preserve-color` | Preserve ANSI color codes in filtered output |\n| `--otel-export` | Export metrics via OpenTelemetry OTLP for this invocation |\n\n### Run-specific flags\n\n| Flag | Description |\n|------|-------------|\n| `--baseline-pipe \u003ccmd\u003e` | Pipe command for fair baseline accounting (injected by rewrite rules) |\n| `--prefer-less` | Compare filtered vs piped output and use whichever is smaller |\n\n---\n\n## Filter resolution order\n\ntokf searches for matching filters in three tiers, stopping at the first match:\n\n1. **Project-local** — `.tokf/filters/` in the project root\n2. **User-level** — `~/.config/tokf/filters/` (or `$TOKF_HOME/filters/`)\n3. **Standard library** — built-in filters shipped with tokf\n\nTo override a built-in filter, eject it to your project or user directory:\n\n```sh\ntokf eject cargo/test              # copy to .tokf/filters/ (project-local)\ntokf eject --global cargo/test     # copy to ~/.config/tokf/filters/ (user-level)\n```\n\nThe ejected copy takes priority on subsequent runs. See `tokf eject --help` for details.\n\n---\n\n## Directory layout\n\n```\n~/.config/tokf/                    # global config directory ($TOKF_HOME overrides)\n├── config.toml                    # history, sync, telemetry settings\n├── rewrites.toml                  # shell rewrite rules\n├── auth.toml                      # registry credentials (managed by tokf auth)\n├── machine.toml                   # machine UUID for remote sync\n└── filters/                       # user-level filter overrides\n    └── cargo/\n        └── test.toml\n\n~/.local/share/tokf/               # data directory\n└── tracking.db                    # token savings database ($TOKF_DB_PATH overrides)\n\n~/.cache/tokf/                     # cache directory\n├── manifest.bin                   # binary filter discovery cache\n└── shims/                         # generated shim scripts for inject_path\n\n\u003cproject\u003e/\n└── .tokf/                         # project-local overrides\n    ├── config.toml                # project-specific settings\n    ├── rewrites.toml              # project-specific rewrite rules\n    └── filters/                   # project-specific filters\n        └── custom/\n            └── lint.toml\n```\n\n---\n\n\nFor logic that TOML can't express — numeric math, multi-line lookahead, conditional branching — embed a [Luau](https://luau.org/) script:\n\n```toml\ncommand = \"my-tool\"\n\n[lua_script]\nlang = \"luau\"\nsource = '''\nif exit_code == 0 then\n    return \"passed\"\nelse\n    return \"FAILED: \" .. output:match(\"Error: (.+)\") or output\nend\n'''\n```\n\nAvailable globals: `output` (string), `exit_code` (integer — the underlying command's real exit code, unaffected by `--no-mask-exit-code`), `args` (table).\nReturn a string to replace output, or `nil` to fall through to the rest of the TOML pipeline.\n\n### Sandbox\n\nAll Lua execution is sandboxed — both in the CLI and on the server:\n\n- **Blocked libraries:** `io`, `os`, `package` — no filesystem or network access.\n- **Instruction limit:** 1 million VM instructions (prevents infinite loops).\n- **Memory limit:** 16 MB (prevents memory exhaustion).\n\nScripts that exceed these limits are terminated and treated as a passthrough (the TOML pipeline continues as if no Lua script was configured).\n\n### External script files\n\nFor local development you can keep the script in a separate `.luau` file:\n\n```toml\n[lua_script]\nlang = \"luau\"\nfile = \"transform.luau\"\n```\n\nOnly one of `file` or `source` may be set — not both. When you run `tokf publish`, file references are automatically inlined (the file content is embedded as `source`) so the published filter is self-contained. The script file must reside within the filter's directory — path traversal (e.g. `../secret.txt`) is rejected.\n\n---\n\n\ntokf records input/output byte counts per run in a local SQLite database:\n\n```sh\ntokf gain              # summary: total bytes saved and reduction %\ntokf gain --daily      # day-by-day breakdown\ntokf gain --by-filter  # breakdown by filter\ntokf gain --json       # machine-readable output\n```\n\n## Remote gain\n\nView aggregate savings across all your registered machines via the tokf server:\n\n```sh\ntokf gain --remote              # summary across all machines\ntokf gain --remote --by-filter  # breakdown by filter\ntokf gain --remote --json       # machine-readable output\n```\n\nRemote gain requires authentication (`tokf auth login`). The `--daily` flag is not available remotely. See [Remote Sharing](#remote-sharing) for the full setup workflow.\n\n## Output history\n\ntokf records raw and filtered outputs in a local SQLite database, useful for debugging filters or reviewing what an AI agent saw:\n\n```sh\ntokf raw last                  # print raw output of last filtered command\ntokf raw 42                    # print raw output of entry #42\ntokf history list              # recent entries (current project)\ntokf history list -l 20        # show 20 entries\ntokf history list --all        # entries from all projects\ntokf history show 42           # full details for entry #42\ntokf history show --raw 42     # print only the raw captured output (long form)\ntokf history search \"error\"    # search by command or output content\ntokf history clear             # clear current project history\ntokf history clear --all       # clear all history (destructive)\n```\n\n## History hint\n\nWhen an LLM receives filtered output it may not realise the full output exists. Two mechanisms can automatically append a hint line pointing to the history entry:\n\n**1. Filter opt-in** — set `show_history_hint = true` in a filter TOML to always append the hint for that command:\n\n```toml\ncommand = \"git status\"\nshow_history_hint = true\n\n[on_success]\noutput = \"{branch} — {counts}\"\n```\n\n**2. Automatic repetition detection** — tokf detects when the same command is run twice in a row for the same project. This is a signal the caller didn't act on the previous filtered output and may need the full content:\n\n```\n🗜️ ✓ cargo test: 42 passed (2.31s)\n🗜️ compressed — run `tokf raw 99` for full output\n```\n\nThe `🗜️` prefix appears on all filtered output (disable with `tokf config set output.show_indicator false` or `TOKF_SHOW_INDICATOR=false`). The hint line is appended to stdout so it is visible to both humans and LLMs in the tool output. The history entry itself always stores the clean filtered output, without the hint line or indicator.\n\n## Context injection\n\nDuring `tokf hook install`, tokf creates a `.claude/TOKF.md` file and adds an `@TOKF.md` reference to `.claude/CLAUDE.md`. This gives LLMs a two-line context explaining what `🗜️` means and how to retrieve full output (`tokf raw last`). Use `--no-context` to skip this step.\n\n---\n\n\n## Setup\n\n```sh\ntokf auth login            # authenticate via GitHub device flow\ntokf remote setup          # register this machine\ntokf sync                  # upload pending usage events\ntokf gain --remote         # view aggregate savings across all machines\n```\n\n## Authentication\n\ntokf uses the [GitHub device flow](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/authorizing-oauth-apps#device-flow) so no secrets are handled locally. Tokens are stored in your OS keyring (Keychain on macOS, Secret Service on Linux, Credential Manager on Windows).\n\n```sh\ntokf auth login    # start device flow — prints a one-time code, opens browser\ntokf auth status   # show current login state and server URL\ntokf auth logout   # remove stored credentials\n```\n\n## Machine registration\n\nEach machine gets a UUID that links usage events to a physical device. Registration is idempotent — running it again re-syncs the existing record.\n\n```sh\ntokf remote setup    # register this machine with the server\ntokf remote status   # show local machine ID and hostname (no network call)\n```\n\nMachine config is stored in `~/.config/tokf/machine.toml` (or `$TOKF_HOME/machine.toml`).\n\n## Syncing usage data\n\n`tokf sync` uploads pending local usage events to the remote server. Events are deduplicated by cursor — re-syncing the same events is safe.\n\n```sh\ntokf sync              # upload pending events\ntokf sync --status     # show last sync time and pending event count (no network call)\n```\n\nA file lock prevents concurrent syncs. Both `tokf auth login` and `tokf remote setup` must be completed before syncing.\n\n## Viewing remote gain\n\nView aggregate token savings across all your registered machines:\n\n```sh\ntokf gain --remote              # summary: total runs, tokens saved, reduction %\ntokf gain --remote --by-filter  # breakdown by filter\ntokf gain --remote --json       # machine-readable output\n```\n\n\u003e **Note:** `--daily` is not available with `--remote`. Use local `tokf gain --daily` for day-by-day breakdowns.\n\n## Backfill\n\nUsage events recorded before hash-based tracking was added may be missing filter hashes. Backfill resolves them from currently installed filters:\n\n```sh\ntokf remote backfill             # update events with missing hashes\ntokf remote backfill --no-cache  # skip binary config cache during discovery\n```\n\nBackfill runs locally — no network call required.\n\n---\n\nFor discovering and installing community filters, see [Community Filters](#community-filters). To publish your own, see [Publishing Filters](#publishing-filters). For the full server API reference, see [`docs/reference/api.md`](docs/reference/api.md).\n\n---\n\n\n## Claude Code hook\n\ntokf integrates with [Claude Code](https://claude.ai/code) as a `PreToolUse` hook that **automatically filters every `Bash` tool call** — no changes to your workflow required.\n\n```sh\ntokf hook install          # project-local (.tokf/)\ntokf hook install --global # user-level (~/.config/tokf/)\n```\n\nOnce installed, every command Claude runs through the Bash tool is filtered transparently. Track cumulative savings with `tokf gain`.\n\n### Custom binary path\n\nBy default the generated hook script calls bare `tokf`, relying on PATH at runtime. If `tokf` isn't on PATH in the hook's execution environment (common with Linuxbrew or `cargo install` when PATH is only set in interactive shell profiles), pass `--path` to embed a specific binary location:\n\n```sh\ntokf hook install --global --path ~/.cargo/bin/tokf\ntokf hook install --tool opencode --path /home/linuxbrew/.linuxbrew/bin/tokf\n```\n\ntokf also ships a filter-authoring skill that teaches Claude the complete filter schema:\n\n```sh\ntokf skill install          # project-local (.claude/skills/)\ntokf skill install --global # user-level (~/.claude/skills/)\n```\n\n## Gemini CLI\n\ntokf integrates with [Gemini CLI](https://github.com/google-gemini/gemini-cli) as a `BeforeTool` hook that automatically filters `run_shell_command` tool calls.\n\n```sh\ntokf hook install --tool gemini-cli          # project-local (.gemini/)\ntokf hook install --tool gemini-cli --global # user-level (~/.gemini/)\n```\n\nThis registers a hook shim in `.gemini/settings.json` (or `~/.gemini/settings.json` for `--global`). When `--no-context` is not set, it also creates `.gemini/TOKF.md` and patches `.gemini/GEMINI.md` with context about the compression indicator.\n\n## Cursor\n\ntokf integrates with [Cursor](https://cursor.com) via a `beforeShellExecution` hook that automatically filters shell commands.\n\n```sh\ntokf hook install --tool cursor          # project-local (.cursor/)\ntokf hook install --tool cursor --global # user-level (~/.cursor/)\n```\n\nThis registers a hook in `.cursor/hooks.json` (or `~/.cursor/hooks.json` for `--global`). When `--no-context` is not set, it also creates `.cursor/rules/TOKF.md` with context about the compression indicator.\n\n## Cline\n\ntokf integrates with [Cline](https://cline.bot) via a rules file that instructs the agent to prefix supported commands with `tokf run`.\n\n```sh\ntokf hook install --tool cline          # project-local (.clinerules/)\ntokf hook install --tool cline --global # user-level (~/Documents/Cline/Rules/)\n```\n\nThis writes `.clinerules/tokf.md` (or `~/Documents/Cline/Rules/tokf.md` for `--global`), which Cline auto-discovers. The rules file uses `alwaysApply: true` frontmatter.\n\n## Windsurf\n\ntokf integrates with [Windsurf](https://windsurf.com) via a rules file.\n\n```sh\ntokf hook install --tool windsurf          # project-local (.windsurf/rules/)\ntokf hook install --tool windsurf --global # user-level (appends to global rules)\n```\n\nProject-local creates `.windsurf/rules/tokf.md`. Global mode appends a tokf section (with `\u003c!-- tokf:start/end --\u003e` markers for idempotent updates) to `~/.codeium/windsurf/memories/global_rules.md`.\n\n## GitHub Copilot\n\ntokf integrates with [GitHub Copilot](https://github.com/features/copilot) via instruction files. Copilot only supports repo-level instructions (no `--global` option).\n\n```sh\ntokf hook install --tool copilot\n```\n\nThis creates `.github/instructions/tokf.instructions.md` (with `applyTo: \"**\"` frontmatter) and appends a tokf section to `.github/copilot-instructions.md`.\n\n## Aider\n\ntokf integrates with [Aider](https://aider.chat) via conventions files.\n\n```sh\ntokf hook install --tool aider          # project-local (CONVENTIONS.md)\ntokf hook install --tool aider --global # user-level (patches ~/.aider.conf.yml)\n```\n\nProject-local appends a tokf section to `CONVENTIONS.md` (which Aider auto-discovers). Global mode writes a conventions file and adds it to `~/.aider.conf.yml`'s `read:` list.\n\n## OpenCode\n\ntokf integrates with [OpenCode](https://opencode.ai) via a plugin that applies filters in real-time before command execution.\n\n**Requirements:** OpenCode with Bun runtime installed.\n\n**Install (project-local):**\n```sh\ntokf hook install --tool opencode\n```\n\n**Install (global):**\n```sh\ntokf hook install --tool opencode --global\n```\n\nThis writes `.opencode/plugins/tokf.ts` (or `~/.config/opencode/plugins/tokf.ts` for `--global`), which OpenCode auto-loads. The plugin uses OpenCode's `tool.execute.before` hook to intercept `bash` tool calls and rewrites the command in-place when a matching filter exists. **Restart OpenCode after installation for the plugin to take effect.**\n\nIf tokf rewrite fails or no filter matches, the command passes through unmodified (fail-safe).\n\n## OpenAI Codex CLI\n\ntokf integrates with [OpenAI Codex CLI](https://github.com/openai/codex) via a skill that instructs the agent to prefix supported commands with `tokf run`.\n\n**Install (project-local):**\n```sh\ntokf hook install --tool codex\n```\n\n**Install (global):**\n```sh\ntokf hook install --tool codex --global\n```\n\nThis writes `.agents/skills/tokf-run/SKILL.md` (or `~/.agents/skills/tokf-run/SKILL.md` for `--global`), which Codex auto-discovers. Unlike the Claude Code hook (which intercepts commands at the tool level), the Codex integration is skill-based: it teaches the agent to use `tokf run` as a command prefix. If tokf is not installed, the agent falls back to running commands without the prefix (fail-safe).\n\n## Permission engines\n\ntokf supports pluggable permission engines that analyse commands and decide whether to allow, deny, or prompt. This is useful for auto-approving safe commands without manual confirmation.\n\n[Dippy](https://github.com/ldayton/Dippy) is an open-source permission engine with deep semantic analysis of bash commands — 34 CLI handlers covering git, aws, kubectl, docker, and more. See the [external permission engine](rewrites-config.md#external-permission-engine) section for configuration.\n\n### Hook handle command\n\nAll hook-based integrations (Claude Code, Gemini CLI, Cursor) call `tokf hook handle` internally. The `--format` flag tells tokf which response protocol to use:\n\n```sh\ntokf hook handle                    # default: claude-code\ntokf hook handle --format gemini    # Gemini CLI protocol\ntokf hook handle --format cursor    # Cursor protocol\n```\n\nThe hook scripts generated by `tokf hook install` set `--format` automatically — you don't need to pass it manually. The format also determines which JSON field the external permission engine should set (see [hook JSON reference](rewrites-config.md#hook-json-reference-for-engine-developers)).\n\n## Creating Filters with Claude\n\ntokf ships a Claude Code skill that teaches Claude the complete filter schema, processing order, step types, template pipes, and naming conventions.\n\n**Invoke automatically**: Claude will activate the skill whenever you ask to create or modify a filter — just describe what you want in natural language:\n\n\u003e \"Create a filter for `npm install` output that keeps only warnings and errors\"\n\u003e \"Write a tokf filter for `pytest` that shows a summary on success and failure details on fail\"\n\n**Invoke explicitly** with the `/tokf-filter` slash command:\n\n```\n/tokf-filter create a filter for docker build output\n```\n\nThe skill is in `.claude/skills/tokf-filter/SKILL.md`. Reference material (exhaustive step docs and an annotated example TOML) lives in `.claude/skills/tokf-filter/references/`.\n\n## Task runners\n\ntokf also integrates with task runners like `make` and `just` by injecting itself as the task runner's shell. Each recipe line is individually filtered while exit codes propagate correctly. See [Rewrite configuration](#rewrite-configuration-rewritestoml) for details.\n\n---\n\n\n## Rewrite configuration (`rewrites.toml`)\n\ntokf looks for a `rewrites.toml` file in two locations (first found wins):\n\n1. **Project-local**: `.tokf/rewrites.toml` — scoped to the current repository\n2. **User-level**: `~/.config/tokf/rewrites.toml` — applies to all projects\n\nThis file controls custom rewrite rules, skip patterns, and pipe handling. All `[pipe]`, `[skip]`, and `[[rewrite]]` sections documented below go in this file.\n\n## Task runner integration (make, just)\n\nTask runners like `make` and `just` execute recipe lines via a shell (`$SHELL -c 'recipe_line'`). By default, only the outer `make`/`just` command is visible to tokf — child commands (`cargo test`, `uv run mypy`, etc.) pass through unfiltered.\n\ntokf solves this with **built-in wrapper rules** that inject tokf as the task runner's shell. Each recipe line is then individually matched against installed filters:\n\n```sh\n# What you type:\nmake check\n\n# What tokf rewrites it to:\nmake SHELL=tokf check\n\n# What make then does for each recipe line:\ntokf -c 'cargo test'          → filter matches → filtered output\ntokf -c 'cargo clippy'        → filter matches → filtered output\ntokf -c 'echo done'           → no filter → delegates to sh\n```\n\nFor `just`, the `--shell` flag is used instead:\n\n```sh\njust test  →  just --shell tokf --shell-arg -cu test\n```\n\n### Exit code preservation\n\nShell mode (`tokf -c '...'`) always propagates the **real exit code** — no masking, no \"Error: Exit code N\" prefix. This means `make` sees the actual exit code from each recipe line and stops on failure as expected.\n\n### Shell mode (`tokf -c`)\n\nWhen invoked as `tokf -c 'command'` (or with combined flags like `-cu`, `-ec`), tokf enters **string mode**. The command string is passed through the rewrite system, which rewrites matching commands to `tokf run --no-mask-exit-code ...`. The rewritten command is then delegated to `sh -c` for execution. If no filter matches, the command is delegated to `sh` unchanged.\n\nWhen invoked with multiple arguments after `-c` (e.g. `tokf -c git status`), tokf enters **argv mode**. Each argument is shell-escaped and joined into a command string, which is then processed the same way as string mode. This form is used by PATH shims.\n\nShell mode is not typically invoked directly; it is called by task runners (make, just) and PATH shims.\n\n### Compound and complex recipe lines\n\nCompound commands (`\u0026\u0026`, `||`, `;`) are split at chain operators and each segment is individually rewritten. This means both halves of `git add . \u0026\u0026 cargo test` can be filtered. Pipes, redirections, and other shell constructs within each segment are handled by the rewrite system's pipe stripping logic (see [Piped commands](#piped-commands)) or passed through to `sh` unchanged.\n\n### Debugging task runner rewrites\n\nUse `tokf rewrite --verbose \"make check\"` to confirm the wrapper rewrite is active and see which rule fired.\n\nShell mode also respects environment variables for diagnostics (since it has no access to CLI flags like `--verbose`):\n\n```sh\nTOKF_VERBOSE=1 make check     # print filter resolution details for each recipe line\nTOKF_NO_FILTER=1 make check   # bypass filtering entirely, delegate all recipe lines to sh\n```\n\n### Overriding or disabling wrappers\n\nThe built-in wrappers for `make` and `just` can be overridden or disabled via `[[rewrite]]` or `[skip]` entries in `.tokf/rewrites.toml`:\n\n```toml\n# Override the make wrapper with a custom one:\n# \"make check\" → \"make SHELL=tokf .SHELLFLAGS=-ec check\"\n# Note: use (?:[^\\\\s]*/)? prefix to also match full paths like /usr/bin/make\n[[rewrite]]\nmatch = \"^(?:[^\\\\s]*/)?make(\\\\s.*)?$\"\nreplace = \"make SHELL=tokf .SHELLFLAGS=-ec{1}\"\n\n# Or disable it entirely:\n[skip]\npatterns = [\"^make\"]\n```\n\n### Adding wrappers for other task runners\n\nYou can add wrappers for other task runners via `[[rewrite]]`. The exact mechanism depends on how the task runner invokes recipe lines — check its documentation for shell override options:\n\n```toml\n# Example: if your task runner respects $SHELL for recipe execution\n[[rewrite]]\nmatch = \"^(?:[^\\\\s]*/)?mise run(\\\\s.*)?$\"\nreplace = \"SHELL=tokf mise run{1}\"\n```\n\n## Routing to generic commands\n\nFor commands that don't have a dedicated filter, you can route them through [generic commands](generic-commands.md) (`tokf err`, `tokf test`, `tokf summary`) via rewrite rules:\n\n```toml\n# .tokf/rewrites.toml\n\n# Build commands → error extraction\n[[rewrite]]\nmatch = \"^mix compile\"\nreplace = \"tokf err {0}\"\n\n# Test runners → failure extraction\n[[rewrite]]\nmatch = \"^mix test\"\nreplace = \"tokf test {0}\"\n\n# Long-running commands → heuristic summary\n[[rewrite]]\nmatch = \"^terraform plan\"\nreplace = \"tokf summary {0}\"\n```\n\n**Note:** User rewrite rules fire *before* filter matching. Only add these for commands that don't already have a filter — check with `tokf which \"\u003ccommand\u003e\"`. Commands with dedicated filters (e.g. `cargo build`, `git status`) produce better output through `tokf run`.\n\n## Piped commands\n\nWhen a command is piped to a simple output-shaping tool (`grep`, `tail`, or `head`), tokf **strips the pipe automatically** and uses its own structured filter output instead. The original pipe suffix is passed to `--baseline-pipe` so token savings are still calculated accurately.\n\n```sh\n# These ARE rewritten — pipe is stripped, tokf applies its filter:\ncargo test | grep FAILED\ncargo test | tail -20\ngit diff HEAD | head -5\n```\n\nMulti-pipe chains, pipes to other commands, or pipe targets with unsupported flags are left unchanged:\n\n```sh\n# These are NOT rewritten — tokf leaves them alone:\nkubectl get pods | grep Running | wc -l   # multi-pipe chain\ncargo test | wc -l                        # wc not supported\ncargo test | tail -f                      # -f (follow) not supported\n```\n\nIf you want tokf to wrap a piped command that wouldn't normally be rewritten, add an explicit rule to `.tokf/rewrites.toml`:\n\n```toml\n[[rewrite]]\nmatch = \"^cargo test \\\\| tee\"\nreplace = \"tokf run {0}\"\n```\n\nUse `tokf rewrite --verbose \"cargo test | grep FAILED\"` to see how a command is being rewritten.\n\n### Disabling pipe stripping\n\nIf you prefer tokf to never strip pipes (leaving piped commands unchanged), add a `[pipe]` section to `.tokf/rewrites.toml`:\n\n```toml\n[pipe]\nstrip = false   # default: true\n```\n\nWhen `strip = false`, commands like `cargo test | tail -5` pass through the shell unchanged. Non-piped commands are still rewritten normally.\n\n### Prefer less context mode\n\nSometimes the piped output (e.g. `tail -5`) is actually smaller than the filtered output. The `prefer_less` option tells tokf to compare both at runtime and use whichever is smaller:\n\n```toml\n[pipe]\nprefer_less = true   # default: false\n```\n\nWhen a pipe is stripped, tokf injects `--prefer-less` alongside `--baseline-pipe`. At runtime:\n1. The filter runs normally\n2. The original pipe command also runs on the raw output\n3. tokf prints whichever result is smaller\n\nWhen the pipe output wins, the event is recorded with `pipe_override = 1` in the tracking DB. The `tokf gain` command shows how many times this happened:\n\n```\ntokf gain summary\n  total runs:     42\n  input tokens:   12,500 est.\n  output tokens:  3,200 est.\n  tokens saved:   9,300 est. (74.4%)\n  pipe preferred: 5 runs (pipe output was smaller than filter)\n```\n\nNote: `strip = false` takes priority — if pipe stripping is disabled, `prefer_less` has no effect.\n\n## External permission engine\n\nBy default, tokf does **no** permission checking — the AI tool (Claude Code, Gemini, Cursor) handles its own deny/ask rules natively. tokf only rewrites commands that match a filter and auto-allows them.\n\nYou can optionally delegate permission decisions to an external process — a \"sub-hook\" that performs deeper semantic analysis of commands. When configured, the engine is consulted on **every** command, not just ones tokf has a filter for. This lets the engine auto-approve safe commands without the user being prompted.\n\n### Configuration\n\nAdd a `[permissions]` section to `.tokf/rewrites.toml`:\n\n```toml\n[permissions]\nengine = \"external\"\n\n[permissions.external]\ncommand = \"dippy\"\nargs = [\"hook\", \"handle\", \"--mode\", \"{format}\"]\ntimeout_ms = 3000    # default: 5000\non_error = \"builtin\" # what to do if the engine fails\n```\n\n### Tool format (`{format}`)\n\nThe `{format}` placeholder in `args` is replaced with the AI tool identifier before spawning:\n\n| Tool | Default value |\n|---|---|\n| Claude Code | `claude-code` |\n| Gemini CLI | `gemini` |\n| Cursor | `cursor` |\n\nIf the engine expects different names, add a `format_map`:\n\n```toml\n[permissions.external]\ncommand = \"my-engine\"\nargs = [\"check\", \"--tool\", \"{format}\"]\nformat_map = { \"claude-code\" = \"claude\", \"gemini\" = \"google\" }\n```\n\n### Protocol\n\n1. tokf spawns the engine process (`command` + `args`, with `{format}` resolved)\n2. The original hook JSON (from the AI tool) is written to the engine's **stdin**\n3. The engine returns a standard hook response JSON on **stdout** — the same format the AI tool expects\n4. tokf extracts the permission decision from the response and applies it to its own rewritten command\n\nThe engine sees the **original** command (not the rewritten one). tokf controls the rewrite; the engine controls the permission decision. Engines that exit with a non-zero status but still produce valid JSON will have their JSON verdict honoured — this supports engines like Dippy that use exit codes for signalling alongside a valid response.\n\n### Hook JSON reference (for engine developers)\n\nThe engine receives the AI tool's hook JSON verbatim on stdin. The format depends on the tool:\n\n**Claude Code** (`--mode claude-code`):\n\n```json\n{\"tool_name\": \"Bash\", \"tool_input\": {\"command\": \"git push --force\"}}\n```\n\nExpected response — set `permissionDecision` to `\"allow\"`, `\"deny\"`, or `\"ask\"` (or omit for ask):\n\n```json\n{\n  \"hookSpecificOutput\": {\n    \"hookEventName\": \"PreToolUse\",\n    \"permissionDecision\": \"allow\",\n    \"updatedInput\": {\"command\": \"git push --force\"}\n  }\n}\n```\n\n**Gemini CLI** (`--mode gemini`):\n\n```json\n{\"tool_name\": \"run_shell_command\", \"tool_input\": {\"command\": \"git push --force\"}}\n```\n\nExpected response — set `decision` to `\"allow\"`, `\"deny\"`, or `\"ask\"`:\n\n```json\n{\n  \"decision\": \"allow\",\n  \"hookSpecificOutput\": {\n    \"tool_input\": {\"command\": \"git push --force\"}\n  }\n}\n```\n\n**Cursor** (`--mode cursor`):\n\n```json\n{\"command\": \"git push --force\"}\n```\n\nExpected response — set `permission` to `\"allow\"`, `\"deny\"`, or `\"ask\"`:\n\n```json\n{\n  \"permission\": \"allow\",\n  \"updated_input\": {\"command\": \"git push --force\"}\n}\n```\n\n### Error handling (`on_error`)\n\nWhen the engine fails (crash, timeout, invalid output), the `on_error` field determines the fallback:\n\n| Value | Behaviour |\n|---|---|\n| `\"ask\"` (default) | Fail closed — prompt user for permission |\n| `\"allow\"` | Fail open — auto-allow the command |\n| `\"builtin\"` | Fall back to built-in deny/ask rule matching |\n\n### Example: Dippy integration\n\n[Dippy](https://github.com/ldayton/Dippy) is a permission engine that performs deep semantic analysis of bash commands — auto-approving safe commands while blocking dangerous ones.\n\n```toml\n[permissions]\nengine = \"external\"\n\n[permissions.external]\ncommand = \"dippy\"\nargs = [\"hook\", \"handle\", \"--mode\", \"{format}\"]\ntimeout_ms = 3000\non_error = \"builtin\"\n```\n\n## Environment variable prefixes\n\nLeading `KEY=VALUE` assignments are automatically stripped before matching, so env-prefixed commands are rewritten correctly:\n\n```sh\n# These ARE rewritten — env vars are preserved, the command is wrapped:\nDEBUG=1 git status              → DEBUG=1 tokf run git status\nRUST_LOG=debug cargo test       → RUST_LOG=debug tokf run cargo test\nA=1 B=2 cargo test | tail -5   → A=1 B=2 tokf run --baseline-pipe 'tail -5' cargo test\n```\n\nThe env vars are passed through verbatim to the underlying command; tokf only rewrites the executable portion.\n\n### Skip patterns and env var prefixes\n\nUser-defined skip patterns in `.tokf/rewrites.toml` match against the **full** shell segment, including any leading env vars. A pattern `^cargo` will **not** skip `RUST_LOG=debug cargo test` because the segment doesn't start with `cargo`:\n\n```toml\n[skip]\npatterns = [\"^cargo\"]   # skips \"cargo test\" but NOT \"RUST_LOG=debug cargo test\"\n```\n\nTo skip a command regardless of any env prefix, use a pattern that accounts for it:\n\n```toml\n[skip]\npatterns = [\"(?:^|\\\\s)cargo\\\\s\"]   # matches \"cargo\" anywhere after start or whitespace\n```\n\n## Implicit skip rules\n\nThree implicit skip rules are always active and can't be disabled. They cover cases where rewriting would silently corrupt the agent's data, so there's no plausible reading under which the rewrite would be correct.\n\n### Heredocs\n\nCommands that contain a top-level heredoc (`\u003c\u003cEOF`, `\u003c\u003c-EOF`) are passed through unchanged. Wrapping them with `tokf run` would break the lexical binding between the command and its heredoc body.\n\n```sh\n# Not rewritten — heredoc body would be cut off:\ncat \u003c\u003cEOF \u003e /tmp/cfg.yaml\nkey: value\nEOF\n```\n\n### Heredocs inside command substitution\n\nCommands that contain a heredoc anywhere inside a `$(...)` (or backtick) command substitution are also skipped — even when the substitution is buried deep inside an argument like `git commit -m \"$(cat \u003c\u003c'EOF' … EOF)\"`. The heredoc body lives in a logically-separate region of the source from the surrounding command, and downstream re-tokenization (clap argv parsing in `tokf run`, second-pass shell parsers, byte-offset pipe stripping) can slice through it. The canonical failure mode this prevents is `git commit -m \"$(cat \u003c\u003c'EOF' … EOF)\" 2\u003e\u00261 | tail -10` mangling `-m`'s value into `git: error: switch 'm' requires a value`.\n\n```sh\n# Not rewritten — multi-line commit messages, gh PR bodies, etc.:\ngit add foo \u0026\u0026 git commit -m \"$(cat \u003c\u003c'EOF'\nfeat: a thing\n\nwith multi-line body\nEOF\n)\" \u0026\u0026 git push\n\ngh pr create -b \"$(cat \u003c\u003cEOF\n## Summary\n…\nEOF\n)\"\n```\n\nUnlike the output-redirect skip below, this rule fires for the **whole compound**: if any segment contains a substitution-nested heredoc, sibling `git add` / `git push` segments are also passed through. This is intentionally conservative — re-emitting *any* part of a compound that contains this construct risks downstream byte-offset slicing into the heredoc body.\n\n### Output redirection\n\nCommands that redirect output to a file (`\u003e`, `\u003e\u003e`, `\u0026\u003e`, `\u0026\u003e\u003e`, `\u003e|`, `1\u003e`, `2\u003e`, `\u003c\u003e`, etc.) are also passed through unchanged. The agent explicitly redirected to a file because they want the **raw** output for downstream processing — interposing tokf's filter would write filtered bytes into the file and silently corrupt what the agent reads back.\n\n```sh\n# These are NOT rewritten — tokf leaves them alone:\ngit diff \u003e /tmp/diff.txt          # explicit output redirect — agent wants raw form\ngit log --all \u003e history.txt       # for grep/awk processing on the file\ncargo test \u003e test.log 2\u003e\u00261        # combined output to file\ngit status \u003e /dev/null            # output discarded; nothing to filter\nexec 3\u003c\u003e /tmp/sock                # read+write file open\n\n# These ARE still rewritten — fd remap only, no file involved:\ngit diff 2\u003e\u00261                     # merges stderr into stdout, both still feed the agent\ngit diff 1\u003e\u00262                     # redirects stdout to stderr — still no file\ngit diff \u003e\u0026-                      # closes a file descriptor\n\n# Compound commands are handled per-segment — only the segment with the\n# redirect is skipped, the other segments are still rewritten:\ngit diff \u003e foo.txt; git status    # → \"git diff \u003e foo.txt; tokf run git status\"\ngit status \u0026\u0026 git diff \u003e foo.txt  # → \"tokf run git status \u0026\u0026 git diff \u003e foo.txt\"\n```\n\n`tee` in a pipeline (`git diff | tee log.txt`) is **not** currently treated as an output redirect because `tee` is a command argument, not a redirect operator. The current pipe-handling behaviour is preserved. This is a known follow-up.\n\n## Debug settings\n\nThe `[debug]` section enables diagnostic logging for the rewrite system. All settings are off by default.\n\n```toml\n[debug]\nlog_parse_failures = true\n```\n\n| Field | Default | Description |\n|---|---|---|\n| `log_parse_failures` | `false` | Log to stderr when the bash parser (rable) fails to parse a command, causing the rewrite system to fall back to simple string matching. Helps diagnose unexpected \"unmatched quote\" errors or commands that should have been skipped but weren't. |\n\n---\n\n\n## tokf info\n\n`tokf info` prints a summary of all paths, database locations, and filter counts. Useful for debugging when filters aren't being found or to verify your setup:\n\n```sh\ntokf info          # human-readable output\ntokf info --json   # machine-readable JSON\n```\n\nExample output:\n\n```\ntokf 0.2.8\nTOKF_HOME: (not set)\n\nfilter search directories:\n  [local] /home/user/project/.tokf/filters (not found)\n  [user] /home/user/.config/tokf/filters (not found)\n  [built-in] \u003cembedded\u003e (always available)\n\ntracking database:\n  TOKF_DB_PATH: (not set)\n  path: /home/user/.local/share/tokf/tracking.db (will be created)\n\nfilter cache:\n  path: /home/user/.cache/tokf/manifest.bin (will be created)\n\nfilters:\n  local:    0\n  user:     0\n  built-in: 38\n  total:    38\n```\n\n### Environment variables\n\n| Variable | Description | Default |\n|---|---|---|\n| `TOKF_HOME` | Redirect **all** user-level tokf paths (filters, cache, DB, hooks, auth) to a single directory | Platform config dir (e.g. `~/.config/tokf` on Linux) |\n| `TOKF_DB_PATH` | Override the tracking database path only (takes precedence over `TOKF_HOME`) | Platform data dir (e.g. `~/.local/share/tokf/tracking.db`); or `$TOKF_HOME/tracking.db` when `TOKF_HOME` is set |\n| `TOKF_NO_FILTER` | Skip filtering in shell mode (set to `1`, `true`, or `yes`) | unset |\n| `TOKF_VERBOSE` | Print filter resolution details in shell mode | unset |\n\n`TOKF_HOME` works like `CARGO_HOME` or `RUSTUP_HOME` — set it once to relocate everything:\n\n```sh\n# Put all tokf data under /opt/tokf (useful on read-only home dirs or shared systems)\nTOKF_HOME=/opt/tokf tokf info\n\n# Override only the tracking database, leave everything else in the default location\nTOKF_DB_PATH=/tmp/my-tracking.db tokf info\n```\n\nThe `tokf info` output always shows the active `TOKF_HOME` value (or `(not set)`) at the top,\nso you can quickly verify which paths are in effect.\n\n## Rewrite debugging\n\nUse `tokf rewrite --verbose` to see how a command would be rewritten, including which rule fired:\n\n```sh\ntokf rewrite --verbose \"make check\"         # shows wrapper rule\ntokf rewrite --verbose \"cargo test\"          # shows filter rule\ntokf rewrite --verbose \"cargo test | tail\"   # shows pipe stripping\n```\n\nFor shell mode (task runner recipe lines), set `TOKF_VERBOSE=1` to see filter resolution for each recipe line:\n\n```sh\nTOKF_VERBOSE=1 make check    # verbose output on stderr for each recipe\n```\n\n## tokf doctor\n\n`tokf doctor` analyses your local `tracking.db` and reports filters that may be causing **agent confusion** — repeated calls, escape-flag usage, empty-output retries, or filters that are making output *bigger* than the raw command. It's the post-hoc complement to `tokf gain`: where `gain` measures how much you saved, `doctor` looks for places the savings were illusory because the agent had to retry.\n\n```sh\ntokf doctor                              # default: current project, table output\ntokf doctor --json                       # machine-readable\ntokf doctor --filter git/diff            # focus on one filter\ntokf doctor --all                        # include events from every project\ntokf doctor --burst-threshold 3 --window 30   # tighten the burst detector\ntokf doctor --sort bursts                # sort by burst count instead of health\n```\n\nExample output:\n\n```\ntokf doctor — 41057 events, project=/Users/me/repo, threshold≥5 within 60s\n\nfilter                  events  score    bursts max-burst   retries\n──────────────────────────────────────────────────────────────────────\ngit diff                  4056     20        67        17       304\ngit log                    920     35         9         8        46\ngit show                   183     45         2         3         0\ngit status                2418     85         1         5        12\ncargo test                 412    100         0         -         -\n\nretry-burst detail (top 5 by size)\n  ×17 git diff \u003cargs\u003e (git diff)\n  ×12 git diff \u003cargs\u003e (git diff)\n  ×11 git diff \u003cargs\u003e (git diff)\n  ×10 git diff \u003cargs\u003e (git diff)\n  ×9 git diff \u003cargs\u003e (git diff)\n\nworkaround-flag suggestions (consider adding to passthrough_args)\n  git diff: --no-pager×35, --oneline×4\n  git log: --no-pager×64, --pretty×4\n\nfilters with negative token savings (filtered output \u003e raw)\n  +122.8 avg tokens per call — git show\n  +5.4 avg tokens per call — git log\n```\n\n### What each section means\n\n| Section | What it detects | Threshold |\n|---|---|---|\n| **filter table** | Per-filter health summary, sorted by composite score (lower = worse) | `score = 100 − burst_penalty − workaround_penalty − empty_retry_penalty − negative_savings_penalty`, each capped so no single signal can crash the score on its own |\n| **retry-burst detail** | The same exact command run ≥`--burst-threshold` (default `5`) times within `--window` seconds (default `60`). Shows top 5 burst sessions by size. | Strong signal that the model didn't believe / couldn't read the filtered output and kept trying |\n| **workaround-flag suggestions** | Flags like `--no-stat`, `--no-pager`, `-p`, `--name-only`, `--format` that appear often **but are not declared in the filter's `passthrough_args`** | Each occurrence is the agent trying to escape the filter; if a flag appears repeatedly, the filter probably should add it to `passthrough_args` |\n| **filters with negative token savings** | Filters where the average filtered output is **larger** than the raw command output | Usually caused by `on_empty` adding explanatory text to a small command, or stat tables expanding short diffs. The fix is filter-specific |\n\n### Multi-project handling\n\n`tracking.db` records the project root for every event (resolved by walking up from the cwd looking for `.git` / `.tokf`). By default, `tokf doctor` scopes its analysis to the current project. Use:\n\n- `--project /path/to/repo` — analyse a specific project\n- `--all` — analyse events from every project together\n\nEvents recorded **before** the project column was added (legacy rows in upgraded DBs) are visible from every scope until they age out naturally.\n\n### Noise filtering\n\nThe doctor excludes events whose command path looks like a temp-dir or test-fixture invocation by default — `/var/folders/...`, `/tmp/...`, `.tokf-verify-...`, etc. These are usually statusline / shell-prompt callers, `tokf verify` rigs, or hook scripts running before/after every tool call, none of which are agent confusion. Use `--include-noise` to disable the filter when you want to see *everything*.\n\n### What's not included (yet)\n\n`tokf doctor` is the **post-hoc** half of the diagnostics story. Phase 2 will add **runtime surfacing** — an in-process LRU that detects bursts as they happen and prints a `[tokf] notice:` line on stderr in the same tool result the agent sees. Phase 3 will add an `--apply-suggestions` interactive mode that proposes config patches. Both are explicitly out of scope for the current release.\n\n## Cache management\n\ntokf caches the filter discovery index for faster startup. The cache rebuilds automatically when filters change, but you can manage it manually:\n\n```sh\ntokf cache info    # show cache location, size, and validity\ntokf cache clear   # delete the cache, forcing a rebuild on next run\n```\n\n## Shell completions\n\nGenerate tab-completion scripts for your shell:\n\n```sh\ntokf completions bash\ntokf completions zsh\ntokf completions fish\ntokf completions powershell\ntokf completions elvish\ntokf completions nushell\n```\n\n### Installation\n\n**Bash** — add to `~/.bashrc`:\n```sh\neval \"$(tokf completions bash)\"\n```\n\n**Zsh** — add to `~/.zshrc`:\n```sh\neval \"$(tokf completions zsh)\"\n```\n\n**Fish** — save to completions directory:\n```sh\ntokf completions fish \u003e ~/.config/fish/completions/tokf.fish\n```\n\n**PowerShell** — add to your profile:\n```powershell\ntokf completions powershell | Out-String | Invoke-Expression\n```\n\n**Elvish** — add to `~/.elvish/rc.elv`:\n```sh\neval (tokf completions elvish | slurp)\n```\n\n**Nushell** — save and source in your config:\n```sh\ntokf completions nushell | save -f ~/.config/nushell/tokf.nu\nsource ~/.config/nushell/tokf.nu\n```\n\n---\n\n\n## Overview\n\nThe tokf community registry lets you discover filters published by other users, install them\nlocally, and share your own filters with the community.\n\nAuthentication is required for all registry operations. Run `tokf auth login` first.\n\n---\n\n## Searching for Filters\n\n```sh\ntokf search \u003cquery...\u003e\n```\n\nReturns filters whose command pattern matches `\u003cquery\u003e` as a substring, ranked by token savings\nand install count. Multi-word queries work without quotes:\n\n```sh\ntokf search git push         # no quotes needed\ntokf search \"git push\"       # also works\n```\n\n### Interactive Mode\n\nWhen stderr is a terminal, search displays an interactive menu with arrow-key selection.\nChoosing a filter flows directly into `tokf install`:\n\n```\n\u003e git push [stdlib]  @mpecan  savings:45%  tests:3  runs:12,234\n  git push --force   @alice   savings:38%  tests:1  runs:891\n  cargo build        @bob     savings:80%  tests:2  runs:500\n```\n\nPress Enter to install the selected filter, or Escape to cancel.\n\n### Non-interactive Mode\n\nWhen stderr is not a terminal (for example, when its output is piped: `tokf search git 2\u003e\u00261 | cat`), a static table is printed to stderr:\n\n```\nCOMMAND              AUTHOR    SAVINGS%  TESTS      RUNS\ngit push             alice       42.3%      3     1,234\ngit push --force     bob         38.1%      1       891\n```\n\n### Options\n\n| Flag | Description |\n|------|-------------|\n| `-n, --limit \u003cN\u003e` | Maximum results to return (default: 20, max: 100) |\n| `--json` | Output raw JSON array to stdout (no interactive UI) |\n\n\u003e **Note:** Flags (`--json`, `-n`) must come **before** the query words.\n\u003e `tokf search --json git push` works; `tokf search git push --json` sends `--json` as part of\n\u003e the query string.\n\n### Examples\n\n```sh\ntokf search git              # find all git filters (interactive if TTY)\ntokf search cargo test       # multi-word query, no quotes needed\ntokf search -n 50 \"\"         # list 50 most popular filters\ntokf search --json git       # machine-readable JSON output\n```\n\n---\n\n## Installing a Filter\n\n```sh\ntokf install \u003cfilter\u003e\n```\n\n`\u003cfilter\u003e` can be:\n\n- A **command pattern** substring — tokf searches the registry and installs the top match.\n- A **content hash** (64 hex characters) — installs a specific, pinned version.\n\nOn install, tokf:\n\n1. Downloads the filter TOML and any bundled test files.\n2. Verifies the content hash to detect tampering.\n3. Writes the filter under `~/.config/tokf/filters/` (global) or `.tokf/filters/` (local).\n4. Runs the bundled test suite (if any). Rolls back on failure.\n\n### Options\n\n| Flag | Description |\n|------|-------------|\n| `--local` | Install to project-local `.tokf/filters/` instead of global config |\n| `--force` | Overwrite an existing filter at the same path |\n| `--dry-run` | Preview what would be installed without writing any files |\n\n### Examples\n\n```sh\ntokf install git push                  # install top result for \"git push\"\ntokf install git push --local          # install into current project only\ntokf install git push --dry-run        # preview the install\ntokf install \u003c64-hex-hash\u003e --force     # install a pinned version, overwriting existing\n```\n\n### Attribution\n\nInstalled filters include an attribution header at the top of the TOML:\n\n```toml\n# Published by @alice · hash: \u003chash\u003e · https://tokf.net/filters/\u003chash\u003e\n```\n\nThis header is stripped automatically when the filter is loaded.\n\n### Security\n\n\u003e **Warning:** Community filters are third-party code. Review a filter at\n\u003e `https://tokf.net/filters/\u003chash\u003e` before installing it in production environments.\n\ntokf verifies the content hash of every downloaded filter to detect server-side tampering.\nTest filenames are validated to prevent path traversal attacks.\n\n---\n\n## Updating Test Suites\n\nAfter publishing a filter, the filter TOML itself is immutable (same content = same hash), but you\ncan replace the bundled test suite at any time:\n\n```sh\ntokf publish --update-tests \u003cfilter-name\u003e\n```\n\nThis replaces the **entire** test suite in the registry with the current local `_test/` directory\ncontents. Only the original author can update tests.\n\n### Options\n\n| Flag | Description |\n|------|-------------|\n| `--dry-run` | Preview which test files would be uploaded without making changes |\n\n### Examples\n\n```sh\ntokf publish --update-tests git/push            # replace test suite for git/push\ntokf publish --update-tests git/push --dry-run  # preview only\n```\n\n### Notes\n\n- The filter's identity (content hash) does not change.\n- The old test suite is deleted and fully replaced by the new one.\n- You must be the original author of the filter.\n\n---\n\n---\n\n\n## tokf discover\n\n`tokf discover` scans Claude Code session files to find commands that have **no matching tokf filter**, helping you identify where to create new filters for maximum token savings.\n\nBy default, commands that already have a matching filter are hidden — if the hook is installed, those are already being filtered. Use `--include-filtered` to see the full picture including commands with existing filters.\n\n```bash\n# Scan sessions for the current project\ntokf discover\n\n# Also show commands that have existing filters\ntokf discover --include-filtered\n\n# Scan all projects\ntokf discover --all\n\n# Only sessions from the last 7 days\ntokf discover --since 7d\n\n# JSON output for programmatic use\ntokf discover --json\n```\n\nExample output:\n\n```\n[tokf] scanned 12 sessions, 847 commands total\n[tokf] 203 already filtered by tokf\n[tokf] 201 commands have filters (use --include-filtered to show)\n\nCOMMAND                        FILTER               RUNS     TOKENS\n----------------------------------------------------------------------\npython manage.py migrate       (none)                 34      12.1k\nterraform plan                 (none)                 28       9.8k\nhelm upgrade                   (none)                 15       6.2k\n\nTotal unfiltered output: 28.1k tokens across 443 commands\n```\n\n### Options\n\n| Flag | Description |\n|------|-------------|\n| `--project \u003cpath\u003e` | Scan sessions for a specific project path |\n| `--all` | Scan sessions across all projects |\n| `--session \u003cpath\u003e` | Scan a single session JSONL file |\n| `--since \u003cduration\u003e` | Filter by recency: `7d`, `24h`, `30m` |\n| `--limit \u003cn\u003e` | Number of results to show (0 = all, default: 20) |\n| `--json` | Output as JSON |\n| `--include-filtered` | Also show commands that already have a matching filter |\n\n### How It Works\n\n1. Locates Claude Code session JSONL files in `~/.claude/projects/`\n2. Extracts all Bash `tool_use` / `tool_result` pairs\n3. Skips commands already wrapped with `tokf run`\n4. Matches remaining commands against available tokf filters\n5. By default shows only commands with no matching filter\n6. Aggregates and ranks by estimated token count\n\n---\n\n\n## Publishing a Filter\n\n```sh\ntokf publish \u003cfilter-name\u003e\n```\n\nPublishes a local filter to the community registry under the MIT license. Authentication is required — run `tokf auth login` first.\n\n### Requirements\n\n- The filter must be a **user-level or project-local** filter (not a built-in). Use `tokf eject` first if needed.\n- At least one **test file** must exist in the adjacent `_test/` directory. The server runs these tests against your filter before accepting the upload.\n- You must accept the **MIT license** (prompted on first publish, remembered afterwards).\n\n### What happens on publish\n\n1. The filter TOML is read and validated.\n2. If the filter uses `lua_script.file`, the referenced script is **automatically inlined** — its content is embedded as `lua_script.source` so the published filter is self-contained. The script file must reside within the filter's directory (path traversal is rejected).\n3. A content hash is computed from the parsed config. This hash is the filter's permanent identity.\n4. The filter and test files are uploaded. The server verifies tests pass before accepting.\n5. On success, the registry URL is printed.\n\n### Options\n\n| Flag | Description |\n|------|-------------|\n| `--dry-run` | Preview what would be published without uploading |\n| `--update-tests` | Replace the test suite for an already-published filter |\n\n### Examples\n\n```sh\ntokf publish git/push                  # publish a filter\ntokf publish git/push --dry-run        # preview only\ntokf publish --update-tests git/push   # replace test suite\n```\n\n### Size limits\n\n- Filter TOML: 64 KB max\n- Total upload (filter + tests): 1 MB max\n\n### Lua scripts in published filters\n\nPublished filters must use **inline `source`** for Lua scripts — `lua_script.file` is not supported on the server. The `tokf publish` command handles this automatically by reading the file and embedding its content. You don't need to change your filter.\n\nAll Lua scripts in published filters are executed in a sandbox with resource limits (1 million instructions, 16 MB memory) during server-side test verification.\n\n---\n\n## Server API\n\nFor the full API reference (all endpoints, request/response shapes, rate limits, and environment variables), see [`docs/reference/api.md`](docs/reference/api.md). For deployment instructions, see [`DEPLOY.md`](DEPLOY.md).\n\n---\n\n## Acknowledgements\n\ntokf was heavily inspired by [rtk](https://github.com/rtk-ai/rtk) ([rtk-ai.app](https://www.rtk-ai.app/)) — a CLI proxy that compresses command output before it reaches an AI agent's context window. rtk pioneered the idea and demonstrated that 60–90% context reduction is achievable across common dev tools. tokf takes a different approach (TOML-driven filters, user-overridable library, Claude Code hook integration) but the core insight is theirs.\n\n---\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpecan%2Ftokf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmpecan%2Ftokf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpecan%2Ftokf/lists"}