{"id":51118882,"url":"https://github.com/detain/php-dup-finder","last_synced_at":"2026-06-25T00:30:26.047Z","repository":{"id":356637242,"uuid":"1233422271","full_name":"detain/php-dup-finder","owner":"detain","description":"AST-based PHP duplicate-logic detector and refactoring assistant. Finds parameterizable duplication via canonicalization, n-gram fingerprinting, bounded tree-edit-distance, and anti-unification — proposes extract-function signatures with typed parameter holes. CLI, JSON, and static-HTML reports.","archived":false,"fork":false,"pushed_at":"2026-05-31T19:56:14.000Z","size":10442,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-05-31T21:23:20.003Z","etag":null,"topics":["abstract-syntax-tree","anti-unification","ast","cli","clone-detection","code-analysis","code-duplication","code-quality","copy-paste-detector","developer-tools","ngram","php","php8","refactoring","refactoring-tools","similarity","static-analysis","technical-debt","tree-edit-distance"],"latest_commit_sha":null,"homepage":"https://github.com/detain/php-dup-finder","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/detain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-09T00:14:29.000Z","updated_at":"2026-05-31T19:56:19.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/detain/php-dup-finder","commit_stats":null,"previous_names":["detain/php-dup-finder"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/detain/php-dup-finder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/detain%2Fphp-dup-finder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/detain%2Fphp-dup-finder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/detain%2Fphp-dup-finder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/detain%2Fphp-dup-finder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/detain","download_url":"https://codeload.github.com/detain/php-dup-finder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/detain%2Fphp-dup-finder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34755061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-24T02:00:07.484Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["abstract-syntax-tree","anti-unification","ast","cli","clone-detection","code-analysis","code-duplication","code-quality","copy-paste-detector","developer-tools","ngram","php","php8","refactoring","refactoring-tools","similarity","static-analysis","technical-debt","tree-edit-distance"],"created_at":"2026-06-25T00:30:25.437Z","updated_at":"2026-06-25T00:30:26.033Z","avatar_url":"https://github.com/detain.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# phpdup — AST-based PHP duplicate-logic detector\n\n\u003e A semantic clone detector and refactoring assistant for PHP codebases.\n\u003e Behaves more like an \"extract function\" advisor than a copy/paste finder.\n\n[![CI](https://github.com/detain/php-dup-finder/actions/workflows/ci.yml/badge.svg)](https://github.com/detain/php-dup-finder/actions/workflows/ci.yml)\n[![codecov](https://codecov.io/gh/detain/php-dup-finder/branch/master/graph/badge.svg)](https://app.codecov.io/gh/detain/php-dup-finder)\n[![PHP Version](https://img.shields.io/badge/php-%5E8.1-blue.svg)](https://www.php.net)\n[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)\n\n`phpdup` parses every file in a PHP codebase into an Abstract Syntax\nTree, normalizes those ASTs into a canonical form, and finds clusters\nof **parameterizable duplication** — places where the *shape* of the\ncode repeats and only literals, identifiers, method names, table\nnames, or *whole optional segments of code* vary.\n\nFor each cluster it doesn't just point at the duplicates, it tells you\n**what the abstraction would look like** — its parameter list, types,\nand a suggested function name — ready to drop into a refactor.\n\n![phpdup CLI scanning a fixture corpus](docs/media/cli-basic.gif)\n\nA run on `tests/Fixtures` returns its top 2 clusters by impact. Each\ncluster's \"Suggested abstraction\" box is the function signature\nphpdup is recommending you extract; the \"Holes\" table lists every\nparameter with its inferred type and the values observed across cluster\nmembers. Compare with classic copy/paste detectors that only highlight\nthe duplication; phpdup tells you the threshold and the role string\n*are the parameters of the abstraction* with their inferred types and\nobserved values, ready to apply.\n\n---\n\n## Table of contents\n\n- [Features](#features)\n- [Installation](#installation)\n  - [PHAR (recommended)](#installation-phar)\n  - [Composer](#via-composer)\n  - [From source](#from-source)\n- [Self-update](#self-update)\n- [Quick start](#quick-start)\n- [How it works](#how-it-works)\n  - [Pipeline](#pipeline)\n  - [Normalization modes](#normalization-modes)\n  - [Clustering](#clustering)\n  - [Two-tier TED pre-filter](#two-tier-ted-pre-filter)\n  - [Anti-unification](#anti-unification)\n  - [Multi-seed search](#multi-seed-search)\n  - [Pattern recognition](#pattern-recognition)\n  - [Architectural analyzers](#architectural-analyzers)\n  - [Cluster coherence (outlier detection)](#cluster-coherence-outlier-detection)\n  - [Refactor-safety scoring](#refactor-safety-scoring)\n  - [Ranking](#ranking)\n  - [Parallelism](#parallelism)\n  - [Incremental indexing](#incremental-indexing)\n  - [Persistent cluster cache](#persistent-cluster-cache)\n  - [Lazy AST loading](#lazy-ast-loading)\n- [Type-3 / optional-segment detection](#type-3--optional-segment-detection)\n- [Type-4 / behavioural similarity](#type-4--behavioural-similarity-experimental)\n- [ORM- / DB-aware semantic deduplication](#orm---db-aware-semantic-deduplication)\n- [TUI mode](#tui-mode)\n- [Watch mode](#watch-mode)\n- [SIGINT soft-cancel](#sigint-soft-cancel)\n- [`phpdup serve` REST API](#phpdup-serve-rest-api)\n- [Output formats](#output-formats)\n- [Configuration](#configuration)\n  - [Per-directory overrides](#per-directory-overrides)\n  - [Project profiles](#project-profiles)\n  - [Auto-tune](#auto-tune)\n  - [Normalization plugins](#normalization-plugins)\n- [CLI reference](#cli-reference)\n- [Programmatic use](#programmatic-use)\n- [Examples](#examples)\n- [Static analysis \u0026 config validation](#static-analysis--config-validation)\n- [Benchmarks](#benchmarks)\n  - [Comparative benchmark suite](#comparative-benchmark-suite)\n  - [Feature matrix](#feature-matrix)\n  - [Internal scaling benchmark](#internal-scaling-benchmark)\n- [Architecture](#architecture)\n- [Testing](#testing)\n- [Performance](#performance)\n- [Roadmap](#roadmap)\n- [FAQ](#faq)\n- [Contributing](#contributing)\n- [License](#license)\n\n---\n\n## Features\n\n- **Semantic, not textual.** Compares AST structure, not source text — so\n  whitespace, comments, and identifier renames don't fool it.\n- **All four clone types.** Type-1 (exact), type-2 (renamed variables\n  / literals), **type-3** (statements present in some members but\n  absent from others — see\n  [Type-3 / optional-segment detection](#type-3--optional-segment-detection)),\n  and an experimental **type-4** behavioural-similarity scorer that\n  catches I/O-equivalent code with structurally different\n  implementations (e.g. `foreach`-accumulator vs `array_reduce`,\n  `switch` vs `match`). See\n  [Type-4 / behavioural similarity](#type-4--behavioural-similarity-experimental).\n- **Parameter discovery.** For every cluster, identifies the literals,\n  identifiers, method names, class names, *and entire optional code\n  segments* that vary, and proposes them as parameters of a suggested\n  abstraction with inferred types and named placeholders.\n- **Three normalization modes.** From `strict` (variable rename\n  tolerant only) to `aggressive` (also collapses literal values, method\n  names, property names, and class names) — pick the precision/recall\n  trade-off you want.\n- **Three-phase clustering.**\n  - **Hash buckets** for exact canonical matches — O(N).\n  - **N-gram inverted index + Jaccard + APTED tree-edit-distance** for\n    near-duplicates — never quadratic in practice.\n  - **Containment fallback** for type-3 clones whose Jaccard would fail\n    because one block is a near-subset of another.\n- **Anti-unification with statement-array LCS.** For every cluster\n  phpdup computes the most-specific generalization of its members,\n  using LCS over per-statement structural hashes when stmt arrays\n  differ in length. Disagreements become typed parameter holes;\n  per-statement gaps become defaulted boolean parameters.\n- **Pattern recognition** (22 tags). Tags clusters that match\n  well-known refactor archetypes — structural, domain, framework:\n  - **Structural:** `sql-builder` · `crud-handler` ·\n    `validation-chain` · `strategy` · `config-driven` ·\n    `state-machine` · `optional-segments`.\n  - **Domain (I.A.2-7):** `loop-map` (foreach + accumulator append) ·\n    `loop-filter` (foreach + leading guard-continue) · `sql-query`\n    (literal SQL string in body) · `http-call` (Guzzle / cURL /\n    `wp_remote_*` shapes) · `error-handler` (try-catch with logger\n    or rethrow) · `builder-chain` (≥3 method-call chain) ·\n    `container-registration` (DI binders) · `db-op` (DB operation\n    shape — `__DB_READ__` / `__DB_WRITE__` / `__DB_UPSERT__` tokens\n    produced by `--db-aware` canonicalisation).\n  - **Framework (IX.A):** `controller-action` (Laravel/Symfony\n    controllers) · `migration` (Laravel/Doctrine migrations) ·\n    `eloquent-model` (`App\\\\Models\\\\…`) · `repository-method`\n    (`*Repository::find/get/save…`) · `event-listener` ·\n    `service-provider` (Laravel SP / Symfony Bundle) ·\n    `query-builder-chain` (Doctrine/Eloquent QB entry points).\n- **Architectural analyzers** (IX.B). Beyond the pattern tags,\n  three analyzers post-process every cluster and emit\n  `architectural_findings[]` with severity + remedial suggestion:\n  - **SOLID** — flags SRP-mixed bodies (persistence + side-effect\n    calls in one block) and DIP-violating concrete-class-string\n    holes.\n  - **Design pattern** — recognises Strategy / Factory / Builder\n    shapes from existing tags + hole types.\n  - **Anti-pattern** — Long Parameter List (\u003e5 holes), Primitive\n    Obsession (all holes scalar primitives).\n- **Cluster coherence** (VI.A.3). Per-cluster outlier detection\n  via mean pairwise n-gram Jaccard; flags members that don't\n  belong with the rest. Surfaces as `outlier_members[]` in JSON\n  and a ⚠ marker in CLI.\n- **Refactor-safety scoring** (VI.A.1). Combines hole type-safety,\n  cross-namespace span, member count, and pattern-tag deltas into\n  a `[0,1]` score. `--min-safety` filters clusters that look risky\n  to mechanically extract.\n- **Impact-ranked output.** Clusters sorted by how many lines disappear\n  if the abstraction is applied, with a separate confidence score that\n  flags risky refactors (subtree-level holes, cross-namespace spans).\n- **Twelve output formats.**\n  - SugarCraft-styled colorized **CLI** (with a `--plain` switch\n    and `--summary-only` / `--clusters` verbosity modes).\n  - Structured **JSON** (machine-readable, full cluster + hole\n    metadata including `present_in_members[]` for type-3 holes,\n    `outlier_members[]`, `architectural_findings[]`,\n    `safety` score; carries a `schema_version` field —\n    [`docs/JETBRAINS_PLUGIN.md`](docs/JETBRAINS_PLUGIN.md) is the\n    stable contract for IDE plugins).\n  - Interactive **HTML** site (sortable/filterable index, mini-map\n    of cluster impact, copy-signature buttons, syntax-highlighted\n    code, optional-segment rows tinted amber, **per-hole tweaker\n    UI** with `localStorage` persistence + JSON export).\n  - **SARIF 2.1.0** (GitHub Code Scanning / GitLab Code Quality,\n    with grouping fingerprints + `optionalSegmentCount`).\n  - **GitLab SAST v15.x** (MR security widget, severity-bucketed\n    by impact).\n  - **Unified diffs** per cluster + cumulative `--patch` file.\n  - **Checkstyle XML** (Jenkins / Sonar / Bitbucket consumers).\n  - **CSV** (`--csv=FILE`) — one row per cluster member with\n    similarity / impact / safety / signature for spreadsheets and\n    BI ingest.\n  - **Prometheus** text-format (`--prometheus=FILE`) —\n    `phpdup_clusters_total`, `phpdup_total_impact`, per-tag\n    counters; ready for pushgateway scraping.\n  - **Time-series JSONL** (`--timeseries=FILE`) — one append-only\n    line per run, commit-tagged via `GIT_COMMIT` / `GITHUB_SHA` /\n    `CI_COMMIT_SHA`; lets you track duplicate debt over time in\n    BigQuery / ClickHouse / Elastic.\n  - **Graphviz DOT** (`--graphviz=FILE`) — file→cluster bipartite\n    graph; render with `dot -Tpng …`.\n  - **PlantUML** (`--plantuml=FILE`) — class diagram with cluster\n    packages and pattern-tag stereotypes.\n  - **Refactor patches** (`--refactor-patch=DIR`) — heuristic,\n    manual-review-required `.patch` per cluster: adds a\n    `Refactored/\u003cid\u003e.php` skeleton plus per-member edit hints.\n    Bails to a manual-review header when `$this`/`self::`/yield/\n    closure capture would make mechanical replacement unsafe.\n  - **PHPUnit test skeletons** (`--refactor-tests=DIR`) — one\n    `markTestIncomplete()` test class per cluster with a data\n    provider populated from observed hole values.\n- **Optional SugarCraft TUI** (`--tui`). Four-pane FlexBox dashboard\n  driven by the cooperative pipeline — counts, sparkline, OSC 9;4\n  taskbar progress all build up live frame-by-frame as work\n  progresses. Six themes, full keyboard, ←/→ to cycle clusters in\n  detail view.\n- **Watch mode** (`--watch`). Re-runs analysis on file changes via a\n  poll-based `React\\EventLoop` timer; `Ctrl+C` exits cleanly. Combines\n  with `--tui` for a live dashboard that resets and rebuilds on every\n  change.\n- **SIGINT soft-cancel.** First `Ctrl+C` flips\n  `PipelineState::$cancelled`; cooperative stages check it between\n  yields, short-circuit to the Reporting stage, and produce a\n  partial report. Second `Ctrl+C` falls back to the default kill.\n  Exit code is the canonical 130.\n- **REST API server** (`phpdup serve`). Minimal HTTP service for\n  in-house dashboards, CI integrations, and the playground\n  front-end. Routes: `GET /healthz`, `POST /analyze` (sync),\n  `POST /jobs` + `GET /jobs/{id}` (async-shaped). Hand-rolled HTTP/1.1\n  parser — no ReactPHP dep. SSRF guard, Content-Length cap,\n  JSON_THROW_ON_ERROR, default bind to `127.0.0.1`. See\n  [`docs/SERVER.md`](docs/SERVER.md).\n- **PHAR distribution.** `phpdup.phar` is published with every\n  release plus a sha256 sidecar; download once, run anywhere with\n  PHP 8.1+ and `ext-phar`. `phpdup self-update` (aliases\n  `update` / `upgrade`) replaces the running binary in place after\n  hash verification. Build locally with\n  `php -d phar.readonly=0 build-phar.php`. See\n  [`docs/PHAR.md`](docs/PHAR.md).\n- **Block-kind filter** (`--kinds=method,closure`). Drops non-matching\n  block kinds at extraction time so clustering only sees what you\n  asked for.\n- **PHP 8.x syntax-surface canonicalisation.** `match` arms collapsed\n  to switch-equivalent token shape, named arguments reordered to\n  lexicographic order, attributes stripped in `aggressive` mode —\n  so syntactic-only differences across PHP 8.0 → 8.4 don't fool\n  clustering.\n- **Auto-tune** (`--auto-tune`). Probes the corpus before analysis\n  and picks size-appropriate defaults: tiny (\u003c200 files) gets\n  relaxed thresholds, medium (\u003c20k) tightens `max-df`, large\n  (≥20k) forces `--exact-only` to keep memory in check. Explicit\n  CLI flags always override the picks.\n- **Project profiles.** `profiles/{laravel,symfony,drupal,\n  wordpress,myadmin,generic}.json` ship preset config.\n  `--profile=NAME` selects one explicitly; auto-detect kicks in\n  via project-marker files (`artisan`, `bin/console`,\n  `wp-config.php`, `include/Orm`, etc.) when no explicit config\n  is given.\n- **Per-directory config overrides.** `.phpdup.json` files\n  discovered at any depth in the scan tree, validated against the\n  schema, layered ancestor-first so deeper directories override\n  parents.  Symbols (`db_symbols.methods` and `db_symbols.functions`)\n  are additive — per-directory symbols **append** to root-config\n  symbols rather than replacing them, allowing incremental extension\n  without full duplication.\n- **User-defined normalization plugins.** Implement\n  `Phpdup\\Normalization\\NormalizationPlugin`, register the FQCN in\n  `phpdup.json -\u003e normalization.plugins[]`, and your plugin runs\n  after the built-in passes for project-specific canonicalisation\n  (e.g. unify SDK alias methods, project-specific identifier\n  normalisation).\n- **Schema-validated config.** `phpdup.json` is checked against\n  [`docs/config-schema.json`](docs/config-schema.json) at load time;\n  `--validate-config` exits with the field path on the first violation\n  before any analysis runs.\n- **Shell completion** for bash, fish, and zsh via\n  `phpdup completion \u003cshell\u003e`. Output is the standard Symfony Console\n  completion script with commented-out installation instructions\n  prepended, so you can paste-and-follow inline.\n- **Composable pipeline** with cooperative iteration. Five stages\n  (`Scanning`, `Preprocessing`, `Clustering`, `Refactoring`,\n  `Reporting`) all implement a tiny `StageInterface` and share a\n  `PipelineState`; cooperative stages additionally yield\n  mid-execution so the TUI can repaint while parallel work is in\n  flight. A `ProgressListener` interface lets observers (the TUI,\n  watchers) hook in without touching stages.\n- **Streaming worker pool.** `WorkerPool::runStreaming()` returns a\n  `\\Generator` that yields each child-process result as soon as it\n  arrives — multiplexing children's per-process socketpairs via\n  `stream_select` — so `PreprocessStage` can drive the dashboard live\n  instead of blocking on the slowest worker. The classic\n  collect-and-return `run()` is now a thin synchronous drain over the\n  same code path.\n- **Parallelized preprocessing, pair scoring, and refactor.**\n  `pcntl_fork` worker pool batches files for parse + extract +\n  normalize + fingerprint (`PreprocessWorker`); candidate pairs\n  for Jaccard + tree-edit scoring (`PairScoreWorker`); and\n  per-cluster anti-unification + tagging (`RefactorWorker`).\n  Auto CPU detection, serial fallback when pcntl is unavailable.\n- **APTED-style tree edit distance with weighted edits.**\n  Zhang-Shasha forest-distance DP with heavy-path child ordering\n  and bounded early termination, plus a\n  `--ted-weights={default,semantic}` cost model that gives method\n  calls cost 2.0, control flow 1.5, literals 0.5 — better proxy\n  for behavioural similarity than unit costs.\n- **Two-tier TED pre-filter.** Before the O(n²) DP runs, a\n  size-delta check + 64-bit (node-type, depth) shapelet sketch\n  reject obviously-different pairs in ~10 ALU ops. See\n  [Two-tier TED pre-filter](#two-tier-ted-pre-filter).\n- **Three candidate-pair indexes.** `NgramInvertedIndex` (default,\n  small / medium corpora), `BloomCandidateIndex` (drop-in\n  replacement that holds a fixed-size 2 KiB filter per block —\n  trades posting-list memory for O(n²) bit-overlap scoring on\n  huge corpora), and `LshIndex` (MinHash signatures + 32-band /\n  4-row LSH — near-constant lookup time per block).\n- **External-sort streaming-clustering primitive.** Disk-backed\n  K-way merge sort over (key, payload) tuples — foundation for\n  clustering corpora that don't fit in RAM.\n- **Incremental indexing.** Per-file block snapshots keyed by content\n  hash + parser version + config key. Editing one file leaves the\n  other 999 snapshots intact.\n- **Persistent cluster cache** (II.B.4). Snapshots the final\n  cluster list to `\u003ccacheDir\u003e/clusters.idx` after every successful\n  run. Re-running on an unchanged corpus skips Cluster + Refactor\n  stages entirely. Wholesale invalidation on any block change.\n- **Lazy AST loading.** Original ASTs are dropped after fingerprinting\n  and reloaded on demand only for blocks that end up in clusters. RSS\n  scales sub-linearly with corpus size.\n- **AST + token caches.** SHA-1 keyed disk caches (`AstCache` for\n  full Stmt[] and `TokenCache` for raw token streams), both\n  versioned to the parser release so warm-cache runs skip parsing\n  entirely.\n- **Comparative benchmark suite.** `bench/run-all.sh` runs phpdup\n  alongside phpcpd / pmd-cpd / jscpd / simian on a curated mix of\n  real OSS corpora (Symfony Console, Laravel HTTP, PHPUnit,\n  WordPress core) plus a synthetic-fuzz corpus with known ground\n  truth — emits wall-time / RSS / cluster-count comparison plus\n  precision/recall/F1 on the synthetic set. See\n  [Benchmarks](#benchmarks) and `bench/feature-matrix.md`.\n- **Memory ceiling.** `--max-memory=MB` warns and suggests\n  `--exact-only` if peak RSS exceeds the threshold mid-pipeline.\n- **`--stage` halt point.** `--stage=clustering` runs the pipeline only\n  up to (and including) clustering and stops — useful for debugging\n  incremental cache hits or profiling individual stages.\n- **Production-ready PHP.** Strict types throughout, PSR-4\n  autoloaded, PHPStan level 6 clean, Psalm errorLevel 6 clean\n  (no baseline), **396+** PHPUnit tests across four suites\n  (Unit, Integration, Golden snapshots, Fuzz detection-rate),\n  requires PHP 8.1+.\n\n---\n\n## Installation\n\n\u003ca id=\"installation-phar\"\u003e\u003c/a\u003e\n### PHAR (recommended)\n\nThe fastest way to try phpdup. Download the self-contained `phpdup.phar`\nfrom the latest GitHub release, verify the SHA-256, and drop it on\nyour PATH:\n\n```bash\ncurl -sSLO https://github.com/detain/php-dup-finder/releases/latest/download/phpdup.phar\ncurl -sSLO https://github.com/detain/php-dup-finder/releases/latest/download/phpdup.phar.sha256\nsha256sum --check phpdup.phar.sha256\n\nchmod +x phpdup.phar\nsudo mv phpdup.phar /usr/local/bin/phpdup\n```\n\nRequirements: PHP 8.1+ with `ext-phar`. The phar is ~7 MB.\n\nOnce installed, keep it current with the built-in self-updater:\n\n```bash\nphpdup self-update         # download \u0026 swap the binary in place\nphpdup self-update --dry-run   # check what's available, change nothing\n```\n\n`update` and `upgrade` are aliases. See [Self-update](#self-update)\nfor the full flow + offline notes.\n\nBuild the phar yourself (e.g. for a private fork):\n\n```bash\ncomposer install --no-dev --optimize-autoloader\nphp -d phar.readonly=0 build-phar.php\n```\n\nSee [`docs/PHAR.md`](docs/PHAR.md) for the full distribution flow,\nrelease process, and troubleshooting (`phar.readonly`, ext-phar on\nshared hosting, etc.).\n\n### Via Composer\n\n```bash\ncomposer require --dev detain/php-dup-finder\nvendor/bin/phpdup analyze src\n```\n\nThe package isn't on Packagist yet — declare the GitHub repo manually:\n\n```json\n{\n    \"require-dev\": {\n        \"detain/php-dup-finder\": \"dev-master\"\n    },\n    \"repositories\": [\n        { \"type\": \"vcs\", \"url\": \"https://github.com/detain/php-dup-finder\" }\n    ],\n    \"minimum-stability\": \"dev\",\n    \"prefer-stable\": true\n}\n```\n\n### From source\n\n```bash\ngit clone https://github.com/detain/php-dup-finder.git\ncd php-dup-finder\ncomposer install\nbin/phpdup analyze /path/to/your/code\n```\n\nRequirements:\n\n- PHP 8.1 or newer\n- ext-hash (for `xxh128`)\n- ext-pcntl + ext-posix (optional — without them phpdup runs serially\n  with no other change)\n- ext-phar (for the phar distribution; not needed for source/composer)\n- ext-curl (optional; preferred over streams for `MlClient` and the\n  self-updater's HTTP fetches)\n- Composer (for source / composer install only)\n\n---\n\n## Self-update\n\nWhen phpdup is installed as a phar, it can replace itself in place\nwith the latest GitHub release:\n\n```bash\nphpdup self-update           # download, verify sha256, swap\nphpdup self-update --dry-run # report the latest tag, change nothing\nphpdup update                # alias\nphpdup upgrade               # alias\n```\n\nThe flow:\n\n1. Resolves `https://api.github.com/repos/detain/php-dup-finder/releases/latest`.\n2. Downloads `phpdup.phar` and `phpdup.phar.sha256` from the release\n   assets to a temp dir.\n3. Verifies the SHA-256 — refuses to swap on mismatch.\n4. Renames the running binary to `phpdup.phar.old` (rollback safety),\n   moves the new phar into place, and marks it executable.\n5. Prints the new version.\n\nRequires write access to wherever the phar lives (i.e. don't put it\nin `/usr/local/bin` if you wouldn't `sudo` to update; install per-\nuser under `~/.local/bin` instead). Falls back gracefully when\nGitHub returns \"no releases yet\" (`{\"message\":\"Not Found\"}`) so a\nfresh fork can still call `--dry-run` without erroring out.\n\nComposer / source installs don't use `self-update` — pull and\nre-install the usual way:\n\n```bash\ncomposer update detain/php-dup-finder\n# or, for a source clone:\ngit pull \u0026\u0026 composer install\n```\n\n---\n\n## Quick start\n\nScan a directory and print the top duplicates (auto-parallelized):\n\n```bash\nbin/phpdup analyze src\n```\n\nMultiple directories with both reports:\n\n```bash\nbin/phpdup analyze src lib \\\n    --json    duplicates.json \\\n    --html    duplicates-report \\\n    --min-impact 30\n```\n\nUse a config file for repeatable runs:\n\n```bash\nbin/phpdup analyze --config phpdup.json\n```\n\nQuick exact-clones-only pass for CI (very fast, ~6 s on a 3,300-block corpus):\n\n```bash\nbin/phpdup analyze src --exact-only --min-impact 50\n```\n\nEmit every CI-relevant format in one shot:\n\n```bash\nbin/phpdup analyze src \\\n    --sarif       phpdup.sarif \\\n    --gitlab-sast phpdup.gitlab.json \\\n    --diff        ./phpdup-diffs \\\n    --checkstyle  phpdup.xml \\\n    --json        phpdup.json \\\n    --html        phpdup-report\n```\n\nFilter to one block kind and gate on impact:\n\n```bash\nbin/phpdup analyze src --kinds=method --min-impact=50 --exact-only\n```\n\nLive-reload while you refactor:\n\n```bash\nbin/phpdup analyze src --watch\n```\n\nShow the interactive dashboard while analysis runs:\n\n```bash\nbin/phpdup analyze src --tui --theme=dracula\n```\n\n---\n\n## How it works\n\n### Pipeline\n\n```mermaid\nflowchart TB\n    subgraph Scan[\"Scanning\"]\n        A[Scanner walks paths] --\u003e|absolute file paths| B[FileScanner]\n    end\n\n    subgraph Preprocess[\"Preprocessing — parallel via WorkerPool::runStreaming\"]\n        C[AstParser + AstCache] --\u003e D[BlockExtractor with --kinds filter]\n        D --\u003e E[Normalizer\u003cbr/\u003estrict / default / aggressive]\n        E --\u003e F[SubtreeHasher + NgramFingerprint]\n        F --\u003e G[(IndexStore\u003cbr/\u003eper-file snapshot)]\n    end\n\n    subgraph Cluster[\"Clustering\"]\n        H[BlockIndex] --\u003e I[NgramInvertedIndex]\n        I --\u003e|candidate pairs| J{Pair scoring}\n        J --\u003e|hash-bucket: exact match| K[Edge weight = 1.0]\n        J --\u003e|Jaccard \u003e= threshold| L[APTED tree-edit-distance]\n        J --\u003e|Jaccard \u003c threshold\u003cbr/\u003e+ optional_blocks_enabled| M[ContainmentSimilarity\u003cbr/\u003etype-3 fallback]\n        L --\u003e N[Edge weight = min\u003cbr/\u003ejaccard, ted]\n        M --\u003e O[Edge weight = containment]\n        K --\u003e P[Union-find]\n        N --\u003e P\n        O --\u003e P\n        P --\u003e Q[Clusters]\n    end\n\n    subgraph Refactor[\"Refactoring\"]\n        R[AntiUnifier seed = max-size member] --\u003e S{stmt arrays\u003cbr/\u003ediffer in length?}\n        S --\u003e|yes + type-3 enabled| T[LCS on stmt hashes\u003cbr/\u003e→ optional_block holes]\n        S --\u003e|no| U[Recurse normally\u003cbr/\u003e→ literal/identifier/name holes]\n        T --\u003e V[ParameterSynthesizer]\n        U --\u003e V\n        V --\u003e W[bool $includeFooBar = false\u003cbr/\u003eor typed required param]\n        W --\u003e X[SignatureBuilder + PatternRecognizer]\n    end\n\n    subgraph Report[\"Reporting\"]\n        Y[Ranker by impact] --\u003e Z[CLI / JSON / HTML / SARIF\u003cbr/\u003eGitLab SAST / Diff / Checkstyle]\n    end\n\n    B --\u003e C\n    G --\u003e H\n    Q --\u003e R\n    X --\u003e Y\n\n    %% Observer overlay\n    PL[ProgressListener\u003cbr/\u003ee.g. PhpdupModel TUI] -.observes.-\u003e Scan\n    PL -.observes.-\u003e Preprocess\n    PL -.observes.-\u003e Cluster\n    PL -.observes.-\u003e Refactor\n```\n\nThe whole pipeline is also driven cooperatively as a `\\Generator`\n(`Pipeline::iter()`) — each yield point is a chance for the TUI runtime\nto repaint or for a watcher to inject a `RestartPipelineMsg`.\n\n| Stage          | Output                                     |\n|----------------|--------------------------------------------|\n| Scanning       | absolute file paths (glob include/exclude) |\n| Preprocessing  | annotated blocks (canonical AST + n-gram bag + structural hash) per file |\n| Clustering     | clusters with similarity scores + edge weights |\n| Refactoring    | generalized AST, holes, signature, pattern tags |\n| Reporting      | CLI / JSON / HTML / SARIF / GitLab SAST / diff / Checkstyle output |\n\n### Normalization modes\n\n| Mode          | Variable rename | Literal collapse | Name collapse |\n|---------------|:---------------:|:----------------:|:-------------:|\n| `strict`      | yes             | no               | no            |\n| `default`     | yes             | yes              | no            |\n| `aggressive`  | yes             | yes              | yes           |\n\nIn `aggressive` mode (default), two functions with different table\nnames, different method names, and different literal values can still\ncluster together.\n\n### Clustering\n\nThree phases:\n\n1. **Exact canonical clones.** All blocks sharing the same Merkle hash\n   over the canonical AST land in the same bucket. O(N) work.\n2. **Near-duplicates.** For each block, candidates are pulled from a\n   rare-n-gram inverted index (ignoring n-grams that occur in more than\n   `max_df` × N blocks). Each candidate is scored by Jaccard similarity\n   on the canonical n-gram multiset; survivors are refined with APTED-style\n   bounded tree-edit-distance.\n3. **Type-3 fallback.** When Jaccard fails but `ContainmentSimilarity`\n   shows the smaller block is mostly contained in the larger\n   (`containment ≥ 0.85` AND `size_ratio ≥ 0.6`), the pair is accepted\n   anyway. See [Type-3 / optional-segment detection](#type-3--optional-segment-detection).\n\nSurviving edges feed a union-find that merges them into clusters.\n\n### Anti-unification\n\nFor every cluster phpdup computes the most-specific generalization of\nits members. The classic recursion:\n\n```\nau(t1, t2) =\n    if root(t1) == root(t2) and arity matches:\n        Node(root(t1), [au(c1_i, c2_i) for i in children])\n    else:\n        Hole(observed=[t1, t2])\n```\n\nExtensions in phpdup:\n\n- **Seed = max-size member.** Cluster member with the most AST nodes\n  is used as the template, so the \"maximal\" version drives the\n  abstraction and shorter members get optional segments highlighted\n  (instead of the alignment failing because the seed happened to be\n  short).\n- **LCS for stmt arrays.** When two stmts/cases/catches arrays differ\n  in length, phpdup runs LCS over per-statement structural hashes;\n  matched positions recurse, unmatched template positions become\n  `optional_block` holes.\n\nThe resulting template has Hole markers at every position where\nmembers disagreed. Each hole tracks its observed values across all\nmembers (in cluster order), so reports show\n`threshold ∈ {10, 20, 30}` and\n`role ∈ {'admin', 'moderator', 'editor'}`.\n\n### Pattern recognition\n\nAfter anti-unification, each cluster is checked against a small\ncatalog of refactor archetypes (sql-builder, crud-handler,\nvalidation-chain, strategy, config-driven, state-machine,\noptional-segments). Tags are advisory; they don't change clustering,\njust label the cluster in the report.\n\n### Ranking\n\nEach cluster gets two scores:\n\n- **Impact** ≈ `(members - 1) × avgBlockSize - holesPenalty`. How many\n  lines of code disappear if the abstraction is applied.\n- **Confidence** in `[0,1]`. Cluster similarity, penalized for\n  subtree-level holes (large variable subtrees) and cross-namespace\n  spans, bumped for same-class cohesion.\n\nClusters below `min_cluster_impact` are dropped. Survivors are sorted by\ndescending impact, breaking ties by member count and similarity.\n\n### Parallelism\n\n`Phpdup\\Parallel\\WorkerPool` partitions a list of items into N batches,\nforks one child per batch via `pcntl_fork`, runs the closure in the\nchild, and the parent reaps results. There are two collection modes:\n\n- **`run()`** — collect-and-return. Each child writes its full result\n  to a temp file when finished; the parent reads them all once every\n  child has exited.\n- **`runStreaming()`** — yield-as-results-arrive. Children write\n  length-prefixed serialized records (4-byte big-endian uint32 +\n  payload) to per-child `stream_socket_pair`s; the parent multiplexes\n  via `stream_select` and the returned `\\Generator` yields each\n  record live. `PreprocessStage` consumes this so the cooperative\n  pipeline gets mid-stage progress events instead of blocking until\n  every fork exits.\n\nTwo phases use the pool:\n\n- **`PreprocessWorker`** — each child does parse + extract + normalize\n  + hash + n-gram fingerprint for its file batch.\n- **`PairScoreWorker`** — once candidate pairs are generated from the\n  inverted index, the master batches them across workers; each child\n  runs Jaccard + bounded TED + (when enabled) the type-3 containment\n  fallback on its batch and emits surviving edges.\n\nCPU count is auto-detected (`nproc` / `/proc/cpuinfo`) or overridable\nvia `--workers N` / `PHPDUP_WORKERS=N`. When `pcntl_*` is unavailable\n(Windows, sandboxed PHP), the pool detects this at runtime and falls\nback to a serial code path with the same closure interface — callers\ndon't branch.\n\n### Incremental indexing\n\n`Phpdup\\Persistence\\IndexStore` snapshots each file's extracted +\nnormalized + fingerprinted blocks under\n`\u003ccache_dir\u003e/\u003csha1(path)\u003e.idx`. Each snapshot stores:\n\n- `file_hash` — `sha1_file()` of the source.\n- `parser_version` — bumped together with the AST cache key.\n- `config_key` — sha1 of the relevant config fields (block size,\n  normalization mode, n-gram size). Changing any of these invalidates\n  the snapshot automatically.\n- `blocks` — serialized `Block[]` ready to pour into the index.\n\nOn re-runs the master splits files into \"reuse\" (snapshot hit) and\n\"process\" (snapshot miss) buckets and only the latter goes through the\nworker pool. Editing one file leaves the other snapshots intact.\n\nDisable with `--no-incremental` for benchmarking or when paranoid\nabout cache poisoning.\n\n### Lazy AST loading\n\nAfter fingerprinting we drop `Block::$ast` (the original PhpParser\nsubtree) and reload it on demand inside `AntiUnifier` via\n`BlockAstLoader`. The loader walks the file's parse-cached statement\nlist looking for the unique\n(kind, start_line, end_line, declared_name) tuple; matches are\npopulated back into the Block.\n\nThe AST cache is consulted first so on warm runs no parsing happens at\nall. Disable with `--no-lazy-ast` if you have RAM to spare and want\nmaximum speed (the reload overhead is roughly equal to the RSS savings\non small corpora — see BENCHMARKS.md).\n\n---\n\n## Type-3 / optional-segment detection\n\nA \"type-3\" clone is one where the structures match, but some members\nhave extra (or missing) statements relative to others. phpdup detects\nthese in two coordinated stages.\n\n![phpdup detecting an optional-segment cluster](docs/media/optional-blocks.gif)\n\nThe fixture run shows phpdup pulling in two blocks whose statements\nshare a common prefix but where the longer block has two extra calls\n(`some_other_logic($here)` and `and_more($f)`). The \"Suggested\nabstraction\" box ends up with two **defaulted boolean** parameters\nnamed after the absent code:\n\n```php\nfunction extractedFunction(\n    bool $includeSomeOtherLogic = false,\n    bool $includeAndMore = false,\n): mixed\n```\n\n…and the cluster is tagged `optional-segments`. Each row in the Holes\ntable is marked `optional_block` with the literal `\u003cabsent\u003e` sentinel\nshowing which members lacked the segment.\n\n### Clustering: containment fallback\n\nWhen the n-gram Jaccard between two candidate blocks falls below\n`similarity_threshold`, the clusterer tries\n`ContainmentSimilarity = |A ∩ B|min / min(sum(A), sum(B))`, which\nreturns 1.0 whenever the smaller bag is fully contained in the larger\nregardless of size disparity. The pair is accepted with the\ncontainment score as the edge weight only when:\n\n```\ncontainment ≥ optional_blocks_containment   (default 0.85)  AND\nsize_ratio  ≥ optional_blocks_min_overlap    (default 0.6)\n```\n\nThe size-ratio guard prevents a 1-line block from clustering with a\n100-line block on a single shared n-gram.\n\n### Anti-unification: LCS over statement arrays\n\nThe seed (template) is the cluster member with the most AST nodes.\nWhen `walk()` reaches a `stmts` / `cases` / `catches` array whose\nlength differs from the seed's, phpdup runs LCS over each statement's\nstructural hash:\n\n- Matched template positions recurse via the normal walk — variables,\n  literals, names inside the matched statement still produce regular\n  holes.\n- Unmatched template positions become **`optional_block`** holes —\n  one per missing statement, capped at\n  `optional_blocks_max_per_cluster` (default 3) so over-flexible\n  clusters don't explode into seven-boolean signatures.\n\nEach `optional_block` hole becomes a default-`false` `bool` parameter.\nThe name is derived from the first non-stop-word identifier in the\nsegment, e.g. a missing `some_other_logic($here);` becomes\n`bool $includeSomeOtherLogic = false`. `SignatureBuilder` groups\nrequired parameters first then defaulted booleans so the resulting\nsignature is syntactically valid PHP.\n\n### Tunables\n\n```json\n{\n  \"optional_blocks\": {\n    \"enabled\": true,\n    \"containment\": 0.85,\n    \"min_overlap\": 0.6,\n    \"max_per_cluster\": 3,\n    \"min_segment_length\": 1\n  }\n}\n```\n\nCLI overrides:\n\n```bash\nbin/phpdup analyze src \\\n    --optional-blocks=on \\\n    --optional-blocks-containment=0.85\n```\n\nTo disable type-3 detection completely: `optional_blocks.enabled =\nfalse` (or `--optional-blocks=off`); the clusterer reverts to\nJaccard-only and AntiUnifier falls back to whole-array subtree holes\nwhen stmt arrays differ in length.\n\n### How it surfaces in reporters\n\n- **CLI** — Holes table shows `kind = optional_block`, observed values\n  include the literal `\u003cabsent\u003e` marker for members where the segment\n  was missing.\n- **JSON** — every optional_block hole carries\n  `present_in_members: [int, ...]` listing the cluster-member indices\n  that *did* include the segment.\n- **SARIF** — each result's `properties` adds `optionalSegmentCount`\n  and `hasOptionalSegments` so PR-annotation tooling can flag type-3\n  clusters distinctly.\n- **HTML** — optional rows are tinted amber, get a \"type-3\" badge, and\n  italicize the `\u003cabsent\u003e` sentinel.\n\n---\n\n## Type-4 / behavioural similarity (experimental)\n\nType-1/2/3 detection compares **shape** — what tokens the AST emits.\nType-4 detection compares **behaviour** — what data flows through\nthe block, what calls it makes, what it returns. Two functions that\ncompute the same value with structurally different implementations\n(`foreach` accumulator vs `array_reduce`, recursion vs iteration,\n`switch` vs `match`) cluster under type-4 even when AST similarity\nrejects them.\n\nphpdup's type-4 scaffold:\n\n- `Phpdup\\Semantic\\DataflowSummarizer` — walks the block once and\n  emits `(vars, calls, returns, sideEffects)`.\n- `Phpdup\\Semantic\\DbOperationTagger` — separately walks the block\n  and emits a `tag → count` multiset of recognised database\n  operations (`db.read`, `db.write`, `db.delete`, `db.execute`,\n  `db.query`). Library- and extension-agnostic by design — see\n  [Behavioural-tag scoring](#behavioural-tag-scoring) below.\n- `Phpdup\\Similarity\\BehaviouralSimilarity` — weighted Jaccard\n  over those summaries (var-set 1×, call-multiset 2×,\n  return-shape 2×, side-effect-flag 1×, DB-op-tag multiset 2×) →\n  `[0,1]` score.\n- `Phpdup\\Semantic\\CallGraph` and `Phpdup\\Semantic\\ControlFlowGraph`\n  — coarse per-block summaries used by future type-4 boost paths.\n\nType-4 has higher false-positive risk than type-1/2/3 by design.\nToday it's a foundational scaffold; the live wiring into the\nclusterer (as a fourth-tier fallback after Jaccard + APTED +\ncontainment all reject) is gated behind a future `--type4` flag.\nSee [`docs/algorithms/anti-unification.md`](docs/algorithms/anti-unification.md)\nfor the algorithmic reference and\n[`docs/plans/orm-db-semantic-dedup.md`](docs/plans/orm-db-semantic-dedup.md)\nfor how this connects to the broader plan for ORM/DB\nsemantic-equivalence detection.\n\n---\n\n## ORM- / DB-aware semantic deduplication\n\nStock AST clustering misses duplicates that the developer thinks of as\n\"the same operation\" because the surface call shape is different —\ntypically when the same database write/read is expressed via an ORM\nin one place and via raw SQL in another. The four examples below all\nperform the **same write**, but none cluster without `--db-aware`:\n\n```php\n// Eloquent\n$user = User::find($id);\n$user-\u003ename = 'Bob';\n$user-\u003esave();\n\n// Doctrine\n$user = $em-\u003efind(User::class, $id);\n$user-\u003esetName('Bob');\n$em-\u003eflush();\n\n// Raw PDO\n$pdo-\u003equery(\"UPDATE users SET name = 'Bob' WHERE id = {$id}\");\n\n// Query builder\n$db-\u003etable('users')-\u003ewhere('id', $id)-\u003eupdate(['name' =\u003e 'Bob']);\n```\n\nPass `--db-aware` and phpdup runs `Phpdup\\Normalization\\DbOpCanonicalizer`\nas a pre-pass during normalisation. It rewrites recognised database\ncalls into canonical synthetic FuncCalls — `__DB_FIND__(\"user\")`,\n`__DB_QUERY__(\"users\", \"SELECT\")`, `__DB_WRITE__(\"users\")` etc. —\nso equivalent variants produce identical token streams and cluster\ntogether in tier-1 / tier-2.\n\n### What is recognised\n\nThe stock symbol table in `Phpdup\\Normalization\\DbOpRegistry` covers:\n\n- **Eloquent / Laravel** — `Model::find`, `Model::all`,\n  `Model::create`, `DB::table`, `DB::select`, `Model::where(...)\n  -\u003efirst|get|update|delete()`, etc.\n- **Doctrine ORM** — `EntityManager::find`, `EntityManager::flush`,\n  `EntityManager::persist`, `Repository::findOneBy`, `Repository::findAll`.\n- **PDO** — `PDO::query`, `PDO::prepare`, `PDOStatement::execute`,\n  `PDOStatement::fetch*`.\n- **mysqli (object + procedural)** — `mysqli::query`,\n  `mysqli_query`, `mysqli_stmt_execute`, `mysqli_fetch_*`.\n- **PostgreSQL** — `pg_query`, `pg_query_params`, `pg_fetch_*`,\n  `pg_insert`, `pg_update`, `pg_delete`.\n- **Firebird / InterBase** — `ibase_query`, `ibase_prepare`,\n  `ibase_execute`, `ibase_fetch_row`, `ibase_fetch_assoc`,\n  `ibase_fetch_object`, `ibase_commit`, `ibase_rollback`.\n- **MSSQL / DB-Library** — `mssql_query`, `mssql_fetch_row`,\n  `mssql_fetch_array`, `mssql_fetch_assoc`, `mssql_fetch_object`,\n  `mssql_num_rows`.\n- **IBM DB2** — `db2_prepare`, `db2_execute`, `db2_query`,\n  `db2_fetch_row`, `db2_fetch_assoc`, `db2_fetch_array`,\n  `db2_fetch_object`, `db2_num_rows`.\n- **Async MySQL clients** — `amphp/mysql`\n  (`amysql_query`, `amysql_fetch_assoc`, `amysql_fetch_row`,\n  `amysql_free_result`), friends-of-reactphp/mysql\n  (`react_mysql_query`, `react_mysql_fetch_assoc`,\n  `react_mysql_fetch_row`), Swoole Coroutine MySQL\n  (`swoole_mysql_query`, `swoole_mysql_fetch_assoc`,\n  `swoole_mysql_fetch_row`), OpenSwoole MySQL\n  (`openswoole_mysql_query`), Workerman mysql\n  (`workerman_mysql_query`).\n- **Async PostgreSQL clients** — `amphp/postgres`\n  (`apg_query`, `apg_fetch_assoc`, `apg_fetch_row`,\n  `apg_free_result`), `reactphp/postgres`\n  (`react_pg_query`, `react_pg_fetch_assoc`,\n  `react_pg_fetch_row`), Swoole coroutine postgres\n  (`swoole_postgres_query`).\n- **Oracle OCI8** — `oci_parse`, `oci_execute`, `oci_fetch`,\n  `oci_fetch_assoc`, `oci_fetch_row`, `oci_fetch_object`,\n  `oci_free_statement`, `oci_commit`, `oci_rollback`.\n- **Cassandra (phpcassa)** — `phpcassa_query`,\n  `phpcassa_fetch`.\n- **LevelDB** — `leveldb_get`, `leveldb_put`, `leveldb_delete`,\n  `leveldb_open`.\n- **Memcached** — `memcached_get`, `memcached_set`,\n  `memcached_add`, `memcached_replace`, `memcached_delete`,\n  `memcached_increment`, `memcached_decrement`,\n  `memcached_flush`.\n- **Async transaction methods** — `beginTransaction`,\n  `commit`, `rollback` are classified `OP_WRITE`; auxiliary\n  methods `affectedRows` → `OP_READ`, `insertId` → `OP_WRITE`,\n  `count` → `OP_READ`.\n- **Generic CRUD verbs** — any method named `find`, `findById`,\n  `save`, `update`, `delete`, `query`, `execute` on an unknown\n  receiver — coarse but high-recall. Also: `table`, `select`,\n  `insert`, `upsert`, `lock`, `unlock` via the illuminate/database\n  query builder facade (Laravel `DB::table()-\u003e...` chains).\n- **Raw SQL strings** — passed to any of the above. The bundled\n  `Phpdup\\Normalization\\SqlTableExtractor` lifts the verb (`SELECT`\n  / `INSERT` / `UPDATE` / `DELETE` / `REPLACE` / `TRUNCATE`) and\n  the primary table out of the literal string so the verb shows\n  up as a token in the synthesised `__DB_\u003cOP\u003e__` call.\n\n### How it slots into the pipeline\n\n```\n   1. ScanningStage\n   2. PreprocessStage\n        a. AstParser       → plain AST\n        b. BlockExtractor  → (file, kind, range)\n        c. Normalizer\n              ├── DbOpCanonicalizer  ← only when --db-aware\n              └── CanonicalizingVisitor (vars, literals, names)\n        d. SubtreeHasher + NgramFingerprint\n   3. ClusterStage\n   4. RefactorStage\n   5. ReportStage\n```\n\nDbOpCanonicalizer runs **before** the standard variable / literal /\nname passes so the synthesised `__DB_\u003cOP\u003e__` token names survive the\naggressive name-canonicalisation pass (they're treated as structural\nfunction names, like `isset` or `count`).\n\n### Risk profile and false positives\n\n`--db-aware` is intentionally biased toward **high recall** — a few\nbenign false positives (e.g. an unrelated `query()` method on a\nnon-DB class folding to `__DB_QUERY__`) are cheaper than missing\nreal ORM ↔ raw-SQL clones. Two safeguards:\n\n1. **Off by default.** Tier-1 AST-only clustering remains the\n   unmodified path; nothing in the existing report stream changes\n   without `--db-aware`.\n2. **Cache invalidated on toggle.** `Phpdup\\Pipeline\\Stages\\PreprocessStage`\n   includes `dbAware` in its config-key hash, so flipping the flag\n   between runs reprocesses every file rather than reusing\n   stale-canonicalised blocks.\n\nFor per-project tuning, the `DbOpRegistry` constructor accepts\n`customMethodOps` and `customFunctionOps` maps so you can extend or\noverride the stock dispatch table — wire that into a normalisation\nplugin (see [Normalisation plugins](#normalisation-plugins)) for\nproject-specific rewrites.\n\n### Trinity-collapse — `--trinity-collapse`\n\n`--db-aware` folds individual DB calls. The natural follow-up is\n**trinity-collapse**: detecting the canonical CRUD shape\n\n```php\n$user = User::find($id);          // (1) read\n$user-\u003ename = 'Bob';              // (2) mutate\n$user-\u003esave();                    // (3) save\n```\n\n…and rewriting the three statements as a single\n`__DB_UPSERT__(\"user\")` synthetic call so the ORM idiom clusters\nwith the raw equivalent\n\n```php\n$pdo-\u003equery(\"UPDATE users SET name = 'Bob' WHERE id = $id\");\n```\n\n(which `--db-aware` separately folds to a `__DB_WRITE__` token).\n\n`Phpdup\\Normalization\\TrinityCollapser` walks every statement-array\nin the AST (function/method/closure bodies, if/else branches, loop\nbodies) and looks for triplets of the shape **read → mutate(s) → save**\nwhere:\n\n- The read is any `DbOpRegistry::OP_READ` call assigned to a\n  variable (`$x = User::find($id)`, `$x = $em-\u003efind(User::class, $id)`,\n  `$x = $repo-\u003efindOneBy([...])`, …).\n- The mutate(s) are property assignments (`$x-\u003ename = 'Bob'`,\n  `$x-\u003ename .= 'X'`) or setter calls (`$x-\u003esetName('Bob')`,\n  `$x-\u003ewithFoo(...)`, `$x-\u003eaddThing(...)`) on the bound variable.\n  At least one mutation is required — read+save with no change in\n  between is left untouched.\n- The save terminates the chain: receiver-bound writes\n  (`$x-\u003esave()`, `$x-\u003eupdate()`), Doctrine flush (`$em-\u003eflush()`),\n  or `$em-\u003epersist($x)` (with `$x` matching the bound variable as\n  the first argument).\n\n**Mutation detection improvements.** The collapser recognises\nadditional mutation patterns beyond simple property assignment:\n\n- **ArrayAccess / array assignment** — `$x['key'] = value` via\n  `Node\\Expr\\ArrayDimFetch` on the bound variable.\n- **Direct array append** — `$x[] = value` appends on the bound\n  variable.\n- **Extended ORM mutator prefixes** — setter-method prefixes\n  `force`, `update`, `change`, `modify` are recognised in addition\n  to the base `set` / `with` / `add` / `remove` / `append` /\n  `replace` set, so idioms like `$x-\u003eforceUpdate(...)` or\n  `$x-\u003echangeName(...)` are correctly classified as mutations.\n\nAny unrelated statement between the read and save abandons the\ntrinity — the dataflow walker is intentionally conservative; false\nmisses are preferable to false matches.\n\n`--trinity-collapse` composes with `--db-aware` (the typical\ncombination), and runs as a *pre-pass* before `DbOpCanonicalizer`\nso the collapser can pattern-match on the original read/save call\nshapes.\n\n### Behavioural-tag scoring\n\nThe Type-4 behavioural scorer (`Phpdup\\Similarity\\BehaviouralSimilarity`)\ngains a fifth band: a **DB op-tag multiset Jaccard** weighted equally\nwith the existing call-name and return-shape bands.\n\n`Phpdup\\Semantic\\DbOperationTagger` walks each block once and produces\na coarse `tag → count` summary:\n\n```\n['db.read' =\u003e 2, 'db.write' =\u003e 1, 'db.execute' =\u003e 1]\n```\n\nTwo functions with the same DB-shape — same number of reads, writes,\ndeletes, executes, and queries — score similarly under the tag band\n*regardless of which library or extension* delivers each operation.\nThat collapses the library/extension axis (Eloquent vs Doctrine vs\nPDO vs mysqli vs `pg_*`) without needing per-pair surface analysis.\n\nThe tag band is a *no-op* for blocks that do not touch the database\n(empty bag vs empty bag is `1.0` by convention) so non-DB code is\nunaffected. The behavioural scorer's total weight is now `1 + 2 + 2 +\n1 + 2 = 8` (variables + calls + returns + side-effects + DB tags),\nnormalised back to `[0, 1]`.\n\nThe tagger reuses the same `DbOpRegistry` as `DbOpCanonicalizer`, so\nthe recognised call set is consistent across the canonicalisation\nand scoring layers, and synthetic `__DB_\u003cOP\u003e__` calls produced by\n`--db-aware` / `--trinity-collapse` are also tagged correctly.\n\n### Symbol equivalence classes\n\nThe stock `DbOpRegistry` covers the obvious surface area but every\ncodebase has wrappers, helpers, and homegrown façades that do DB\nwork without using a recognised name. **Option 4** of the plan\nexposes a user-extensible symbol equivalence registry: declare in\n`phpdup.json` (or in a profile JSON) that `app_db_get`,\n`MyRepo::lookup`, and `LegacyDb::raw` all mean the same thing as\nthe stock entries — and they fold to the same canonical\n`__DB_\u003cOP\u003e__` tokens.\n\n```json\n{\n    \"db_aware\": true,\n    \"db_symbols\": {\n        \"methods\": {\n            \"lookup\":     \"db.read\",\n            \"persistMe\":  \"db.write\",\n            \"wipe\":       \"db.delete\"\n        },\n        \"functions\": {\n            \"app_db_get\":   \"db.read\",\n            \"app_db_query\": \"db.query\"\n        }\n    }\n}\n```\n\nAllowed canonical ops: `db.read`, `db.write`, `db.delete`,\n`db.execute`, `db.query`. Custom entries override stock ones with\nthe same name; everything else is additive.\n\n**Bundled symbol packs.** Twenty-eight framework-flavoured packs ship\nout of the box and can be loaded via `--profile`:\n\n| Profile name                | What it adds                                                                                         |\n|-----------------------------|------------------------------------------------------------------------------------------------------|\n| `db-aware-laravel`          | Eloquent / Laravel methods (`firstWhere`, `pluck`, `chunk`, `increment`, raw\\* helpers).             |\n| `db-aware-doctrine`         | Doctrine ORM / DBAL (`createQuery`, `executeStatement`, `fetchAssociative*`, transaction helpers).  |\n| `db-aware-cake`             | CakePHP ORM (`patchEntity`, `saveOrFail`, `findThreaded`, `loadInto`).                              |\n| `db-aware-codeigniter`      | CodeIgniter 4 DB Query Builder (`get`, `insert`, `update`, `delete`, `countAll`, `escape`).         |\n| `db-aware-thinkorm`         | ThinkPHP 6.x / think-orm (`find`, `select`, `insert`, `update`, `delete`, `count`, aggregation).   |\n| `db-aware-yii`              | Yii 2 ActiveRecord / DB (`find`, `findOne`, `findAll`, `save`, `insert`, `update`, `delete`, `createCommand`). |\n| `db-aware-medoo`            | Medoo (`select`, `insert`, `update`, `delete`, `create`, `drop`, `query`, `exec`).                 |\n| `db-aware-propel`           | Propel ORM (`doSelect`, `doInsert`, `doUpdate`, `doDelete`, `find`, `save`).                        |\n| `db-aware-redbean`          | RedBeanPHP (`find`, `dispense`, `store`, `trash`, `save`, `load`, `wipe`, `related`).              |\n| `db-aware-cycle`            | Cycle ORM (`find`, `findAll`, `persist`, `delete`, `select`, `where`, `aggregate`).                |\n| `db-aware-phpactiverecord`  | PHP ActiveRecord (`find`, `all`, `first`, `last`, `create`, `save`, `update`, `delete`, `destroy`).|\n| `db-aware-myadmin`          | MyAdmin custom db_abstraction (`MyDb\\Mysqli\\Db`, `MyDb\\Pdo\\Db` — `query`, `qr`, `next_record`, `prepare`, `execute`). |\n| `db-aware-myadmin-orm`      | MyAdmin custom ORM (`MyAdmin\\Orm\\*` extending `Base\\Orm` — `find`, `load`, `save`, `update`, `insert`, `delete`, `truncate`). |\n| `db-aware-illuminate`       | Illuminate Database standalone (`DB::table()`, `DB::select()`, `DB::insert()`, `DB::update()`, `DB::delete()`, `DB::raw()`). |\n| `db-aware-aura`             | Aura.Sql (`fetchAll`, `fetchOne`, `query`, `execute`, `quote`, `begin/commit/rollback`).                                      |\n| `db-aware-atlas`            | Atlas.PDO (`fetchAll`, `fetchAssoc`, `fetchSelect`, `begin/commit/rollback`).                                                 |\n| `db-aware-easydb`           | EasyDB (`query`, `fetchAll`, `fetchOne`, `iterator`, `safeQuery`, `build`).                                                   |\n| `db-aware-dibi`             | Dibi (`query`, `fetchAll`, `fetch`, `test`, `begin/commit/rollback`).                                                         |\n| `db-aware-pixie`            | Pixie Query Builder (`table`, `get`, `insert`, `update`, `delete`, `where`, `orderBy`).                                      |\n| `db-aware-redis`            | Redis (Predis, Credis, phpredis) — `get`, `set`, `mget`, `mset`, `hget`, `hset`, `del`, `expire`, etc.                      |\n| `db-aware-mongodb`          | MongoDB driver (`find`, `findOne`, `insertOne`, `insertMany`, `updateOne`, `deleteOne`, `aggregate`, `count`, etc.).         |\n| `db-aware-elasticsearch`    | Elasticsearch PHP client (`search`, `index`, `get`, `mget`, `bulk`, `delete`, `update`, `count`, `scroll`, `msearch`).     |\n| `db-aware-neo4j`            | Neo4j PHP client (`run`, `match`, `create`, `merge`, `set`, `delete`, `detachDelete`).                                      |\n| `db-aware-influxdb`         | InfluxDB client (`query`, `write`, `ping`, `bucket`, `organization`, `flux`).                                               |\n| `db-aware-couchdb`          | Doctrine CouchDB ODM (`find`, `findBy`, `persist`, `save`, `remove`, `delete`, `refresh`).                                  |\n| `db-aware-couchbase`        | Couchbase SDK (`get`, `upsert`, `insert`, `replace`, `remove`, `lookupIn`, `mutateIn`, `view`, `query`).                    |\n| `db-aware-idiorm`           | Idiorm/Paris (`find`, `findOne`, `where`, `orderBy`, `count`, `create`, `save`, `update`, `delete`).                        |\n| `db-aware-laminas`          | Laminas\\Db (formerly Zend\\Db) adapter (`select`, `fetchAll`, `fetchOne`, `query`, `insert`, `update`, `delete`).           |\n| `db-aware-phalcon`          | Phalcon ORM (`find`, `findFirst`, `save`, `create`, `update`, `delete`, `refresh`, `query`, `aggregate`, `sum`, `count`, `average`, `min`, `max`). |\n\nCompose them with `--db-aware` for the canonicalisation pass plus\nthe stock DB call coverage. The user's own `db_symbols` in\n`phpdup.json` win over profile-provided symbols, which in turn\nwin over the stock registry.\n\n### IR — intermediate representation lift\n\n`Phpdup\\Ir\\` is the foundational scaffold for **option 5** of the\nplan: a canonical, language-/library-agnostic representation that\nsits between the PHP AST and the similarity scorer.\n\n```\nBlock (PHP AST) → IR (canonical operation graph) → fingerprint / score\n```\n\nThe IR replaces PHP-specific syntax (the difference between `$x-\u003ey`\nand `$x['y']`, between `for` and `foreach`, between `if/else` and\n`match`) with a small set of operation-shaped nodes:\n\n| IR node          | Lifts from                                                            |\n|------------------|-----------------------------------------------------------------------|\n| `DbReadIr`       | `Model::find`, `$em-\u003efind(...)`, `$pdo-\u003equery(\"SELECT …\")`, `pg_*` reads. |\n| `DbWriteIr`      | `$x-\u003esave`, `$em-\u003eflush`, `$pdo-\u003equery(\"UPDATE …\")`, `pg_insert/update`. |\n| `DbDeleteIr`     | `Model::destroy`, `$em-\u003eremove`, `DELETE FROM …`, `TRUNCATE …`.       |\n| `DbQueryIr`      | Generic `query`/`exec` calls with no clean read/write classification. |\n| `DbExecuteIr`    | `prepare`/`execute`-style two-phase calls.                             |\n| `AssignIr`       | Local assignments — LHS shape collapsed to `var`/`prop`/`index`/…     |\n| `BranchIr`       | `if`/`else`, `match`, `switch` (unfolded into nested branches), ternaries. |\n| `LoopIr`         | `for`, `foreach`, `while`, `do-while` — keyword distinction erased.   |\n| `CallIr`         | Any unrecognised method/function/static call.                         |\n| `ReturnIr`       | `return [expr];` and bare `return;`.                                  |\n| `VarIr`          | Variable references (name collapsed to `__V`).                        |\n| `LiteralIr`      | Scalar literals (only the *type* — `str`/`int`/`float`/`bool`/`null` — survives). |\n| `BlockIr`        | Statement sequences (function/method bodies, branch arms, loop bodies). |\n\n`Phpdup\\Ir\\IrLifter` walks a PhpParser AST and produces an IR tree.\nLifting is **partial by design**: unrecognised input shapes (e.g.\n`eval`, `goto`, complex variable-variables) fall through to a\ngeneric `CallIr` carrying the node class name; if the lift fails\noutright the lifter returns `null` and callers fall back to\nAST-level scoring (per the plan's risk-mitigation note).\n\n`Phpdup\\Ir\\IrPrinter` produces a deterministic token stream and a\nhuman-readable pretty-print. `Phpdup\\Ir\\IrSimilarity` scores two\nIR trees: `1.0` for identical printed-token streams (a SHA-1-hash\nfast path is exposed via `IrSimilarity::hash()`), then a\nmultiset-Jaccard fallback for partial overlap.\n\nThe IR scorer is wired into the `Clusterer` as a fifth tier behind\n`--scorer=ir`. When AST Jaccard / TED / containment all reject a\npair, the clusterer falls back to multiset Jaccard over the\npre-computed IR token bags (`Block::$irBag`, populated by\n`PreprocessWorker` when `scorer=ir`); pairs at or above\n`--ir-threshold` (default `0.85`) form edges weighted by the IR\nsimilarity. Lift failure on either block leaves `irBag` null and\nthe IR tier silently skips that pair, preserving the plan's\nrisk-mitigation note (\"fall back to AST scoring on lift failure\").\n\n### ML-learned pair similarity (sidecar)\n\nphpdup ships a sidecar contract for **option 6** of the plan: an\nexternal ML pair-scoring service trained on a labelled corpus that\nincludes ORM ↔ raw-SQL examples. The PHP side stays light:\n\n- `Phpdup\\Ml\\PairFeatures` extracts an 11-field feature vector from\n  a `(blockA, blockB)` pair using the existing dataflow summary,\n  DB op tags, and IR token-stream — no ML libraries required.\n- `Phpdup\\Ml\\MlPairClient` is the HTTP companion to the existing\n  `MlClient` (cluster-safety scoring). It POSTs the feature vector\n  to `/score-pair` on a configured sidecar and consumes a single\n  `{similarity, confidence}` response.\n- `MlPairClient::score()` returns `null` on transport error, malformed\n  response, or unsafe URL — callers fall back to AST-level scoring\n  without code-path branching, mirroring the IR lifter's\n  fail-graceful pattern.\n\nThe wire contract is documented in [`docs/ML.md`](docs/ML.md); the\nlabelled-corpus format the sidecar expects is documented in\n[`docs/ml-corpus-format.md`](docs/ml-corpus-format.md). A\n`feature_version` field is embedded in every payload so the sidecar\ncan warn when the model was trained against a different feature\nshape (the `PairFeatures::FEATURE_VERSION` constant bumps any time\nthe schema changes).\n\nThe model itself, the training pipeline, and the labelled corpus\nall live in a sister repository — phpdup ships only the contract\nand the feature extractor.\n\nOnce a sidecar is running, point phpdup at it via `--ml-pair-url`\n(or `ml_pair_url` in `phpdup.json`):\n\n```\nbin/phpdup analyze src --ml-pair-url=https://ml.example.com/api \\\n                       --ml-pair-threshold=0.85\n```\n\n`Phpdup\\Clustering\\Clusterer` and `Phpdup\\Parallel\\PairScoreWorker`\nboth consult the configured `Phpdup\\Ml\\PairScorer` as the **last**\nclustering tier — after structural-hash, AST Jaccard + TED,\ncontainment, and IR have all rejected a pair. The client returns\nnull on transport failure (model unavailable, SSRF-rejected URL,\nmalformed response) so the rest of the pipeline keeps running with\nreduced precision rather than failing the run, mirroring the IR\nlifter's fail-graceful contract.\n\n---\n\n## TUI mode\n\n`--tui` opens a SugarCraft dashboard that drives the analysis pipeline\nfrom inside the runtime — counts and timings tick up as work\nprogresses instead of appearing post-hoc.\n\n![phpdup TUI dashboard with live spinner and progress](docs/media/tui-live.gif)\n\nWhat you get:\n\n- **Four-pane FlexBox dashboard** with live counts for Scanning /\n  Preprocessing / Clustering / Refactoring.\n- **SugarCraft spinner** in the running pane (one of 12 animation\n  styles) plus a real elapsed-time stopwatch.\n- **Sparkline of stage durations** as each stage completes.\n- **OSC 9;4 taskbar progress** — ConEmu, WezTerm, and Windows Terminal\n  pin a progress indicator on the OS taskbar.\n- **Six themes**: `ansi` (default), `plain`, `charm`, `dracula`,\n  `nord`, `catppuccin` — pick via `--theme=NAME`.\n- **Keyboard**: `q` / `Ctrl+C` quit, `Ctrl+Z` suspend, `↑/↓` cycle\n  pane focus, `Enter` open detail, `←/→` cycle clusters in detail\n  view, `t` toggle sort (impact / similarity / name), `h` toggle\n  help, `Esc` dismiss detail.\n- **`--plain`** forces plain CLI output even when `--tui` would\n  otherwise fire (handy for CI and pipes).\n\nThe pipeline runs *inside* the runtime via a cooperative\n`Pipeline::iter()` generator. Each `next()` advances to the next\nyield point — pre-stage, post-stage, every 16 files in\n`ScanningStage`, every 32 records streamed back from the parallel\npreprocess pool. Between yields the runtime renders, so the\nsparkline, file counts, parse-error totals, and taskbar progress\nbuild up frame-by-frame.\n\n---\n\n## Watch mode\n\n`--watch` keeps phpdup running and re-analyzes on every file change.\n\n![phpdup watch mode reacting to a file edit](docs/media/watch-mode.gif)\n\nA `React\\EventLoop` periodic timer (default 1.5 s) polls\n`filemtime()` for every scanned file, with `clearstatcache()` before\neach read to defeat PHP's stat cache. When any mtime changes, phpdup\nre-runs the full pipeline and reports `change detected (N files) —\nreload #X`.\n\nPolling instead of `inotify` / FSEvents keeps the watcher\ndependency-free and portable to macOS / Linux without an extension —\nat the cost of a small (≤ 1.5 s) reload latency.\n\n`Ctrl+C` (or `SIGTERM`) triggers a clean teardown via\n`Loop::addSignal`. **`--watch --tui` is supported** — the watcher's\nperiodic timer registers on the same `React\\EventLoop` instance the\nSugarCraft `Program` runs on, so the dashboard stays interactive\nwhile polling. On change, the watcher dispatches a\n`RestartPipelineMsg` to the program; the model resets state, rebuilds\nthe cooperative generator via its factory, and the live counts in the\npanes drop to zero before climbing back up.\n\n---\n\n## SIGINT soft-cancel\n\nPressing `Ctrl+C` mid-analysis (outside `--watch`) doesn't drop the\nwork in progress. `Phpdup\\Cli\\SignalHandler` registers an async\nSIGINT handler that flips `PipelineState::$cancelled`; the\ncooperative pipeline checks this between stages and short-circuits\nstraight to the Reporting stage. The user gets a partial report\ncovering whatever clusters had already formed, plus a non-zero\nexit code (130, the canonical SIGINT code) so CI workflows can\ndistinguish \"user cancelled\" from \"clean run\".\n\nA second `Ctrl+C` restores the default SIGINT handler so a truly\nstuck process can still be killed with the third-time-is-default\nescape hatch.\n\n---\n\n## `phpdup serve` REST API\n\n`phpdup serve` boots a minimal HTTP service on top of the analysis\npipeline. Useful for in-house dashboards, CI integrations, and a\nhosted-backend playground front-end.\n\n```bash\nphpdup serve --host 127.0.0.1 --port 8080\n```\n\nRoutes:\n\n| Method | Path           | Body                              | Returns |\n|--------|----------------|-----------------------------------|---------|\n| GET    | `/healthz`     | —                                 | `text/plain \"ok\"` |\n| POST   | `/analyze`     | JSON: `{\"paths\":[\"src\",\"lib\"]}`   | full JSON report |\n| POST   | `/jobs`        | JSON: same as `/analyze`          | `202 {\"job_id\":\"…\"}` |\n| GET    | `/jobs/{id}`   | —                                 | job status + result |\n\nImplementation is intentionally dependency-light:\n`stream_socket_server` plus a hand-rolled HTTP/1.1 parser. No\nReactPHP / Amp dep — for high-traffic deployments, swap in\nRoadrunner / FrankenPHP / FPM behind nginx and reuse\n`Phpdup\\Server\\Application` directly.\n\nSecurity defaults:\n\n- Default bind is `127.0.0.1` — exposing it externally requires a\n  reverse proxy.\n- Content-Length is capped at 16 MB.\n- JSON parsing uses `JSON_THROW_ON_ERROR` and returns 400 on\n  malformed bodies.\n- The `Application` class is fully transport-agnostic and\n  unit-tested without networking.\n\nSee [`docs/SERVER.md`](docs/SERVER.md) for the full contract,\ndeployment notes, and the playground architecture.\n\n---\n\n## Output formats\n\n```bash\nbin/phpdup analyze src \\\n    --json           phpdup.json \\\n    --html           phpdup-report \\\n    --sarif          phpdup.sarif \\\n    --gitlab-sast    phpdup.gitlab.json \\\n    --diff           ./phpdup-diffs \\\n    --patch          phpdup.patch \\\n    --checkstyle     phpdup.xml \\\n    --csv            phpdup.csv \\\n    --prometheus     phpdup.prom \\\n    --timeseries     phpdup-history.jsonl \\\n    --graphviz       phpdup.dot \\\n    --plantuml       phpdup.puml \\\n    --refactor-patch ./phpdup-refactors \\\n    --refactor-tests ./phpdup-refactor-tests\n```\n\nphpdup ships **twelve** output formats; the additions over the\nclassic five are:\n\n- **CSV** (`--csv=FILE`) — flat one-row-per-cluster-member; ideal\n  for spreadsheets / BI ingest. Multi-line signatures are\n  collapsed to one line so consumers don't have to handle embedded\n  newlines.\n- **Prometheus** text-format (`--prometheus=FILE`) — `# HELP` /\n  `# TYPE` annotated gauges suitable for `pushgateway` scraping or\n  CI dashboards. Per-pattern-tag counters included.\n- **Time-series JSONL** (`--timeseries=FILE`) — one append-only\n  line per run, commit-tagged via `GIT_COMMIT` / `GITHUB_SHA` /\n  `CI_COMMIT_SHA` / `BUILD_VCS_NUMBER` (with `.git/HEAD` fallback)\n  so duplicate-debt curves over time are trivial to track.\n- **Graphviz DOT** (`--graphviz=FILE`) — file→cluster bipartite\n  graph; render with `dot -Tpng phpdup.dot -o phpdup.png`.\n- **PlantUML** (`--plantuml=FILE`) — class-diagram-style with each\n  cluster as a `package` and pattern-tag stereotypes.\n- **Refactor patches** (`--refactor-patch=DIR`) — heuristic, manual-\n  review-required `.patch` files. Each adds a\n  `Refactored/\u003cid\u003e.php` skeleton with the suggested abstraction\n  signature plus per-member edit hints. Bails to a manual-review\n  header when `$this`/`self::`/`yield`/closure capture would make\n  mechanical replacement unsafe.\n- **PHPUnit test skeletons** (`--refactor-tests=DIR`) — one\n  `markTestIncomplete()` test class per cluster with a\n  `casesProvider()` populated from observed hole values.\n\n![phpdup emitting all five machine-readable formats in one run](docs/media/output-formats.gif)\n\n### CLI\n\nSugarCraft-styled colorized terminal output. Honors `--no-ansi` /\nnon-TTY: switches to `Theme::plain()` and skips the styled box / chips,\nproducing clean ASCII.\n\n### JSON (`--json=FILE`)\n\n```json\n{\n  \"phpdup_version\": \"0.1.0\",\n  \"summary\": { \"files\": 1888, \"blocks\": 12340, \"clusters\": 87 },\n  \"clusters\": [\n    {\n      \"id\": \"Xaeb0e34a\",\n      \"kind\": \"method\",\n      \"exact\": true,\n      \"similarity\": 1.0,\n      \"confidence\": 1.0,\n      \"impact\": 74,\n      \"pattern_tags\": [\"config-driven\", \"crud-handler\", \"sql-builder\"],\n      \"signature\": \"function findById(string $value): mixed\",\n      \"members\": [ ... ],\n      \"holes\": [\n        {\n          \"placeholder\": \"__P0\",\n          \"kind\": \"literal\",\n          \"inferred_type\": \"string\",\n          \"suggested_name\": \"$value\",\n          \"observed\": [\n            \"'SELECT * FROM users WHERE id = ?'\",\n            \"'SELECT * FROM products WHERE id = ?'\"\n          ]\n        },\n        {\n          \"placeholder\": \"__O0\",\n          \"kind\": \"optional_block\",\n          \"inferred_type\": \"bool\",\n          \"suggested_name\": \"$includeAudit\",\n          \"observed\": [\"audit($id);\", \"\u003cabsent\u003e\"],\n          \"present_in_members\": [0]\n        }\n      ]\n    }\n  ]\n}\n```\n\n`holes[].present_in_members` is unique to `optional_block` holes — it\nlists the cluster-member indices that *did* include the segment.\n\n### HTML (`--html=DIR`)\n\nA static-site report with:\n\n- Index page sorted by impact, with an **interactive mini-map** of\n  cluster-impact distribution (green bars exact, blue bars\n  near-duplicate; click to jump).\n- Client-side **column sort** (click any header) and a **search\n  filter** input.\n- **Copy-suggested-signature** button per cluster (Clipboard API + an\n  `execCommand` fallback for non-secure contexts).\n- Per-cluster page with member sources side-by-side, **inline syntax\n  highlighting** (no external dep, no build step), holes table,\n  unified diff between the first two members.\n- Optional rows tinted amber, with a \"type-3\" badge and italicized\n  `\u003cabsent\u003e` sentinel.\n\nJS lives in `app.js` next to `style.css` — both inlined into the\nbuild, no external deps, no build step.\n\n### SARIF (`--sarif=FILE`)\n\n[SARIF 2.1.0](https://docs.oasis-open.org/sarif/sarif/v2.1.0/os/sarif-v2.1.0-os.html)\noutput for GitHub Code Scanning and GitLab Code Quality. Each\nduplicate block becomes a `result` with rule\n`phpdup/duplicate-logic`, level `warning` for exact / `note` for\nnear-duplicate, the cluster's suggested signature in\n`properties.suggestedSignature`, and shared\n`partialFingerprints.clusterId` so SARIF consumers can group sibling\nresults into a single cluster annotation. Type-3 clusters add\n`properties.optionalSegmentCount` and `properties.hasOptionalSegments`.\n\nIn a GitHub Actions workflow:\n\n```yaml\n- run: vendor/bin/phpdup analyze src --sarif=phpdup.sarif --min-impact=50\n- uses: github/codeql-action/upload-sarif@v3\n  if: always()\n  with:\n    sarif_file: phpdup.sarif\n```\n\n### GitLab SAST (`--gitlab-sast=FILE`)\n\n[GitLab SAST report v15.x](https://gitlab.com/gitlab-org/security-products/security-report-schemas).\nSeverity is impact-bucketed (`\u003e100` High, `\u003e=50` Medium, `\u003e=20` Low,\notherwise Info). Confidence is `High` for exact clones, otherwise\nderived from the cluster confidence score. Wire it into\n`.gitlab-ci.yml` as a `report.sast` artifact so the MR security\nwidget surfaces duplicates.\n\n### Diff and patch (`--diff=DIR`, `--patch=FILE`)\n\n`--diff=DIR` writes one `.diff` file per cluster — pairwise unified\ndiffs from member[0] to each subsequent member, with a header comment\nshowing the suggested abstraction and anchor location. `--patch=FILE`\nconcatenates every cluster's diff into one cumulative patch file.\n\n### Checkstyle XML (`--checkstyle=FILE`)\n\nConsumable by Jenkins Warnings NG, Bitbucket Reports, Sonar, Detekt\n— anything that parses Checkstyle. Each duplicate is an `\u003cerror\u003e`\nwith `source=\"phpdup.duplicate-logic\"`, severity `warning` for exact\n/ `info` for near-duplicate, and a descriptive message linking back\nto the cluster id.\n\n---\n\n## Configuration\n\nDrop a `phpdup.json` next to your code, or pass `--config`:\n\n```json\n{\n  \"paths\":   [\"src\", \"app\", \"lib\"],\n  \"exclude\": [\"vendor/**\", \"node_modules/**\", \"**/*.tpl.php\", \"tests/**\"],\n  \"min_block_size\":       8,\n  \"max_block_size\":       800,\n  \"normalization_mode\":   \"aggressive\",\n  \"similarity_threshold\": 0.80,\n  \"tree_threshold\":       0.85,\n  \"min_cluster_impact\":   20,\n  \"max_df\":               0.01,\n  \"ngram_size\":           5,\n  \"cache_dir\":            \".phpdup-cache\",\n  \"workers\":              0,\n  \"incremental\":          true,\n  \"lazy_ast\":             true,\n  \"kinds\":                [\"method\", \"closure\"],\n  \"optional_blocks\": {\n    \"enabled\":             true,\n    \"containment\":         0.85,\n    \"min_overlap\":         0.6,\n    \"max_per_cluster\":     3,\n    \"min_segment_length\":  1\n  },\n  \"report\": {\n    \"html\": \"phpdup-report\",\n    \"json\": \"phpdup.json\"\n  }\n}\n```\n\nThe full schema lives at\n[`docs/config-schema.json`](docs/config-schema.json) and is validated\nby `ConfigLoader::validate()` whenever a config file is loaded.\n\nCLI flags override config values. Run `bin/phpdup analyze --help` for\nthe full list, or see [CLI reference](#cli-reference) below for\nper-flag details and effect snippets.\n\n![phpdup --validate-config rejecting a bad phpdup.json](docs/media/validate-config.gif)\n\n`bin/phpdup analyze --config phpdup.json --validate-config` runs\nvalidation in isolation and exits with `0` (OK) or `2` (with the\nfield-path error message), so CI can gate on config drift before any\nanalysis runs.\n\n---\n\n## CLI reference\n\n```\nUsage: phpdup analyze \u003cpaths...\u003e [options]\n```\n\n### Tuning\n\n#### `--min-block-size N` (default: `8`)\n\nMinimum AST node count for a block to be considered. Lower picks up\nsmall fragments (often noisy); higher quiets the report by ignoring\nshort blocks.\n\n```bash\nbin/phpdup analyze src --min-block-size=4   # very chatty — short snippets clustered\nbin/phpdup analyze src --min-block-size=20  # only meaningful methods\n```\n\n#### `--mode strict|default|aggressive` (default: `aggressive`)\n\nNormalization mode. See [Normalization modes](#normalization-modes).\n\n```bash\nbin/phpdup analyze src --mode=strict       # variable rename only — fewer false positives\nbin/phpdup analyze src --mode=aggressive   # also collapse names + literals — more findings\n```\n\n#### `--similarity N` (default: `0.80`)\n\nJaccard similarity threshold for the near-duplicate phase. Below\nthis, the type-3 fallback may still apply.\n\n```bash\nbin/phpdup analyze src --similarity=0.95   # strict — very obvious clones only\nbin/phpdup analyze src --similarity=0.65   # loose — many type-2 clones surfaced\n```\n\n#### `--max-df N` (default: `0.01`)\n\nMaximum document-frequency for an n-gram to be used as a\ncandidate-pair seed. Tuned for large codebases; bump for tiny\nfixtures where every n-gram is \"common\" by ratio.\n\n```bash\nbin/phpdup analyze tests/Fixtures --max-df=0.5    # tiny corpus needs higher cutoff\nbin/phpdup analyze big-codebase --max-df=0.001    # very strict on a 50k-block corpus\n```\n\n#### `--min-impact N` (default: `20`)\n\nMinimum cluster impact (≈ duplicated-line count) to include in\noutput. Quiets the report; doesn't change clustering.\n\n```bash\nbin/phpdup analyze src --min-impact=50    # only show big wins\nbin/phpdup analyze src --min-impact=1     # show everything that clustered\n```\n\n#### `--exact-only`\n\nSkip the near-duplicate phase entirely. ~6× faster on large corpora;\nemits only Type-1 (canonical-hash-equal) clusters.\n\n```bash\nbin/phpdup analyze src --exact-only --min-impact=50\n```\n\n#### `--kinds K1,K2,...`\n\nComma-separated block kinds to extract. Default = all of:\n`function|method|closure|arrow|if|for|foreach|while|do|try|switch|match`.\n\n```bash\nbin/phpdup analyze src --kinds=method                # methods only\nbin/phpdup analyze src --kinds=method,closure        # methods + closures\nbin/phpdup analyze src --kinds=if,foreach,switch     # control structures only\n```\n\n![phpdup --kinds=method narrowing scope](docs/media/kinds-filter.gif)\n\n#### `--max-memory MB`\n\nSoft memory ceiling. When peak RSS exceeds this mid-pipeline,\nphpdup logs `phpdup: peak RSS X MB exceeded --max-memory=Y` and\nsuggests `--exact-only`. `0` (default) disables.\n\n```bash\nbin/phpdup analyze huge-monorepo --max-memory=1024   # warn if RSS \u003e 1 GB\n```\n\n#### `--optional-blocks on|off` (default: `on`)\n\nType-3 / \"optional-segment\" detection master switch. See\n[Type-3 / optional-segment detection](#type-3--optional-segment-detection).\n\n```bash\nbin/phpdup analyze src --optional-blocks=off    # legacy Jaccard-only behaviour\n```\n\n#### `--optional-blocks-containment N` (default: `0.85`)\n\nContainment-fallback threshold for the type-3 path.\n\n```bash\nbin/phpdup analyze src --optional-blocks-containment=0.95   # very strict\nbin/phpdup analyze src --optional-blocks-containment=0.7    # permissive\n```\n\n### Output\n\n#### `--html DIR`\n\nWrite the interactive HTML report into DIR.\n\n```bash\nbin/phpdup analyze src --html=phpdup-report\n# open phpdup-report/index.html in a browser\n```\n\n#### `--json FILE`\n\nStructured JSON dump.\n\n```bash\nbin/phpdup analyze src --json=phpdup.json\njq '.clusters | length' phpdup.json\n```\n\n#### `--sarif FILE`\n\nSARIF 2.1.0 output for GitHub PR annotations.\n\n```bash\nbin/phpdup analyze src --sarif=phpdup.sarif\ngh codeql-action upload-sarif phpdup.sarif       # in CI\n```\n\n#### `--gitlab-sast FILE`\n\nGitLab SAST report v15.x for the MR security widget.\n\n```bash\nbin/phpdup analyze src --gitlab-sast=gl-sast-report.json\n```\n\n#### `--diff DIR`\n\nOne `.diff` file per cluster (pairwise unified diffs from member[0]).\n\n```bash\nbin/phpdup analyze src --diff=./phpdup-diffs\nls phpdup-diffs/*.diff | wc -l\n```\n\n#### `--patch FILE`\n\nSingle cumulative patch file containing every cluster diff.\n\n```bash\nbin/phpdup analyze src --patch=phpdup.patch\n```\n\n#### `--checkstyle FILE`\n\nCheckstyle XML for Jenkins / Sonar / Bitbucket consumers.\n\n```bash\nbin/phpdup analyze src --checkstyle=phpdup.xml\n```\n\n#### `--limit N` (default: `50`)\n\nMaximum number of clusters to print to the terminal. Doesn't affect\nfile outputs.\n\n```bash\nbin/phpdup analyze src --limit=10        # top 10 in the CLI\nbin/phpdup analyze src --limit=1000      # show everything\n```\n\n#### `--sort KEY[:asc|desc]` (default: `impact:desc`)\n\nOrder clusters by a chosen attribute and direction. Affects every\noutput format (CLI, JSON, HTML, SARIF, GitLab SAST, Diff, Checkstyle)\nbecause it runs inside the `Ranker` after impact + confidence are\ncomputed.\n\n| Key            | Sorts clusters by …                                                            |\n|----------------|---------------------------------------------------------------------------------|\n| `impact`       | Estimated lines saved if the abstraction is applied (the default).              |\n| `members`      | Number of duplicate blocks in the cluster (alias: `size`, `count`).             |\n| `block-size`   | Average AST node count per member (proxy for \"how big each duplicate is\").      |\n| `lines`        | Total duplicated lines across all members.                                      |\n| `similarity`   | Edge weight (1.0 for exact, lower for type-2/3 near-dupes).                     |\n| `confidence`   | The Ranker's `[0..1]` \"how safe is this refactor\" score.                        |\n| `name`         | First member's qualified name (`Namespace\\Class::method`), alphabetical.        |\n| `file`         | First member's file path, alphabetical.                                         |\n| `id`           | Cluster id, alphabetical (mainly useful for stable diffs across runs).          |\n\nDirection defaults to `desc`. Use `:asc` / `:desc`, or the shortcut\nprefixes `-` (desc) and `+` (asc):\n\n```bash\nbin/phpdup analyze src --sort=members:desc       # most-duplicated clusters first\nbin/phpdup analyze src --sort=block-size:desc    # biggest blocks first\nbin/phpdup analyze src --sort=lines              # most duplicated lines (desc default)\nbin/phpdup analyze src --sort=similarity:asc     # weakest matches first — review marginal type-3\nbin/phpdup analyze src --sort=confidence:desc    # safest refactors first\nbin/phpdup analyze src --sort=name:asc           # alphabetical\nbin/phpdup analyze src --sort=-impact            # leading - = desc shortcut\nbin/phpdup analyze src --sort=+lines             # leading + = asc shortcut\n```\n\nTies are broken consistently (members DESC ▸ similarity DESC ▸ id ASC)\nso the same input always produces the same final ordering regardless\nof the user's primary direction.\n\nIn the TUI, the `t` key cycles through the same sort keys live (impact\n→ members → block-size → lines → similarity → confidence → name → wrap),\nand shift-`T` toggles asc/desc. The current sort is shown in the Tip line.\n\n#### `--stats`\n\nPrint pipeline stage timings, block-kind histogram, and worker info\nafter the report.\n\n```bash\nbin/phpdup analyze src --stats\n```\n\n### Runtime\n\n#### `-c, --config FILE`\n\nLoad settings from a `phpdup.json` file. CLI flags override file\nvalues.\n\n```bash\nbin/phpdup analyze --config=phpdup.json src\n```\n\n#### `-j, --workers N` (default: `0` = auto)\n\nWorker count for parallel preprocess + pair scoring. `0` autodetects\nfrom `nproc` / `/proc/cpuinfo`. `1` forces serial.\n\n```bash\nbin/phpdup analyze src --workers=8       # explicit\nbin/phpdup analyze src -j 1              # serial — debugging\n```\n\n#### `--no-cache`\n\nDon't read or write the AST cache for this run.\n\n```bash\nbin/phpdup analyze src --no-cache    # benchmarking, or after upgrading php-parser\n```\n\n#### `--no-incremental`\n\nDisable per-file index reuse. Forces re-fingerprinting every file\neven when the IndexStore has a hit.\n\n```bash\nbin/phpdup analyze src --no-incremental    # benchmarking, or paranoid about cache\n```\n\n#### `--no-lazy-ast`\n\nKeep all original ASTs in memory throughout the run. Higher RSS,\nslightly faster anti-unification.\n\n```bash\nbin/phpdup analyze src --no-lazy-ast    # have RAM, want speed\n```\n\n#### `--stage NAME`\n\nHalt the pipeline after STAGE (one of `scanning`, `preprocessing`,\n`clustering`, `refactoring`, `reporting`). Useful for debugging\nincremental cache hits or profiling individual stages.\n\n```bash\nbin/phpdup analyze src --stage=preprocessing --stats   # measure how long parsing takes\nbin/phpdup analyze src --stage=clustering              # see clusters before refactor synthesis\n```\n\n### TUI / watch\n\n#### `--tui`\n\nShow the interactive SugarCraft dashboard while analysis runs.\nRequires a real TTY.\n\n```bash\nbin/phpdup analyze src --tui\n```\n\n#### `--theme NAME` (default: `ansi`)\n\nTUI theme: `ansi` | `plain` | `charm` | `dracula` | `nord` |\n`catppuccin`.\n\n```bash\nbin/phpdup analyze src --tui --theme=dracula\nbin/phpdup analyze src --tui --theme=catppuccin\n```\n\n#### `--plain`\n\nForce plain CLI output (no TUI, no ANSI colours). Useful when a CI\nshell reports `isatty()=true` but you want non-coloured output.\n\n```bash\nbin/phpdup analyze src --plain\n```\n\n#### `--watch`\n\nStay running and re-analyze on file changes via a poll-based\n`React\\EventLoop` timer. Combines with `--tui` for a live dashboard.\n\n```bash\nbin/phpdup analyze src --watch              # plain mode, prints reload messages\nbin/phpdup analyze src --watch --tui        # interactive dashboard, resets on change\n```\n\n### Validation\n\n#### `--validate-config`\n\nValidate the `--config` file against the documented schema and exit\nwithout running analysis. Exit `0` = OK, `2` = error (with field\npath).\n\n```bash\nbin/phpdup analyze --config=phpdup.json --validate-config \u0026\u0026 echo OK\n```\n\n### Shell completion\n\nphpdup ships its own `completion` sub-command. The output is the standard\nSymfony Console completion script for the chosen shell, with **commented-out\ninstallation instructions** prepended so you can `cat` the dump and follow the\nsteps inline:\n\n```bash\nbin/phpdup completion bash    # → bash script + comments showing 3 install paths\nbin/phpdup completion fish    # → fish script + comments\nbin/phpdup completion zsh     # → zsh script + comments (#compdef stays first)\n```\n\nThe most common one-liner per shell:\n\n```bash\n# bash — per-user completion under XDG home\nmkdir -p ~/.local/share/bash-completion/completions\nphpdup completion bash \u003e ~/.local/share/bash-completion/completions/phpdup\n\n# fish — auto-loaded on next shell start\nmkdir -p ~/.config/fish/completions\nphpdup completion fish \u003e ~/.config/fish/completions/phpdup.fish\n\n# zsh — pick a directory on $fpath BEFORE compinit\nmkdir -p ~/.zsh/completions\nphpdup completion zsh \u003e ~/.zsh/completions/_phpdup\n# then in ~/.zshrc, before 'compinit':  fpath=(~/.zsh/completions $fpath)\n```\n\nWhen `shell` is omitted, `$SHELL` is consulted. Unknown shells exit `2`.\n\n### Exit codes\n\n| Code | Meaning                                              |\n|------|------------------------------------------------------|\n| `0`  | Analysis ran. Note: phpdup does NOT exit non-zero    |\n|      | when clusters are found. Use the JSON report to gate |\n|      | CI; an empty `clusters` array means clean.           |\n| `1`  | Internal error.                                      |\n| `2`  | Missing required argument or invalid configuration.  |\n\n### Environment variables\n\n| Variable          | Effect                                                |\n|-------------------|-------------------------------------------------------|\n| `PHPDUP_WORKERS`  | Override worker count (lower precedence than `-j`).   |\n| `COLUMNS`         | Override terminal width detection for the CLI report. |\n\n---\n\n## Programmatic use\n\nThe pipeline is fully composable from PHP. The simplest entrypoint is\n`Pipeline` itself; everything else is plugged into it via stage\nconstructors:\n\n```php\nuse Phpdup\\Cli\\Config;\nuse Phpdup\\Pipeline\\Pipeline;\nuse Phpdup\\Pipeline\\PipelineState;\nuse Phpdup\\Pipeline\\Stages\\ScanningStage;\nuse Phpdup\\Pipeline\\Stages\\PreprocessStage;\nuse Phpdup\\Pipeline\\Stages\\ClusterStage;\nuse Phpdup\\Pipeline\\Stages\\RefactorStage;\nuse Phpdup\\Pipeline\\Stages\\ReportStage;\nuse Symfony\\Component\\Console\\Output\\NullOutput;\n\n$config = new Config(\n    paths: ['src'],\n    exclude: ['vendor/**'],\n    optionalBlocksEnabled: true,\n);\n$state = new PipelineState($config);\n\n(new Pipeline([\n    new ScanningStage(),\n    new PreprocessStage(useCache: true),\n    new ClusterStage(exactOnly: false),\n    new RefactorStage(useCache: true),\n    new ReportStage(limit: 50, showStats: false),\n]))-\u003erun($state, new NullOutput());\n\nforeach ($state-\u003eclusters as $c) {\n    echo \"{$c-\u003esize()} members, signature: {$c-\u003esignature}\\n\";\n}\n```\n\nFor finer-grained control — e.g. to run preprocessing without\nclustering, or to drive the pipeline cooperatively from your own\nevent loop — use `Pipeline::iter()` and pump the generator yourself.\nThat's exactly how the TUI works.\n\n---\n\n## Examples\n\n### Threshold-gated notification (type-2)\n\nInput:\n\n```php\npublic function notifyHigh($user, int $score): void {\n    if ($score \u003e 10) { $this-\u003emailer-\u003esend('admin', $user); }\n}\npublic function notifyMid($user, int $score): void {\n    if ($score \u003e 20) { $this-\u003emailer-\u003esend('moderator', $user); }\n}\n```\n\nOutput:\n\n```\nfunction notifyByThresholdAndStrategy(\n    int $threshold,\n    string $value,\n): mixed\n```\n\nHoles:\n\n| Param        | Type    | Observed                          |\n|--------------|---------|-----------------------------------|\n| `$threshold` | int     | `10, 20`                          |\n| `$value`     | string  | `'admin', 'moderator'`            |\n\nPatterns: `config-driven`.\n\n### Repository CRUD (type-1 / type-2)\n\nThree classes with `findById($db, $id)` differing only in table name.\n\n```\nfunction findById(string $value): mixed\n```\n\n| Param    | Type   | Observed                                                |\n|----------|--------|---------------------------------------------------------|\n| `$value` | string | `'SELECT * FROM users WHERE id = ?'`,                   |\n|          |        | `'SELECT * FROM products WHERE id = ?'`,                |\n|          |        | `'SELECT * FROM orders WHERE id = ?'`                   |\n\nPatterns: `config-driven, crud-handler, sql-builder`.\n\n### Optional segments (type-3)\n\nThe user-asked-for case — see\n[Type-3 / optional-segment detection](#type-3--optional-segment-detection)\nfor the algorithm. Two `if` bodies share the same first three\nstatements; the longer one has two extra calls at the tail.\n\n```\nfunction extractedFunction(\n    bool $includeSomeOtherLogic = false,\n    bool $includeAndMore = false,\n): mixed\n```\n\n| Param                     | Type | Kind             | Observed                              |\n|---------------------------|------|------------------|---------------------------------------|\n| `$includeSomeOtherLogic`  | bool | `optional_block` | `some_other_logic($here);`, `\u003cabsent\u003e` |\n| `$includeAndMore`         | bool | `optional_block` | `and_more($f);`, `\u003cabsent\u003e`            |\n\nPatterns: `optional-segments`.\n\n### Strategy dispatch\n\nA chain of `if (...)` calls each invoking a different validator, all\nwith the same shape. Cluster tagged `strategy`, single hole on the\ncall name, with the list of method names as observed values — a\nclear hint to extract an interface and an array of strategies.\n\n---\n\n## Static analysis \u0026 config validation\n\nCI runs three layers of static checks on every push and PR:\n\n```yaml\n- run: find src tests -name '*.php' -print0 | xargs -0 -n1 -P4 php -l \u003e /dev/null\n- run: vendor/bin/phpstan analyse --memory-limit=1G --no-progress\n- run: vendor/bin/psalm --no-progress --no-cache\n```\n\n- **PHPStan level 6** is clean (no baseline). Configuration in\n  [`phpstan.neon`](phpstan.neon).\n- **Psalm** runs at error level 6 with a tracked baseline\n  (`psalm-baseline.xml`) for legacy `InvalidArrayOffset` /\n  `MissingParamType` findings in the APTED implementation.\n- **Config schema validation** — `ConfigLoader::validate()` mirrors\n  [`docs/config-schema.json`](docs/config-schema.json) and throws a\n  `RuntimeException` whose message names the offending field on the\n  first violation. `--validate-config` runs validation in isolation\n  for CI gating.\n\n---\n\n## Benchmarks\n\nphpdup ships a real comparative benchmark harness — not synthetic\nmicrobenchmarks. The suite runs phpdup alongside the other PHP\nduplicate-detection tools that anyone might reach for, on a curated\nmix of public OSS corpora, and reports wall time, peak RSS, cluster\ncounts, plus precision / recall / F1 against ground truth. It's\ndesigned to be **honest** — phpdup is not the best tool in every\ncell, and the matrix below says so out loud.\n\n### Comparative benchmark suite\n\n```bash\nbench/run-all.sh\n```\n\nThat's the whole flow:\n\n1. Auto-download `phpcpd.phar` (phar-only; ~3 MB).\n2. `npm install jscpd@4` into `bench/tools/` (skipped if no node).\n3. Shallow-clone OSS corpora (Symfony Console, Laravel HTTP,\n   PHPUnit, WordPress core).\n4. Generate the synthetic-fuzz corpus deterministically.\n5. Run every available tool against every corpus with a per-tool\n   wall-time cap.\n6. Score each tool's output against the synthetic corpus's\n   `.ground-truth.json`.\n\nOutputs:\n\n- `bench/results/latest.md` — per (tool, corpus) wall time / RSS /\n  cluster count.\n- `bench/results/detection-rate.md` — per-tool precision / recall /\n  F1 on the synthetic corpus (the only fair-fight scenario, since\n  we know the right answer).\n\nTools probed (auto-skipped when missing — `—` in the table):\n\n| Tool        | Source                                | Auto-installed |\n|-------------|---------------------------------------|----------------|\n| **phpdup**  | `bin/phpdup` (this repo)              | always         |\n| phpcpd      | `bench/tools/phpcpd.phar`             | yes            |\n| pmd-cpd     | system `pmd cpd`                      | no — install separately |\n| jscpd       | `bench/tools/node_modules/.bin/jscpd` | yes (npm)      |\n| simian      | system `simian`                       | no — commercial |\n\nSee [`bench/README.md`](bench/README.md) for adding new tools or\ncorpora, and `bench/feature-matrix.md` for the capability comparison.\n\n### Feature matrix\n\nA hand-curated 40+ row capability comparison vs phpcpd, pmd-cpd,\njscpd, and simian — full table in\n[`bench/feature-matrix.md`](bench/feature-matrix.md). The summary:\n\n**Where phpdup wins:**\n\n- **Suggested-refactor output.** No other tool produces a\n  parameterised function signature, hole inventory, and unified-diff\n  patches. If your goal is \"I want to refactor the duplicates\",\n  phpdup is the only practical choice.\n- **Type-3 detection.** `--optional-blocks` finds clones that\n  differ in optional segments — phpcpd / pmd-cpd / jscpd require\n  contiguous identical token runs and miss this entirely.\n- **Reporter coverage.** phpdup ships 12 output formats including\n  SARIF, GitLab SAST, Prometheus, Graphviz, and time-series JSONL.\n  None of the others ship more than three.\n- **Pattern-tag classification.** Tagging clusters as `sql-builder`,\n  `crud-handler`, `controller-action`, etc. is unique and lets you\n  filter / route findings by type.\n- **Live workflow.** `--watch` + TUI is for the inner-loop developer\n  who wants feedback as they refactor; nothing else covers this.\n- **Honesty.** phpdup ships a synthetic-fuzz ground-truth scorer\n  (`bench/score.php`) and publishes precision/recall — the others\n  publish neither.\n\n**Where phpdup loses:**\n\n- **Pure speed for exact clones.** phpcpd's tokenizer is dead\n  simple and beats phpdup's AST + APTED pipeline on raw wall time\n  when all you want is \"is anything copy-pasted verbatim?\". Use\n  `phpdup --exact-only` to close most of that gap.\n- **Cold-start RSS for tiny inputs.** phpcpd peaks ~50 MB; phpdup\n  peaks ~60–80 MB because it loads php-parser, Symfony Console,\n  and the whole pipeline scaffolding.\n- **Mature IDE integration.** PMD has a JetBrains plugin going back\n  over a decade; phpdup has none yet (the contract for a future\n  IntelliJ plugin lives at\n  [`docs/JETBRAINS_PLUGIN.md`](docs/JETBRAINS_PLUGIN.md)). If your\n  team already lives inside PhpStorm's PMD inspections, that's a\n  real lock-in.\n- **Multi-language support.** jscpd handles PHP, JS/TS, Python,\n  Java, Ruby, Go, Rust, etc. from a single binary. phpdup is\n  PHP-only by design.\n- **Maintenance perception.** phpcpd is archived but it's the\n  historical default — many CI pipelines still run it.\n\n**Quick chooser:**\n\n| If you need…                         | Reach for…  |\n|--------------------------------------|-------------|\n| Refactor-actionable output, type-3 detection, pattern tags, any of the unique reporters | **phpdup** |\n| A fast exact-clone gate in CI, no other features | phpcpd |\n| JetBrains IDE inspection / polyglot Java + PHP project | pmd-cpd |\n| Multi-language: PHP + JS + Python + … from one binary | jscpd |\n| Commercial line-based diff finder | simian |\n\n### Internal scaling benchmark\n\nphpdup's own parallel scaling on a real PHP corpus\n(`include/Api/`: 530 files, 3,295 comparable blocks, 96 clusters):\n\n| Configuration                              | Wall time | vs serial |\n|--------------------------------------------|----------:|----------:|\n| serial (`--workers 1`), cold cache         | 61.13 s   | 1.00×     |\n| 4 workers, cold cache                      | 30.39 s   | 2.01×     |\n| 8 workers, cold cache                      | 21.11 s   | 2.90×     |\n| 16 workers, cold cache                     | 17.47 s   | 3.50×     |\n| 8 workers, `--exact-only`                  |  5.74 s   | 10.65×    |\n\nCluster output is byte-identical across configurations — the\nspeedups don't come from skipping work. APTED does correct\nZhang-Shasha work (slower per pair than a bounded heuristic would\nbe); the user-visible win comes from parallelism stacking on top.\n\nReproduce on your own corpus:\n\n```bash\nrm -rf .phpdup-cache\n/usr/bin/time -f \"%e s wall, %M KB rss\" \\\n  bin/phpdup analyze /path/to/your/code \\\n    --min-impact 100 --stats --workers 8 --no-cache\n```\n\n#### Wall-time stage breakdown (8 workers, cold cache)\n\n| Stage         | Time   | Share | Implementation                                       |\n|---------------|-------:|------:|------------------------------------------------------|\n| Preprocess    | 1.4 s  |  7%   | Parse + extract + normalize + fingerprint, parallel  |\n| Cluster       | 14.3 s | 68%   | APTED + parallel candidate-pair scoring              |\n| Refactor      | 4.2 s  | 20%   | Anti-unify + synthesize + pattern-tag (parallel via `RefactorWorker`) |\n| Reporting/IO  | 1.2 s  |  5%   |                                                      |\n| **Total**     | 21.1 s | 100%  | Peak RSS 464 MB                                      |\n\nClustering still dominates, with the parallel TED workload itself\nbeing the largest single chunk.\n\n#### Honest reporting on what doesn't pay off\n\n- **APTED alone is a serial regression.** Correct, but slower than\n  the older bounded top-down: 61 s vs 35 s on this corpus, single\n  thread. The win has to come from parallelism stacking on top.\n- **Diminishing returns past 8 workers.** 16 → 8 saves 4 s; the\n  serial sections impose an Amdahl's-law ceiling at ~2× on this\n  corpus.\n- **Lazy AST is currently no faster than full-memory mode at this\n  scale** — the ~2 s reload overhead roughly cancels the small RSS\n  saving. The default (`lazy_ast: true`) is conservative; if you\n  have a large host, set it to `false` in `phpdup.json` for a speed\n  bump.\n- **Incremental warm-cache savings are modest** when the cluster\n  phase dominates total wall time. Incremental shines when the\n  corpus grows by a few files per run.\n- **Persistent cluster cache** wins big on no-change re-runs — it\n  skips Cluster + Refactor entirely. But invalidates wholesale on\n  any block change, so it's \"warm\" only when no source moved.\n\n#### Tuning knobs that move the needle\n\n- `--workers N` — parallelism level (0 = auto-detect).\n- `--exact-only` — skip near-duplicate detection. **5.74 s wall** on\n  the corpus above (8 workers) — the fastest \"is this clean?\" gate.\n- `--similarity` (default 0.80) — raising prunes more pairs before\n  TED.\n- `--min-block-size` (default 8) — fewer blocks, fewer pairs.\n- `--max-df` (default 0.01) — stricter rare-gram cutoff.\n- `--auto-tune` — picks the size-appropriate combination of the\n  above based on your corpus shape; explicit flags still win.\n- `--ted-weights=semantic` — weights method calls heavier than\n  literals; slower per pair, but better at clustering behavioural\n  near-duplicates.\n- `--no-incremental` — disable per-file index snapshots.\n- `--no-lazy-ast` — keep all original ASTs in RAM.\n\nFor exploratory analysis on any codebase:\n\n```bash\nbin/phpdup analyze src --auto-tune --workers $(nproc)\n```\n\nCI gate (exact clones only, fastest):\n\n```bash\nbin/phpdup analyze src --exact-only --min-impact 30 --workers 8\n```\n\nMassive monorepo (low memory budget):\n\n```bash\nbin/phpdup analyze src \\\n  --workers $(nproc) \\\n  --min-block-size 20 \\\n  --similarity 0.88\n# lazy_ast and incremental are on by default.\n```\n\n---\n\n## Architecture\n\nSee [`ARCHITECTURE.md`](ARCHITECTURE.md) for the full design document\nincluding data structures, algorithm details, and the staged\nimplementation plan.\n\nThe project is organized as:\n\n```\nsrc/\n  Cli/            CLI entry point, ConfigLoader (with schema validation), Command\n  Pipeline/       Stage enum, PipelineState, ProgressListener,\n                  CooperativeStageInterface, Pipeline orchestrator,\n                  five Stage classes (Scanning/Preprocess/Cluster/Refactor/Report).\n  Scanning/       File walking and glob filtering\n  Parsing/        nikic/php-parser wrapper + AST cache\n  Extraction/     Block selection (with --kinds filter) + lazy AST loader\n  Normalization/  Three-pass canonicalization\n  Fingerprint/    Structural hash + n-gram bag\n  Index/          In-memory + inverted index\n  Persistence/    IndexStore (per-file block snapshots)\n  Similarity/     Jaccard, ContainmentSimilarity (type-3), APTED tree-edit-distance\n  Clustering/     Hash-bucket + union-find with type-3 containment fallback\n  Parallel/       WorkerPool (run + runStreaming) + Preprocess/PairScore workers\n  Refactor/       Anti-unification with statement-array LCS,\n                  parameter/signature synth (incl. optional_block bools),\n                  pattern recognition.\n  Reporting/      CLI / JSON / HTML / SARIF / GitLab SAST / Diff / Checkstyle\n                  reporters + ranker\n  Tui/            PhpdupModel (SugarCraft Model + ProgressListener), ViewState,\n                  Msg types (StagePumpedMsg, RestartPipelineMsg),\n                  TuiRunner (theme resolution + Program boot, with\n                  shared-loop support for --watch + --tui)\n  Watch/          WatchRunner (poll-based React\\EventLoop watcher)\n  Util/           AST serializer, hash helpers, line range\n```\n\nModules have small, documented surfaces. New normalization rules,\nsimilarity metrics, or pattern recognizers plug in without touching\nthe rest of the pipeline.\n\n---\n\n## Testing\n\n```bash\ncomposer test                  # full suite\nvendor/bin/phpunit --testsuite Unit\nvendor/bin/phpunit --testsuite Integration\ncomposer coverage:html         # writes tests/phpunit/coverage-html/\n```\n\nThe test suite (165+ tests, 469+ assertions) covers:\n\n- Scanner glob semantics\n- Normalizer canonicalization across all three modes\n- N-gram fingerprint determinism + Jaccard floor on unrelated code\n- ContainmentSimilarity: subset/overlap/empty cases + size-ratio guard\n- APTED correctness on identical, renamed, unrelated trees, plus\n  bounded short-circuit behavior\n- WorkerPool serial path, parallel path (skipped without pcntl), empty\n  input, CPU-count detection, and the streaming variant including\n  child-exception propagation and array-vs-generator task returns\n- IndexStore round-trip, file-change invalidation, config-key\n  invalidation\n- Pipeline orchestration: stage ordering, `stopAfter` halting at the\n  right boundary, `stageProgress` reset between stages, cooperative\n  iter() yielding pre/post-stage and mid-stage for cooperative stages\n- ProgressListener wiring: per-file scan + preprocess events fire,\n  null listener is the default and never branches stage behaviour\n- BlockExtractor `--kinds` filter rejects unknown kinds and keeps the\n  ones you ask for\n- ConfigLoader schema validation: every field's bounds + the\n  cross-field `min_block_size \u003c= max_block_size` rule + the\n  `optional_blocks` sub-object\n- Anti-unifier hole discovery on t","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdetain%2Fphp-dup-finder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdetain%2Fphp-dup-finder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdetain%2Fphp-dup-finder/lists"}