{"id":40765668,"url":"https://github.com/asynkron/Asynkron.QuickDup","last_synced_at":"2026-01-30T20:00:40.108Z","repository":{"id":332408421,"uuid":"1127246595","full_name":"asynkron/Asynkron.QuickDup","owner":"asynkron","description":"AI-era code clone detector using structural shape fingerprinting","archived":false,"fork":false,"pushed_at":"2026-01-22T21:05:00.000Z","size":3520,"stargazers_count":2,"open_issues_count":9,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-23T13:11:46.451Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/asynkron.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-03T13:45:52.000Z","updated_at":"2026-01-17T17:13:52.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/asynkron/Asynkron.QuickDup","commit_stats":null,"previous_names":["asynkron/asynkron.quickdup"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/asynkron/Asynkron.QuickDup","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asynkron%2FAsynkron.QuickDup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asynkron%2FAsynkron.QuickDup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asynkron%2FAsynkron.QuickDup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asynkron%2FAsynkron.QuickDup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/asynkron","download_url":"https://codeload.github.com/asynkron/Asynkron.QuickDup/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asynkron%2FAsynkron.QuickDup/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28884244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-29T19:55:09.949Z","status":"ssl_error","status_checked_at":"2026-01-29T19:55:08.490Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-21T18:00:26.181Z","updated_at":"2026-01-30T20:00:40.075Z","avatar_url":"https://github.com/asynkron.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"[![CI](https://github.com/asynkron/Asynkron.QuickDup/actions/workflows/quickdup.yml/badge.svg)](https://github.com/asynkron/Asynkron.QuickDup/actions/workflows/quickdup.yml)\n\n\u003cimg src=\"assets/images/logo.png\" width=\"100%\" /\u003e\n\n# QuickDup - Delta-Indent Clone Detection\n\n## The shape of code\n\nQuickDup is a fast structural code clone detector that:\n* Identifies duplicate code patterns using indent-delta fingerprinting.\n* Is designed as a candidate generator for AI-assisted code review.\n\n## Performance\n\n- **~100k lines of code in ~500 ms** on 8 cores\n- Parallel file parsing and pattern detection\n- Lightweight fingerprinting (no AST parsing)\n\n## Philosophy\n\nTraditional clone detection optimizes for **precision** — minimizing false positives. QuickDup optimizes for **speed and recall** — surface candidates fast, let AI verify.\n\n```\n┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐\n│    QuickDup     │ ──▶ │    AI Agent     │ ──▶ │  Human Decision │\n│  (candidates)   │     │  (verification) │     │                 │\n└─────────────────┘     └─────────────────┘     └─────────────────┘\n```\n\n## Algorithm\n\n### Phase 1: Parse Files (Parallel)\n\nExtract structural fingerprint per line:\n\n| Field         | Description                              |\n| ------------- | ---------------------------------------- |\n| `IndentDelta` | Change in indentation from previous line |\n| `Word`        | First token on the line                  |\n| `SourceLine`  | Original source for output               |\n\nComments and blank lines are skipped. Comment prefixes are auto-detected by file extension.\n\nExample:\n\u003cimg src=\"assets/images/code-shape-fingerprint.png\" width=\"100%\" /\u003e\n\nThis pattern, generates a fingerprint sequence based on the indentation deltas and first words that is deterministic, every occurance of this pattern will yield the same fingerprint sequence.\n\nThe fingerprint may seem too naive, but in practice it captures the **structural shape** of code well enough to find \"candidate\" duplicates quickly, the token similarity phase later filters out most of the false positives.\n\nThe actual identification of duplicates happens after phase 3, you feed this information to an AI model to verify if the candidates are true duplicates or not, and either automatically refactor them or present them to a human for review.\n\n### Phase 2: Grow-Based Pattern Detection (Parallel)\n\n1. Generate base patterns of minimum size (default: 3 lines)\n2. Keep patterns with 3+ occurrences\n3. Grow patterns by 1 line, repeat until no patterns survive\n4. Track which occurrences grew vs. stopped (only report maximal patterns)\n\nThis finds the **longest** duplicate patterns, not just fixed windows.\n\n### Phase 3: Token Similarity \u0026 Scoring\n\nPatterns with similar structure but different actual code are filtered:\n\n1. Tokenize source lines of each occurrence\n2. Compute Jaccard similarity (intersection/union of token sets)\n3. Filter patterns below threshold (default: 50%)\n4. Score patterns: `uniqueWords + (similarity × 5)`\n\nThis eliminates most false positives like \"all error handlers look similar structurally but have different messages.\" High similarity (especially 100% verbatim matches) boosts the score, surfacing the most actionable duplications first.\n\n### Phase 4: Output\n\nResults written to `.quickdup/` directory:\n- `results.json` — Machine-readable patterns with locations\n\n## Installation\n\n**Linux/macOS:**\n```bash\ncurl -sSL https://raw.githubusercontent.com/asynkron/Asynkron.QuickDup/main/install.sh | bash\n```\n\n**Windows (PowerShell):**\n```powershell\niwr -useb https://raw.githubusercontent.com/asynkron/Asynkron.QuickDup/main/install.ps1 | iex\n```\n\n**From source:**\n```bash\ngo install github.com/asynkron/Asynkron.QuickDup/cmd/quickdup@latest\n```\n\n## Usage\n\n```bash\n# Scan Go files in current directory\nquickdup -path . -ext .go\n\n# Scan C# files with stricter similarity threshold\nquickdup -path ./src -ext .cs -min-similarity 0.9\n\n# Show top 20 patterns, require 5+ occurrences\nquickdup -path . -ext .ts -top 20 -min 5\n\n# Show detailed code for patterns 0-5\nquickdup -path . -ext .go -select 0..5\n\n# Exclude generated files\nquickdup -path . -ext .go -exclude \"*.pb.go,*_gen.go\"\n\n# Use a different detection strategy\nquickdup -path . -ext .go -strategy word-only\n\n# Compare duplicates between commits\nquickdup -path . -ext .go -compare origin/main..HEAD\n\n# Cap pattern growth at 50 lines\nquickdup -path . -ext .go -max-size 50\n\n# Verbose progress for long-running phases\nquickdup -path . -ext .go -debug\n```\n\n## Flags\n\n| Flag                  | Default             | Description                                                      |\n| --------------------- | ------------------- | ---------------------------------------------------------------- |\n| `-path`               | `.`                 | Directory to scan recursively                                    |\n| `-file`               |                     | Scan a single file (overrides `-path`)                           |\n| `-ext`                | `.go`               | File extension to match                                          |\n| `-min`                | `2`                 | Minimum occurrences to report                                    |\n| `-min-size`           | `3`                 | Base pattern size (lines) to start growing from                  |\n| `-max-size`           | `0`                 | Maximum pattern size to grow to (0 = no limit)                   |\n| `-min-score`          | `5`                 | Minimum score (unique words + similarity bonus)                  |\n| `-min-similarity`     | `0.75`              | Minimum token similarity between occurrences (0.0-1.0)           |\n| `-top`                | `10`                | Show top N patterns by score                                     |\n| `-select`             |                     | Show detailed output for patterns (format: `skip..limit`)        |\n| `-strategy`           | `normalized-indent` | Detection strategy (see below)                                   |\n| `-comment`            | auto                | Override comment prefix (auto-detected by extension)             |\n| `-exclude`            |                     | Exclude files matching patterns (comma-separated globs)          |\n| `-no-cache`           | `false`             | Disable incremental caching, force full re-parse                 |\n| `-keep-overlaps`      | `false`             | Keep overlapping occurrences (don't prune adjacent matches)      |\n| `-github-annotations` | `false`             | Output GitHub Actions annotations for inline PR comments         |\n| `-github-level`       | `warning`           | GitHub annotation level: `notice`, `warning`, or `error`         |\n| `-git-diff`           |                     | Only annotate files changed vs this git ref (e.g., `origin/main`)|\n| `-compare`            |                     | Compare duplicates between two commits (format: `base..head`)    |\n| `-debug`              | `false`             | Print verbose progress for long-running phases                   |\n| `-timeout`            | `20`                | Hard timeout in seconds (0 disables)                             |\n\n## Detection Strategies\n\n| Strategy            | Description                                                |\n| ------------------- | ---------------------------------------------------------- |\n| `normalized-indent` | Default. Uses indent deltas and first word per line        |\n| `word-indent`       | Uses raw indentation level and first word                  |\n| `word-only`         | Ignores indentation, matches on first words only           |\n| `inlineable`        | Detects small patterns suitable for inline extraction      |\n\n## GitHub Actions Integration\n\nQuickDup can output annotations that GitHub displays as inline comments on pull requests:\n\n```yaml\n- name: Run QuickDup\n  run: quickdup -path . -ext .go --github-annotations --no-cache\n\n# Only annotate changed files in a PR\n- name: Run QuickDup on changed files\n  run: quickdup -path . -ext .go --github-annotations --git-diff origin/main\n\n# Use error level instead of warning\n- name: Run QuickDup (fail on duplicates)\n  run: quickdup -path . -ext .go --github-annotations --github-level error\n```\n\nWhen `--github-annotations` is enabled, QuickDup outputs in GitHub's annotation format.\n\n## Incremental Caching\n\nQuickDup caches parsed file data in `.quickdup/cache.gob`. On subsequent runs, only modified files are re-parsed:\n\n```\nParsed 558 files (542 cached, 16 parsed) (98234 lines of code)\n```\n\nThis dramatically speeds up repeated runs during development. Use `-no-cache` to force a full re-parse.\n\n## Ignoring Patterns\n\nCreate `.quickdup/ignore.json` to suppress known patterns:\n\n```json\n{\n  \"description\": \"Patterns to ignore\",\n  \"ignored\": [\n    \"56c2f5f9b27ed5a0\",\n    \"c32ca0ee344f8e23\"\n  ]\n}\n```\n\nPattern hashes are shown in the output for easy copy-paste.\n\n## Supported Languages\n\nComment prefixes are auto-detected for:\n\n- **C-style** (`//`): Go, C, C++, Java, JavaScript, TypeScript, C#, Swift, Kotlin, Rust, PHP, Dart, Zig\n- **Hash** (`#`): Python, Ruby, Shell, Perl, R, YAML, TOML, PowerShell, Nim, Julia, Elixir\n- **Double-dash** (`--`): SQL, Lua, Haskell, Elm, Ada, VHDL\n- **Semicolon** (`;`): Lisp, Clojure, Scheme, Assembly\n- **Percent** (`%`): LaTeX, MATLAB, Erlang, Prolog\n\nUse `-comment` to override for unsupported extensions.\n\n## Example Output\n\n```\nScanning 558 files using 8 workers...\nParsed 558 files (98234 lines of code)\nDetecting patterns...\nGrowth stopped at 148 lines\nFiltered 23 low-similarity patterns (similarity \u003c 75%)\n\nDuplication hotspots (lines):\n  1131 src/services/auth.go\n   940 src/services/oauth.go\n   894 src/services/saml.go\n\nTotal: 774 duplicate patterns in 558 files (98234 lines) in 544ms\nResults written to: .quickdup/normalized-indent-results.json\n```\n\nWith `--select 0..3`:\n\n```\nPattern 1  [a1b2c3d4e5f67890]  Score 19  100% similar  47 lines  2 occurrences\n\n  Occurrence 1 src/services/auth.go:142\n    // ... code block with syntax highlighting ...\n\n  Occurrence 2 src/services/oauth.go:89\n    // ... code block with syntax highlighting ...\n\n───────────────────────────────────────────────────────────────────────────────\nShowing pattern 0 to 3\n\nTotal: 774 duplicate patterns in 558 files (98234 lines) in 544ms\nResults written to: .quickdup/normalized-indent-results.json\n```\n\n## Screenshots\n\n`quickdup --path . --ext .go --select 4..1 --max-size 7`\n\n![QuickDup select output](assets/images/quickdup-select-output.png)\n\n## Limitations\n\nThis is a **heuristic candidate generator**:\n\n- **False positives** — Structural similarity doesn't guarantee semantic duplication\n- **False negatives** — Different structure with same semantics won't match\n\nToken similarity filtering and clustering catch cases where occurrences differ significantly. Small differences (a few tokens in a large pattern) won't affect similarity much — which is intentional, as those are likely real duplicates with minor variations.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasynkron%2FAsynkron.QuickDup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasynkron%2FAsynkron.QuickDup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasynkron%2FAsynkron.QuickDup/lists"}