{"id":51229240,"url":"https://github.com/thromel/patchsmith","last_synced_at":"2026-06-28T14:31:09.163Z","repository":{"id":364094246,"uuid":"1266350897","full_name":"thromel/patchsmith","owner":"thromel","description":"Research platform for evaluating AI software-maintenance agents with sandboxed patch validation and evidence reports.","archived":false,"fork":false,"pushed_at":"2026-06-11T17:05:00.000Z","size":778,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-11T17:06:46.639Z","etag":null,"topics":["ai-agents","benchmarking","deepagents","program-repair","software-maintenance"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thromel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-11T14:36:41.000Z","updated_at":"2026-06-11T17:05:06.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/thromel/patchsmith","commit_stats":null,"previous_names":["thromel/patchsmith"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/thromel/patchsmith","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thromel%2Fpatchsmith","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thromel%2Fpatchsmith/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thromel%2Fpatchsmith/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thromel%2Fpatchsmith/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thromel","download_url":"https://codeload.github.com/thromel/patchsmith/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thromel%2Fpatchsmith/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34892547,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","benchmarking","deepagents","program-repair","software-maintenance"],"created_at":"2026-06-28T14:31:08.388Z","updated_at":"2026-06-28T14:31:09.157Z","avatar_url":"https://github.com/thromel.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PatchSmith\n\n[![CI](https://github.com/thromel/patchsmith/actions/workflows/ci.yml/badge.svg)](https://github.com/thromel/patchsmith/actions/workflows/ci.yml)\n\nPatchSmith is a research platform for evaluating AI software-maintenance agents.\n\nGive it a repository, an issue, and a test command. PatchSmith retrieves likely\ncontext, lets an agent propose a bounded patch, runs validation in a sandbox,\nand writes down the evidence: diffs, traces, stdout, stderr, timing, reports,\nand model-cost metadata when a live model is used.\n\nThe goal is not to pretend every passing test means \"the agent fixed it.\"\nPatchSmith is built to answer the more useful question: what happened, why did\nit happen, and can someone audit the repair attempt later?\n\n## Status\n\nPatchSmith is active R\u0026D code.\n\nThe seeded benchmark lane is the stable development path. It is useful for\ntesting retrieval, scaffold behavior, sandbox execution, reporting, and release\ngates. The public GitHub issue lane exists, but it is still calibration work.\nTreat public-issue repair results as experimental unless they come with a saved\nartifact directory that includes the repository state, issue spec, reproduction\ncommand, patch, validation output, and model metadata.\n\nCurrent public status is `ready_with_caveats`: the offline seeded-suite demo is\ncoherent, and live LLM calibration evidence exists for saved DeepAgents\n`gpt-5.4-mini` runs in that benchmark lane only. Do not generalize those results\nto public-issue repair quality without matching reproduction and validation\nartifacts.\n\n## What It Does\n\n- Clones or copies target repositories into controlled workspaces.\n- Indexes files, symbols, and issue text for context retrieval.\n- Supports native keyword, hybrid, graph, and `ctxhelm` context providers.\n- Runs repair attempts through deterministic baselines and the `deepagents`\n  runtime adapter.\n- Includes a native DeepAgents planner with file reads, todo state, a\n  skills-backed repair contract, structured patch output, a patch-review\n  subagent, and sandbox-feedback retries.\n- Applies model output through PatchSmith's own bounded text-replacement gate.\n- Runs local or Docker sandbox validation with command-policy checks.\n- Produces Markdown, JSON, HTML, trace, diff, stdout, stderr, timing, and cost\n  artifacts.\n- Ships local quality gates for tests, static checks, package build, release\n  hygiene, demo readiness, and artifact indexing.\n\n## Why This Exists\n\nMost coding-agent demos compress the whole story into one number: did the final\ntest pass?\n\nThat is too coarse for repair research. A run can fail because retrieval missed\nthe right file, the prompt scaffold asked the wrong thing, the model produced an\nunsafe edit, the reproduction command was weak, the sandbox was misconfigured,\nor the issue was never reproducible in the first place.\n\nPatchSmith keeps those pieces separate. It is meant for comparing repair systems\nwithout hiding setup failures, weak validation, or lucky patches.\n\n## What PatchSmith Is Not\n\n- It is not an autonomous GitHub issue fixer.\n- It is not a hosted repair service.\n- It is not proof that a focused test pass equals upstream acceptance.\n- It is not a leaderboard claim against SWE-agent, OpenHands, or coding models.\n- It is not a replacement for human review.\n\nThe model gets to propose. PatchSmith owns the mutation.\n\n## How It Works\n\n```text\nissue + repository + test command\n  -\u003e clone or copy repository\n  -\u003e index files and symbols\n  -\u003e retrieve candidate context\n  -\u003e run a repair runtime\n  -\u003e apply one bounded patch\n  -\u003e run policy-checked validation\n  -\u003e write reports, traces, logs, diffs, and metrics\n```\n\nPatchSmith does not let a model write freely into the repository. A runtime can\ninspect context and propose an edit. PatchSmith applies the final patch through\nits own gate and records the result.\n\n## Install\n\n```bash\ngit clone https://github.com/thromel/patchsmith.git\ncd patchsmith\n\npython3 -m venv .venv\n. .venv/bin/activate\npython -m pip install --upgrade pip\npython -m pip install -e \".[dev]\"\n```\n\nFor native DeepAgents experiments:\n\n```bash\npython -m pip install -e \".[dev,deepagents]\"\n```\n\n## Quickstart\n\nRun the test suite:\n\n```bash\nPYTHONPATH=src python -m pytest -q\n```\n\nRun PatchSmith as a terminal coding agent against the current repository:\n\n```bash\npatchsmith agent \"Fix the failing parser test\" \\\n  --test-command \"python -m pytest tests/test_parser.py -q\"\n```\n\nRun the canonical five-minute demo:\n\n```bash\npatchsmith demo seeded-logic-bug\n```\n\nThen inspect the generated run directory:\n\n```bash\npatchsmith inspect artifacts/demo/seeded_logic_bug/runs/\u003crun_id\u003e\n```\n\nThe demo writes `report.md`, `final.diff`, `traces.jsonl`, validation logs,\n`metadata.json`, and `context/selected_files.json`. A static preview lives in\n[`docs/sample_artifacts/seeded_logic_bug/`](docs/sample_artifacts/seeded_logic_bug/).\n\nPreflight the same command before spending on a model call:\n\n```bash\npatchsmith agent \"Fix the failing parser test\" \\\n  --test-command \"python -m pytest tests/test_parser.py -q\" \\\n  --preflight \\\n  --json\n```\n\nPreflight mode validates the task text, selected context/provider settings,\nDeepAgents response/token budget, native DeepAgents prompt/tool token headroom,\nreasoning-model token headroom, optional apply target, and whether\n`OPENAI_API_KEY` is set in the environment. It does not call a model.\n\nStart an interactive agent session:\n\n```bash\npatchsmith chat \\\n  --test-command \"python -m pytest tests/test_parser.py -q\"\n```\n\n`patchsmith agent --interactive` starts the same session shell. Inside chat,\nplain text is treated as a repair task in act mode, while `/mode plan` makes\nplain text run `/preflight` only until the pending planned task is approved\nwith `go ahead` or `/run`, discarded with `cancel plan`, or until `/mode act` is\nselected again. Slash commands keep the session keyboard-first. Obvious control\nphrases such as `what next?`, `show status`, `review diff`, `apply check`,\n`go ahead`, `cancel plan`, and `apply it` route to their matching slash\ncommands instead of starting the wrong operation:\n\nActual `patchsmith chat`, `patchsmith agent --interactive`, and one-shot\n`patchsmith agent` runs perform a live OpenAI model availability/auth preflight\nbefore starting DeepAgents. `/preflight` remains a no-model-call local\ndiagnostic. Use `--skip-model-preflight` only for offline harnesses or deliberate\nprovider experiments.\n\n- `/preflight \u003ctask\u003e` validates config without a model call.\n- `/mode [act|plan]` switches plain-text behavior between repair runs and\n  preflight-only planning.\n- `/run` executes the pending plan-mode task; `/run \u003ctask\u003e` executes an\n  explicit task immediately.\n- `/cancel [plan]` discards the pending plan-mode task without clearing the\n  rest of the session.\n- `/context add \u003cpath[#symbol]\u003e` pins a repo-relative file or symbol hint for\n  later runs.\n- `/context show`, `/context remove \u003cpath\u003e`, and `/context clear` inspect or\n  update those forced context hints.\n- `/model [id|clear]` shows, sets, or clears the DeepAgents model override for\n  later runs in the session.\n- `/budget [responses \u003cn\u003e|tokens \u003cn\u003e|set \u003cresponses\u003e \u003ctokens\u003e|clear]` shows or\n  changes the session response/token caps. Use `-1` to disable a cap.\n- `/doctor` checks local DeepAgents dependencies, API-key presence, model\n  selection, budget caps, native DeepAgents prompt/tool token headroom,\n  reasoning-model token headroom, and apply readiness before a run.\n- `/cost` summarizes tasks, runs, validation count, run errors, model calls,\n  responses, tokens, and estimated session cost from the transcript.\n- `/metrics` summarizes transcript-derived process metrics: manual preflights,\n  automatic run preflights, runs, validation rate, apply success rate, custom\n  command count, context/model/budget/permission updates, verify runs, diff\n  views/reviews, instruction and memory views, feedback views/updates, run\n  evidence views, timeline views, next recommendations, apply approvals/rejections/blocks,\n  checkpoints/restores, model tokens, and cost per validated run.\n- `/gate [validated|clean|reviewed|applied|cost \u003cusd\u003e]` evaluates the current\n  transcript against built-in evidence gates. `reviewed` requires a validated\n  clean run, deterministic diff review, no high-risk diff review, and a ready\n  `/apply check`.\n- `/trace` or `/evidence` summarizes the last run's report, trace, diff,\n  changed files, model usage, and cost without leaving chat.\n- `/verify [command]` reruns the configured or explicit allowed test command\n  through PatchSmith's sandbox command policy and records the result.\n- `/run \u003ctask\u003e` records the same readiness checks as `/preflight`, runs the\n  live model preflight supplied by the CLI, then starts the DeepAgents repair\n  loop.\n- `/status` shows session, model, budget, context, and last-run artifact state.\n- `/history` lists tasks from the current session.\n- `/timeline [n]` shows the recent transcript event trail for task, run, gate,\n  trace, checkpoint, feedback, hook, and config activity.\n- `/next` recommends the next evidence-backed action from the transcript, such\n  as approving or cancelling a pending plan-mode task, trace review, gating,\n  apply, checkpoint, export, breaking a repeated failure loop, or turning a\n  rejected apply decision into feedback before retrying.\n- `/sessions` lists resumable chat transcripts under the current artifacts\n  directory with usage, validation, error, and cost summaries.\n- `/commands` lists project custom slash commands loaded from\n  `.patchsmith/commands/*.md`.\n- `/hooks` lists project lifecycle hooks loaded from `.patchsmith/hooks.json`.\n- `/agents` lists project agent profiles loaded from `.patchsmith/agents/*.md`.\n- `/agent [name|clear]` shows, selects, or clears the active project agent\n  profile for later runs.\n- `/instructions [show|reload|clear]` shows, reloads, or disables\n  AGENTS.md/CLAUDE.md-style project instructions for later runs.\n- `/memory [show|reload|clear]` shows the same loaded project memory through a\n  Claude Code-style command and records separate memory-view transcript\n  evidence.\n- `/plan [show|set|add|start|done|block|skip|pending|clear]` manages a\n  transcripted session plan. Active plan items are included as context for\n  later runs.\n- `/feedback \u003ctext\u003e`, `/feedback show`, and `/feedback clear` capture\n  human-in-the-loop guidance for later runs without changing the base task.\n- `/checkpoint [label]`, `/checkpoints`, and `/restore \u003cid|label\u003e` save, list,\n  and restore chat state without rewriting the transcript.\n- `/permissions [show|apply auto|apply manual|dirty allow|dirty deny]` shows\n  or changes apply intent and whether dirty worktrees are accepted by apply\n  operations. In interactive chat, `apply auto` is artifact-first: after a run,\n  PatchSmith records `apply_auto_deferred` and requires `/diff review`,\n  `/apply check`, and `/apply` instead of mutating immediately.\n- `/approve apply \u003creason\u003e` records an explicit human approval for the current\n  reviewed diff after `/apply check`; it is required before applying high-risk\n  reviewed diffs.\n- `/reject apply \u003creason\u003e` records an explicit human denial for the current\n  reviewed diff after `/apply check`; `/apply` stays blocked until a newer\n  approval supersedes it, and `/next` recommends feeding the denial back into a\n  retry instead of applying against the transcript.\n- `/compact [note]` records a deterministic transcript summary and clears the\n  in-memory task history while preserving last-run artifact pointers.\n- `/clear` clears in-memory task, run, and apply state while retaining the\n  transcript for audit and resume.\n- `/export [path]` writes a Markdown session report from the JSONL transcript.\n- `/diff` shows the last generated diff path, `/diff stat` summarizes files\n  and line counts, `/diff show [lines]` prints a bounded diff preview, and\n  `/diff review` runs deterministic patch-risk checks before apply.\n- `/apply check` runs the same local repo, dirty-worktree, empty-diff, and\n  `git apply --check` safeguards without mutating the working tree.\n- `/apply` applies the last generated diff only after the current transcript has\n  a non-high-risk `/diff review` and a ready `/apply check`; it then runs the\n  same safe `git apply` checks used by `patchsmith agent --apply`. High-risk\n  reviews require an explicit `/approve apply \u003creason\u003e` event after the ready\n  apply check before mutation is allowed; a newer `/reject apply \u003creason\u003e`\n  blocks mutation until superseded by a newer approval.\n- `/rewind` or `/undo` reverses the last generated diff through\n  `git apply --reverse --check` and then `git apply --reverse`.\n- `/exit` or `/quit` ends the session.\n\nProject custom commands are Markdown prompt templates under\n`.patchsmith/commands/`. The filename becomes the slash command name, and\nsubdirectories become namespaces: `.patchsmith/commands/review.md` maps to\n`/review`, while `.patchsmith/commands/bench/live.md` maps to `/bench:live`.\nOptional frontmatter fields such as `description` and `argument_hint` appear in\n`/commands` and `--list-commands --json`. PatchSmith substitutes `$ARGUMENTS`,\n`{{arguments}}`, or `{{ args }}` with the text after the slash command, records\nthe command source in the transcript, and then runs the rendered prompt through\nthe same DeepAgents runner, trace, diff, cost, and apply machinery as `/run`.\n\nProject agent profiles are Markdown specialist definitions under\n`.patchsmith/agents/`. They are PatchSmith's lightweight version of file-based\nsubagents: the profile can set model/budget/context defaults and prepend\nspecialist instructions to the next bounded repair prompt, but it still runs\nthrough the same safe runner and transcript path.\n\n```markdown\n---\ndescription: Verification-focused repair mode\nmodel: gpt-5-mini\nsubagents: inline\nmax_context_files: 3\nmax_model_responses: 4\nmax_model_tokens: 90000\ntest_command: pytest tests/test_parser.py -q\ncontext_paths: |\n  - src/parser.py#parse\n  - tests/test_parser.py\n---\n\nLocalize the failure before editing.\nReject broad rewrites.\n```\n\nUse `/agents` to list profiles and `/agent verifier` to select one inside\nchat. One-shot runs can apply the same profile with `--agent-profile verifier`.\nSelecting a profile is transcripted, replayed on resume, included in runtime\nmetadata, and counted in `/metrics`.\n\nPatchSmith automatically loads concise project instruction files from the repo\nroot before a chat or one-shot agent run:\n\n- `AGENTS.md`\n- `CLAUDE.md`\n- `CODEX.md`\n- `GEMINI.md`\n- `.cursorrules`\n- `.patchsmith/instructions.md`\n\nThe loaded instruction context is size-capped, listed by `/instructions`, stored\nin the session transcript, and included in runtime metadata. Extra repo-relative\nfiles can be added with `--instruction-path`, and automatic instruction loading\ncan be disabled with `--no-agent-instructions`. These files guide the repair\nprompt, but they do not override PatchSmith's safe runner, patch safety checks,\nhooks, validation, or apply policy.\n\nSession plans are lightweight task lists for complex work:\n\n```text\n/plan set inspect parser; write focused test; run validation\n/plan start 1\n/run Fix the parser edge case\n/plan done 1\n/plan show\n```\n\nPlans use explicit statuses: `pending`, `in_progress`, `completed`, `blocked`,\nand `skipped`. PatchSmith stores plan changes in the transcript, restores them\non resume, includes active plan state in the next repair prompt, and counts plan\nviews/updates in `/metrics`.\n\nRewind is PatchSmith's lightweight checkpoint path for applied agent patches.\nIt is intentionally diff-scoped: `/rewind` reverses the last generated diff\ninstead of resetting the whole repository. That preserves unrelated worktree\nstate while still giving the user a quick way to back out a bad applied patch.\nRewind attempts and successes are transcripted and included in `/metrics`.\n\nSession gates let you check whether an interactive session is ready to promote\nwithout leaving the shell:\n\n```text\n/gate validated\n/gate clean\n/gate reviewed\n/gate applied\n/gate cost 0.05\n```\n\n`validated` requires at least one validated run. `clean` additionally requires\n100% validation rate and zero run errors. `reviewed` adds deterministic diff\nrisk-review evidence, no high-risk diff reviews, and a ready non-mutating apply\ncheck. `applied` requires successful apply attempts. `cost \u003cusd\u003e` caps cost per\nvalidated run. Gate results are recorded in the transcript and counted in\n`/metrics`; saved sessions can still be gated offline with\n`patchsmith chat --session-gate`.\n\nProject hooks are deterministic shell commands configured in\n`.patchsmith/hooks.json`. PatchSmith passes a JSON envelope on stdin and records\nhook outcomes in the chat transcript. Blocking hooks fail closed before\nexpensive or risky actions such as model runs and patch application:\n\n```json\n{\n  \"hooks\": {\n    \"PreRun\": [\n      {\n        \"name\": \"budget-guard\",\n        \"matcher\": \"benchmark|live\",\n        \"command\": \"python scripts/check_budget.py\",\n        \"timeout_seconds\": 10\n      }\n    ],\n    \"PreApply\": [\n      {\n        \"name\": \"review-before-apply\",\n        \"command\": \"python scripts/check_diff_policy.py\"\n      }\n    ]\n  }\n}\n```\n\nHook commands can print `{\"decision\": \"block\", \"reason\": \"...\"}` to stdout or\nexit non-zero to stop the lifecycle event. Supported chat lifecycle events are\n`SessionStart`, `SessionEnd`, `UserPromptSubmit`, `UserPromptExpansion`,\n`PreRun`, `PostRun`, `PreApply`, and `PostApply`. `PreRun` and `PreApply` are\nthe main safety gates; post-event hooks are recorded as evidence but cannot\nundo completed work.\n\nChat sessions write JSONL transcripts under `artifacts/chat_sessions/` so runs,\nmanual preflights, automatic run preflights, context changes, usage/cost\nfields, apply decisions, and follow-up tasks remain auditable.\n\nResume a saved chat transcript by session id:\n\n```bash\npatchsmith chat --resume 20260614T195042Z-4be8450e\n```\n\nList saved chat sessions without starting the shell:\n\n```bash\npatchsmith chat --list-sessions\n```\n\nList project custom commands without starting the shell:\n\n```bash\npatchsmith chat --list-commands\npatchsmith chat --list-commands --json\n```\n\nList project agent profiles without starting the shell:\n\n```bash\npatchsmith chat --list-agents\npatchsmith chat --list-agents --json\n```\n\nList project instruction files without starting the shell:\n\n```bash\npatchsmith chat --list-instructions\npatchsmith chat --list-instructions --json\n```\n\nList project hooks without starting the shell:\n\n```bash\npatchsmith chat --list-hooks\npatchsmith chat --list-hooks --json\n```\n\nPrint saved session metrics without starting the shell:\n\n```bash\npatchsmith chat --session-metrics 20260614T195042Z-4be8450e --json\n```\n\nPrint the deterministic next recommendation for a saved session:\n\n```bash\npatchsmith chat --session-next 20260614T195042Z-4be8450e --json\n```\n\nGate a saved session for CI or benchmark promotion:\n\n```bash\npatchsmith chat \\\n  --session-gate 20260614T195042Z-4be8450e \\\n  --require-validated-run \\\n  --min-validation-rate 1.0 \\\n  --require-diff-review \\\n  --max-high-risk-diff-reviews 0 \\\n  --require-ready-apply-check \\\n  --max-cost-per-validated-run-usd 0.05 \\\n  --max-run-errors 0\n```\n\nExport a saved transcript without resuming the shell:\n\n```bash\npatchsmith chat \\\n  --export-session 20260614T195042Z-4be8450e \\\n  --export-path artifacts/chat_sessions/20260614T195042Z-4be8450e.md\n```\n\nRun a scripted chat session:\n\n```bash\npatchsmith chat --script scripts/live-smoke.patchsmith\n```\n\nThe script file contains the same lines you would type interactively, such as\n`/doctor`, `/model gpt-5-mini`, `/budget set 4 60000`, `/review target`, or\n`/exit`. This is useful for repeatable smoke runs and CI-friendly dry runs;\ncommands in the script still use the same chat transcript, preflight, runner,\ntrace, diff, cost, and apply boundaries.\n\nThe `agent` command defaults to:\n\n- `--repo .`\n- `--runtime deepagents`\n- `--planner deepagents`\n- `--context-provider native_hybrid`\n- `--deepagents-subagents auto`\n- `--max-model-responses 12`\n- `--max-model-tokens 200000`\n\nIt writes the proposed patch, trace, validation output, and report under\n`artifacts/runs/\u003crun_id\u003e/`. This is intentionally artifact-first: the current\nCLI does not mutate the user's working tree unless asked.\n\nTo apply the generated diff back to a clean local Git worktree:\n\n```bash\npatchsmith agent \"Fix the failing parser test\" \\\n  --test-command \"python -m pytest tests/test_parser.py -q\" \\\n  --apply\n```\n\n`--apply` is intentionally explicit. It requires `--repo` to be a local Git\nrepository, rejects dirty worktrees by default, checks the generated diff with\n`git apply --check`, and then applies it with `git apply`. Inside interactive\nchat, `/apply` is stricter: it first requires the session to record\nnon-high-risk `/diff review` evidence and a ready `/apply check`. Pass\n`--allow-dirty-apply` only when you intentionally want to apply into a worktree\nthat already has uncommitted changes.\n\nRun a deterministic repair on a seeded bug:\n\n```bash\nPYTHONPATH=src python -m patchsmith.cli run \\\n  --repo evals/tasks/seeded_bugs_v1/task_001_logic_bug/repo \\\n  --issue-file evals/tasks/seeded_bugs_v1/task_001_logic_bug/issue.md \\\n  --test-command \"python3 -m pytest\" \\\n  --runtime heuristic \\\n  --context-provider native_hybrid \\\n  --artifacts-dir artifacts \\\n  --json\n```\n\nExpected behavior: PatchSmith edits the seeded task repository, runs the pytest\ncommand, and writes a run report under `artifacts/runs/\u003crun_id\u003e/`.\n\nIf you already know a likely file, force it into the context:\n\n```bash\nPYTHONPATH=src python -m patchsmith.cli run \\\n  --repo path/to/repo \\\n  --issue-file path/to/issue.md \\\n  --context-provider native_hybrid \\\n  --context-path \"src/package/module.py#suspected_symbol\" \\\n  --json\n```\n\n`--context-path` can be repeated. PatchSmith strips the optional `#symbol`\nsuffix before reading the file, but keeps the full hint in the issue text for\nrepair flows that provide reviewed hints.\n\n## DeepAgents\n\nPatchSmith has two DeepAgents modes:\n\n- `runtime=deepagents, planner=heuristic`: adapter and scaffold compatibility.\n- `runtime=deepagents, planner=deepagents`: native DeepAgents planning with a\n  live OpenAI-compatible chat model.\n\nThe native planner seeds DeepAgents with a state-backed virtual filesystem,\nread-only file permissions, a compact `/.patchsmith/repair-interface.md`\nrun manifest, a `/.patchsmith/acceptance-rubric.md` verifier checklist, a\n`/.patchsmith/repo-instructions.md` scoped repository-instruction manifest, a\nPatchSmith repair skill, a durable memory file, structured `PatchPlan` output,\na `failure-localizer` subagent, and a `patch-reviewer` subagent for ambiguous\nor feedback-driven repairs. The repair interface is the\nagent-computer interface for a run: it lists required manifests, mounted source\npaths, subagent routing mode, and output constraints before the model explores\nsource files. The acceptance rubric is generated from the issue, mounted files,\npreferred targets, validation fixtures, and unsafe-patch exclusions; it gives\nthe model a task-local checklist to verify the selected path, old span,\nvalidation claim, and patch shape before final output. Repo instructions are\ndiscovered only from AGENTS.md-style files at the repository root and ancestors\nof mounted context paths, capped before mounting, and framed as scoped\nconstraints rather than permission for broad exploration. Native\nDeepAgents plans must also include compact localization fields that name the\nfailing runtime mechanism and justify why the selected file/span controls it.\nFor cost experiments, `PATCHSMITH_DEEPAGENTS_SUBAGENTS=inline` disables the\nDeepAgents subagents and requires the main planner to do localization and review\ninline. Keep this mode experimental until saved live artifacts show lower\nresponse count, token use, and cost without reducing validation.\n`PATCHSMITH_DEEPAGENTS_SUBAGENTS=auto` is the safer ablation: it keeps\nsubagents for retries, reviewed source hints, validation fixtures, and\nmulti-context repairs, but disables them for simple single-control-point runs.\nThe default remains `full`.\n\nPatchSmith only requests `reasoning.encrypted_content` for OpenAI model ids that\nare expected to support reasoning items, such as `gpt-5*`, `o1*`, `o3*`, and\n`o4*`. Non-reasoning models such as `gpt-4.1-mini` still use the Responses API\nwithout that include flag, avoiding provider-side compatibility failures. Set\n`PATCHSMITH_DEEPAGENTS_ENCRYPTED_REASONING=enabled` or\n`PATCHSMITH_DEEPAGENTS_ENCRYPTED_REASONING=off` only for explicit provider\ncapability experiments; the default `auto` mode is recorded in DeepAgents\ncontract metadata.\n\nPreflight a model before spending money:\n\n```bash\nOPENAI_API_KEY=... \\\nPYTHONPATH=src python -m patchsmith.cli openai-model-preflight \\\n  --model \u003cmodel\u003e \\\n  --json\n```\n\nRun the native DeepAgents planner on the seeded benchmark:\n\n```bash\nOPENAI_API_KEY=... \\\nPATCHSMITH_DEEPAGENTS_MODEL=\u003cmodel\u003e \\\nPYTHONPATH=src python -m patchsmith.cli eval-repair \\\n  --dataset evals/tasks/seeded_bugs_v1 \\\n  --runtime deepagents \\\n  --planner deepagents \\\n  --max-retries 1 \\\n  --max-tasks 10 \\\n  --context-provider native_hybrid \\\n  --output artifacts/experiments/deepagents_native_repair_eval_v1 \\\n  --json\n```\n\nDo not cite model performance from a README command. Use the saved artifact\ndirectory for the exact run: model name, account, prompt, dataset, commit, diff,\nlogs, and validation output all matter.\nKeep `--max-tasks` set on live runs unless you are intentionally expanding the\nbenchmark and budget.\n\nOptional context-budget experiments:\n\n```bash\nPATCHSMITH_DEEPAGENTS_CONTEXT_MODE=span \\\nPATCHSMITH_DEEPAGENTS_CONTEXT_WINDOW_LINES=80 \\\nPATCHSMITH_DEEPAGENTS_SUBAGENTS=auto \\\nPYTHONPATH=src python -m patchsmith.cli execute-public-issue-repairs \\\n  --deepagents-max-context-files 2 ...\n```\n\n`--deepagents-max-context-files` limits the number of repository files mounted\ninto the DeepAgents virtual filesystem while preserving reviewed source hints,\nvalidation fixtures, and strong target-localization signals such as\nsymbol-qualified control points first. The default `0` keeps the full retrieved\ncontext because smaller mounts can make the model ask for more reasoning tokens;\nuse this knob only when the saved trace and suite gate prove a net improvement.\n`PATCHSMITH_DEEPAGENTS_CONTEXT_MODE=span` is a separate first-attempt\ncompression lane: mounted repository paths stay stable, but each mounted source\nfile is narrowed to a focused line window around matched symbols, runtime-cache\ncues, or reviewed source hints. Keep it opt-in until a saved live suite shows\nlower tokens without losing target alignment or validation.\nComplex benchmark\nreports include DeepAgents virtual-file count, context-cap usage, tokens, and\ncost so these experiments can be compared from saved artifacts. They also\nreport repair-interface and acceptance-rubric manifest tasks plus read-first\nrates, and repo-instructions manifest tasks plus read-first rate, so\ninterface/verifier/context changes are visible in benchmark summaries instead\nof requiring manual trace inspection. Trajectory reports keep the legacy agent\ntrajectory score stable and expose contextual-verifier coverage as a separate\nrate, so verifier adoption can be compared without silently moving older score\nthresholds. Use `min_contextual_verifier_rate` or\n`--min-contextual-verifier-rate` when a complex suite must prove verifier\ncoverage from saved traces; `refresh-evidence` exposes the same gate as\n`--complex-suite-min-contextual-verifier-rate`. Use\n`min_repo_instructions_manifest_rate` and\n`min_repo_instructions_read_first_rate` when a context-policy lane must prove\nscoped AGENTS.md-style repository guidance was mounted and read before source\nedits. Use\n`evals/issue_corpora/public_issue_smoke_v1/complex_suite_verifier.template.json`\nfor the next rubric-enabled live lane; the older `complex_suite.template.json`\nremains the historical pre-rubric baseline.\nPublic issue repair summaries and rows also record actual model calls, tokens,\nand estimated cost; rows that exceed configured post-run live-cost,\nresponse-count, or token-count caps after execution are not counted as\nvalidated claims.\nUse `--max-actual-model-responses` and `--max-actual-model-tokens` on live\npublic-issue runs when a benchmark lane needs hard claim limits for DeepAgents'\ninternal call volume. PatchSmith mounts those limits into\n`/.patchsmith/repair-interface.md` as a resource budget so the agent sees the\nclaim boundary before exploring source, and the native DeepAgents planner now\ninstalls an active response-budget callback that blocks the next model call once\nthe configured response count is exhausted. Token caps are still checked from\nrecorded provider usage after each response, so token overages remain failed\nclaims even when the final patch passes tests. Reasoning-model runs can spend\ntens of thousands of tokens before producing visible output, so calibrate\ninitial token caps from a small preflight lane before treating low caps as\nproduct defaults. For response ceilings of six or\nfewer, PatchSmith also switches the repair interface into budget-critical mode:\ngeneric memory/skill reads are no longer required, the first preferred\nsource/symbol is mounted as a compact Fast Patch Packet, and the prompt asks the\nagent to return a structured `PatchPlan` as soon as the controlling branch is\nclear. Add `--deepagents-subagents auto` for budgeted calibration lanes; it\nkeeps retries eligible for subagents but uses compact inline\nlocalization/review on the first attempt.\nThe subagent mode is reported in the DeepAgents contract as `subagent_mode`\nwith a `subagent_routing` reason list. The same contract records\n`repair_interface_manifest_path` and `repair_interface_manifest_read_first` so\nsaved artifacts prove which run interface the model received. Use `auto` or\n`inline` only for side-by-side calibration runs, not as default performance\nclaims.\n\n## Evaluation Commands\n\nValidate the seeded dataset:\n\n```bash\nPYTHONPATH=src python -m patchsmith.cli validate-dataset \\\n  --dataset evals/tasks/seeded_bugs_v1 \\\n  --output artifacts/experiments/seeded_dataset_validation_v1 \\\n  --json\n```\n\nCompare retrieval providers:\n\n```bash\nPYTHONPATH=src python -m patchsmith.cli eval-retrieval \\\n  --dataset evals/tasks/seeded_bugs_v1 \\\n  --context-provider native \\\n  --context-provider native_hybrid \\\n  --context-provider native_graph \\\n  --context-provider ctxhelm_cli \\\n  --output artifacts/experiments/retrieval_eval_v1 \\\n  --json\n```\n\nRun a DeepAgents compatibility repair benchmark:\n\n```bash\nPYTHONPATH=src python -m patchsmith.cli eval-repair \\\n  --dataset evals/tasks/seeded_bugs_v1 \\\n  --runtime deepagents \\\n  --planner heuristic \\\n  --context-provider native_hybrid \\\n  --output artifacts/experiments/deepagents_compatibility_repair_eval_v1 \\\n  --json\n```\n\nCompare scaffold variants:\n\n```bash\nPYTHONPATH=src python -m patchsmith.cli eval-scaffold \\\n  --dataset evals/tasks/seeded_bugs_v1 \\\n  --variant agentless \\\n  --variant heuristic \\\n  --variant deepagents \\\n  --context-provider native_hybrid \\\n  --output artifacts/experiments/scaffold_comparison_v1 \\\n  --json\n```\n\nSummarize a completed public-issue repair lane as a complex benchmark:\n\n```bash\nPYTHONPATH=src python -m patchsmith.cli eval-complex \\\n  --attempt-dir artifacts/experiments/public_issue_corpus_v1 \\\n  --benchmark public_issue_smoke_v1_latest_all \\\n  --output artifacts/experiments/complex_deepagents_public_issue_smoke_v1_latest_all \\\n  --json\n```\n\n`eval-complex` reads saved repair-attempt artifacts, traces, and reports. It\ndoes not run repositories, execute tests, or call a model provider.\n\nAggregate multiple saved public-issue repair lanes into a gated complex suite:\n\n```bash\nPYTHONPATH=src python -m patchsmith.cli eval-complex-suite \\\n  --suite-spec evals/issue_corpora/public_issue_smoke_v1/complex_suite.template.json \\\n  --validate-only \\\n  --json\n\nPYTHONPATH=src python -m patchsmith.cli eval-complex-suite \\\n  --suite-spec evals/issue_corpora/public_issue_smoke_v1/complex_suite.template.json \\\n  --json\n```\n\nThe suite spec contains the saved attempt directories, output directory, and\ngate thresholds. Selected cost/token/response caps measure the chosen best\nattempt per task; attempted cost/token/response caps measure total spend across\nall evaluated attempts, so exploratory changes that validate but waste model\nbudget or internal DeepAgents calls still trip the suite gate. Max\nattempted/selected task caps catch single-task cost, token, or response-count\noutliers that an aggregate average can hide. Suites can also set\n`min_target_alignment_rate` to require final patches to stay inside paths\nlocalized by explicit target candidates or by DeepAgents' structured\nfailure-localization rationale for the selected patch plan. Complex summaries\nalso report selected-attempt context-efficiency proxies: virtual files, virtual\nfiles per validated task, tokens per virtual file, and responses per virtual\nfile. They also report selected context-target recall and precision whenever\nthe saved trace includes both localized target paths and DeepAgents mounted\nsource paths. Suite gates can cap the proxy metrics with\n`max_selected_virtual_files_per_validated_task`,\n`max_selected_tokens_per_virtual_file`, and\n`max_selected_responses_per_virtual_file`, and can require target/context\ncoverage with `min_selected_context_target_recall` and\n`min_selected_context_target_precision`. Complex reports also expose a\ntrace-derived progress score so failed long-horizon attempts can distinguish\nreproduction-only, patch-generated, target-aligned, quality-warning, and clean\nvalidated stages. Use `min_selected_progress_score` when a suite must enforce\nthat retained attempts reached a minimum partial-progress floor. They also emit\ndeterministic failure-class counts for triage buckets such as validated,\nquality-risk, preflight-blocked, reproduction-failed, no-patch,\ntarget-misaligned, runtime/tool failure, retry-exhausted, and validation-failed;\nthese labels are artifact-derived benchmark signals, not human root-cause\nannotations. Complex reports also aggregate a HarnessFix-style\n`harness_layer` label so failed attempts can be triaged by the implicated\nlayer: budget, model, sandbox, preflight, reproduction, planning, context,\npatch quality, retry, runtime, validation, or orchestration. DeepAgents feedback retries also carry a narrower runtime\n`retry_failure_class` in the retry artifact and trace payload so the next\nattempt sees whether it is handling validation failure, safety-gate rejection,\nquality risk, repeated-target failure, or missing validation before editing.\nComplex summaries aggregate those retry classes as `retry_failure_class_counts`\nso retry-policy experiments can be compared from saved traces. They also emit\nprocess-quality labels and flags derived from the same trace. A validated patch\ncan still be marked process-risky when the trace lacks verification, uses an\nunclassified retry, churns through repeated failed events, or edits again after\nsuccessful verification; use `process_quality_label_counts`,\n`process_quality_flag_counts`, and `process_risky_validated_tasks` when\nchecking for AgentLens-style lucky-pass risk.\nSuites can enforce those process diagnostics with `min_process_quality_score`\nand `max_process_risky_validated_tasks`.\nVerifier lanes can also require the\ntask-local acceptance rubric to be mounted and read before final output with\n`min_acceptance_rubric_manifest_rate` and\n`min_acceptance_rubric_read_first_rate`. They can also require the selected\npatch to satisfy deterministic rubric-alignment proxies with\n`min_acceptance_rubric_alignment_rate`, which checks that the patch was rubric\nbacked, target-aligned, mounted-context bounded, and free of patch-quality\nwarnings. The `refresh-evidence` CLI exposes the\nsame gates as `--complex-suite-min-acceptance-rubric-manifest-rate` and\n`--complex-suite-min-acceptance-rubric-read-first-rate`, plus\n`--complex-suite-min-acceptance-rubric-alignment-rate`.\n`refresh-evidence --complex-suite-spec ...` can regenerate the same suite report\nand `complex_benchmark_suite_gate.json` from saved attempt artifacts without\nspending live-model tokens. The validation-only command checks the suite\ninterface first: attempt directories, required result files, output path, and\ngate-threshold count. When verifier or acceptance-rubric gates are requested\nand older saved attempts lack that evidence, the suite also emits\n`verifier_contract_rerun` follow-up candidates and a shell-ready runbook for\nrubric-backed DeepAgents reruns.\n\nBuild an artifact index:\n\n```bash\nPYTHONPATH=src python -m patchsmith.cli index-artifacts \\\n  --artifacts-dir artifacts \\\n  --output artifacts/experiments/index.md \\\n  --json-output artifacts/experiments/index.json \\\n  --html-output artifacts/experiments/index.html \\\n  --run-detail-output-dir artifacts/experiments/run-details \\\n  --json\n```\n\n## Public Issue Corpus\n\nThe public issue smoke lane lives under\n`evals/issue_corpora/public_issue_smoke_v1`.\n\nIt has separate gates for corpus validation, repository preflight, source-free\ncontext preview, task materialization, focused test planning, setup validation,\nreproduction evidence, repair readiness, and repair attempts.\n\nThat separation is intentional. A public issue repair should only count after\nthe failing behavior is reproduced, a patch is generated, and validation passes.\nPassing setup checks are useful evidence, but they do not prove repair quality.\n\n## Docker Sandbox\n\nBuild the seeded smoke image:\n\n```bash\ndocker build -f docker/seeded-smoke.Dockerfile -t patchsmith-seeded-smoke:py312 .\n```\n\nRun with Docker isolation:\n\n```bash\nPYTHONPATH=src python -m patchsmith.cli run \\\n  --repo evals/tasks/seeded_bugs_v1/task_001_logic_bug/repo \\\n  --issue-file evals/tasks/seeded_bugs_v1/task_001_logic_bug/issue.md \\\n  --test-command \"python3 -m pytest\" \\\n  --runtime heuristic \\\n  --context-provider native_hybrid \\\n  --sandbox-mode docker \\\n  --sandbox-image patchsmith-seeded-smoke:py312 \\\n  --artifacts-dir artifacts \\\n  --json\n```\n\nDocker mode disables implicit image pulls, disables network by default, drops\ncapabilities, mounts the repository at `/workspace`, applies resource limits,\nand records the selected sandbox in the trace.\n\n## Quality Gate\n\nRun the local release gate:\n\n```bash\nPYTHONPATH=src python -m patchsmith.cli quality-gate \\\n  --project-root . \\\n  --artifacts-dir artifacts \\\n  --output artifacts/experiments/quality_gate.md \\\n  --json-output artifacts/experiments/quality_gate.json \\\n  --logs-dir artifacts/experiments/quality_gate_logs \\\n  --json\n```\n\nThe gate runs compile checks, whitespace checks, the full pytest suite, and a\npackage build. CI also runs Ruff, Ruff format check, mypy, compile checks,\npytest, and package build.\n\n## Repository Layout\n\n```text\nsrc/patchsmith/\n  cli/                     CLI commands\n  evaluation/              seeded and public-issue evaluation flows\n  observability/           artifact index, failure reports, renderers\n  portfolio/               readiness, release, and demo reports\n  runtime/                 agent runtime adapters\n  deepagents_planner.py    native DeepAgents planner\n  deepagents_prompts.py    native DeepAgents prompts, memory, and skill text\n  deepagents_schema.py     native DeepAgents structured output schema\n  retrieval.py             native retrieval providers\n  sandbox.py               local and Docker command execution\n\ndocs/                       architecture, safety, evaluation, and runbook docs\nevals/                      seeded tasks and public issue corpora\nadr/                        architecture decision records\nexperiments/                experiment plans\ntemplates/                  report and ADR templates\n```\n\n## Good First Commands\n\n```bash\nPYTHONPATH=src python -m pytest -q\nPYTHONPATH=src python -m patchsmith.cli demo seeded-logic-bug\nPYTHONPATH=src python -m patchsmith.cli quality-gate --json\nPYTHONPATH=src python -m patchsmith.cli project-status --json\nPYTHONPATH=src python -m patchsmith.cli demo-readiness --json\n```\n\n## Public Evidence Docs\n\n- [Artifact gallery](docs/artifact_gallery.md)\n- [Failure taxonomy](docs/failure_taxonomy.md)\n- [Benchmark manifest schema](docs/benchmark_manifest_schema.md)\n- [Runtime fairness checks](docs/runtime_fairness_checks.md)\n- [Comparison matrix](docs/comparison_matrix.md)\n- [Changelog](CHANGELOG.md)\n\n## Roadmap\n\n- Calibrate live DeepAgents runs against the seeded benchmark.\n- Expand public-issue reproduction coverage without treating setup success as\n  repair success.\n- Add more model/provider cost accounting.\n- Improve artifact comparison for retrieval, patch quality, retries, and\n  validation strength.\n- Keep the runtime adapters small enough to read and test.\n\n## License\n\nMIT. See [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthromel%2Fpatchsmith","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthromel%2Fpatchsmith","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthromel%2Fpatchsmith/lists"}