{"id":51524787,"url":"https://github.com/ahrefs/reviewotron","last_synced_at":"2026-07-08T20:01:15.938Z","repository":{"id":359151259,"uuid":"1210635095","full_name":"ahrefs/reviewotron","owner":"ahrefs","description":"automated review bot","archived":false,"fork":false,"pushed_at":"2026-07-01T09:52:07.000Z","size":2178,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-01T11:25:51.364Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ahrefs.png","metadata":{"files":{"readme":"docs/README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"docs/security_debug_signals_proof_corpus_metrics.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-04-14T15:55:08.000Z","updated_at":"2026-07-01T09:52:11.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ahrefs/reviewotron","commit_stats":null,"previous_names":["ahrefs/reviewotron"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ahrefs/reviewotron","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Freviewotron","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Freviewotron/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Freviewotron/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Freviewotron/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ahrefs","download_url":"https://codeload.github.com/ahrefs/reviewotron/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Freviewotron/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35276781,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-08T02:00:06.796Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-08T20:01:13.817Z","updated_at":"2026-07-08T20:01:15.920Z","avatar_url":"https://github.com/ahrefs.png","language":"OCaml","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reviewotron\n\nAn agentic code review bot that uses Claude AI to review GitHub pull requests and push events. It posts inline review comments on PRs, commit comments on pushes to `develop`, and sends Slack notifications.\n\nReviewotron includes a **multi-agent security analysis pipeline** that detects injection, XSS, command injection, authentication, authorization, and SSRF vulnerabilities. Security findings go through adversarial validation before being reported, keeping noise low.\n\n## Table of Contents\n\n- [How It Works](#how-it-works)\n- [Agent Helper Mode](#agent-helper-mode)\n- [Setup](#setup)\n- [Configuration](#configuration)\n- [Security Review Pipeline](#security-review-pipeline)\n- [Slack Integration](#slack-integration)\n- [State and Persistence](#state-and-persistence)\n- [Review Feedback](#review-feedback)\n- [CLI Usage](#cli-usage)\n- [Cost Tracking](#cost-tracking)\n- [Limitations](#limitations)\n- [Known Issues](#known-issues)\n- [Troubleshooting](#troubleshooting)\n\n---\n\n## How It Works\n\nReviewotron runs as an HTTP server that receives GitHub webhook events. It can review on PR open/update, on pushes to `develop`, or when someone posts a `REVIEW` comment on a PR. **All triggers are off by default** — see [Defaults](#defaults) below.\n\nFor each enabled trigger, the bot:\n\n1. **Receives the webhook** at the `/github` endpoint\n2. **Validates the signature** using the configured webhook secret (HMAC-SHA256)\n3. **Fetches the repo config** from `.reviewotron.json` in the repo (via GitHub API), or uses defaults\n4. **Fetches the diff** for the PR or push (for `REVIEW` comments, also fetches the full PR via the API to recover `head.sha`, since `issue_comment` webhooks don't carry it)\n5. **Filters the diff** — removes ignored paths, checks size limits\n6. **Runs review plugins** concurrently:\n   - **General review** — Claude analyzes the diff for bugs, style, logic, performance, etc.\n   - **Security review** — A multi-agent pipeline scans for vulnerabilities (see [below](#security-review-pipeline))\n7. **Posts results**:\n   - PR events: a single GitHub PR review with inline comments when findings or errors exist\n   - Push events: commit comments for critical/warning findings + a Slack message\n   - `REVIEW` comments: same as PR events\n\n### Event Flow\n\n```\nGitHub Webhook (POST /github)\n    │\n    ├─ Signature validation (HMAC-SHA256)\n    ├─ Event parsing (pull_request, push, or issue_comment)\n    ├─ Config fetch from .reviewotron.json\n    ├─ Diff fetch + filtering\n    │\n    ├─ General Review Plugin (Claude Sonnet)\n    │     └─ Structured output: summary + findings\n    │\n    ├─ Security Review Plugin (multi-agent)\n    │     ├─ Triage Agent (Haiku) → route signals\n    │     ├─ Analysis Agents (Sonnet, parallel) → candidate findings\n    │     ├─ Validator Agent (Sonnet) → confirm/reject\n    │     └─ Memory Curator (Haiku, async) → update memory\n    │\n    ├─ Merge + deduplicate findings\n    │\n    └─ Post results\n          ├─ PR → GitHub PR review when there is something to report\n          └─ Push → commit comments + Slack notification\n```\n\n### Supported GitHub Events\n\n| Event | Trigger | Gated by | Output |\n|-------|---------|----------|--------|\n| `pull_request` (opened, reopened, ready_for_review) | PR opened, reopened, or marked ready | `auto_review_pr_open` | GitHub PR review with inline comments when there is something to report |\n| `pull_request` (synchronize) | New commits pushed to a PR | `auto_review_pr_sync` | GitHub PR review with inline comments when there is something to report |\n| `push` (to `refs/heads/develop`) | Code pushed to develop | `review_pushes_to_develop` | Commit comments + Slack message |\n| `issue_comment` (created, on a PR, body equals `REVIEW`) | Manual trigger via PR comment | `auto_review_on_comment` | GitHub PR review with inline comments when there is something to report |\n\nThe `REVIEW` trigger is exact-match: the comment body must equal the literal string `REVIEW` after trimming whitespace. Anything else (including `REVIEW please` or quoted text) is ignored silently. The bot must have the `pull_request` GitHub App permission and the **Issue comment** webhook event subscribed.\n\nFor PR reviews, Reviewotron adds an `eyes` reaction while a review is running. On automatic PR events the reaction is attached to the PR; on manual `REVIEW` comments it is attached to the trigger comment. The `eyes` reaction is removed before posting a review. If the review completes with no findings and no failure notice, no PR review is posted and Reviewotron posts a PR comment saying `LGTM :+1:`.\n\nEvents are processed asynchronously — the webhook returns `200 accepted` immediately, and the review runs in the background.\n\n### Defaults\n\n**All four automatic-review triggers default to `false`.** A repo without a `.reviewotron.json` (or one that doesn't set the relevant flags) receives no reviews. Opt in via `.reviewotron.json`:\n\n| Flag | Effect when `true` |\n|------|--------------------|\n| `auto_review_pr_open` | Review PRs on open / reopen / ready-for-review |\n| `auto_review_pr_sync` | Review PRs when new commits land on them |\n| `review_pushes_to_develop` | Review pushes to the `develop` branch |\n| `auto_review_on_comment` | Review when someone posts a `REVIEW` comment on a PR |\n\nManual `REVIEW` comments bypass the dedup that protects the automatic flow from re-reviewing the same head SHA — by design, since the manual trigger means the user wants a fresh review.\n\n---\n\n## Agent Helper Mode\n\nReviewotron ships as a single self-contained binary that another agent can call\nto review code on demand — for example, an app-building agent reviewing the\nproject it just generated before publishing, then re-running after each change.\nNothing has to be deployed alongside the binary: the API key comes from the\nenvironment and the review configuration is passed inline.\n\n### Key points\n\n- **No files required.** The Anthropic API key is read from `--anthropic-api-key`,\n  else the `ANTHROPIC_API_KEY` environment variable, else a `--secrets` file if\n  you choose to provide one (in that order). A `secrets.json` is *not* read\n  unless you pass `--secrets` explicitly.\n- **Configurable on the fly.** Pass configuration inline with `--config '\u003cjson\u003e'`\n  (the same schema as `.reviewotron.json`; omitted fields fall back to defaults).\n  Precedence: `--config` \u003e a config file under `--root`/`PATH` \u003e built-in defaults.\n- **Self-describing config.** `reviewotron config-help` prints the config JSON\n  Schema (field names, types, enum domains, descriptions) so an agent can\n  discover the available knobs before deciding what to pass via `--config`.\n- **Security on by default.** In local mode the multi-agent security pipeline\n  runs by default (it is off by default for webhooks). Disable it with\n  `--no-security`. The general code review also runs by default.\n- **Three ingestion modes**, all printing the same review JSON:\n\n  | Mode | Command | What it reviews |\n  |------|---------|-----------------|\n  | Single file | `review-path FILE` | One file, as newly-added code |\n  | Folder (Git or not) | `review-path DIR` | Every file under a directory, as newly-added code |\n  | Diff / delta | `review-diff --diff -` | A unified diff on stdin (or `--diff FILE`, or a generated Git working-tree diff) |\n\n### Output contract\n\nWith `--output json`:\n\n- **Success** → stdout is `{ \"summary\": \"...\", \"findings\": [ ... ] }`, exit code `0`.\n- **Failure** (bad path, missing key, invalid config, review error) → stdout is\n  `{ \"error\": \"\u003cmessage\u003e\" }`, non-zero exit code.\n\nA caller can branch on the exit code and parse one JSON object either way. Logs\ngo to stderr; only the JSON object is written to stdout.\n\n### Examples\n\nReview a finished app folder (raise the size limits for whole-project reviews):\n\n```bash\nexport ANTHROPIC_API_KEY=sk-ant-...\nreviewotron review-path ./my-app \\\n  --config '{\"max_files\": 500, \"max_diff_lines\": 50000}' \\\n  --output json\n```\n\nReview a single file:\n\n```bash\nreviewotron review-path ./my-app/src/payments.ts --output json\n```\n\nReview an incremental change passed as a diff:\n\n```bash\ngit -C ./my-app diff | reviewotron review-diff --diff - --root ./my-app --output json\n```\n\nDiscover the config knobs, then run without the security pipeline:\n\n```bash\nreviewotron config-help                                   # JSON Schema of the config\nreviewotron review-path ./my-app --no-security --output json\n```\n\n### Notes\n\n- The security pipeline runs by default in local mode; turn it off with\n  `--no-security`. The flag owns the on/off decision, while `--config` still\n  controls the security details (`vuln_classes`, model tiers, thresholds).\n  Security analysis adds extra model calls (triage + per-class analysis +\n  validation), so expect higher cost and latency than a general-only review.\n- `review-path` treats every file as newly added, so the whole file is in scope\n  (not only changed lines). Directory walks skip hidden entries (`.git`, `.env`,\n  …), build/dependency directories (`node_modules`, `_build`, `dist`, `build`,\n  `target`, `vendor`, `venv`, `__pycache__`, `coverage`), symlinks, and\n  binary/oversized files.\n- Whole-folder reviews easily exceed the default `max_files` (50) and\n  `max_diff_lines` (2000); raise them via `--config` (e.g.\n  `'{\"max_files\": 500, \"max_diff_lines\": 50000}'`), otherwise the run returns an\n  error explaining which limit was hit.\n- Each invocation runs independently. Omit `--state` (the default) so repeated\n  runs always produce a fresh review instead of skipping as a duplicate.\n\n---\n\n## Setup\n\n### Prerequisites\n\n- OCaml toolchain with opam\n- An Anthropic API key\n- A GitHub personal access token (or GitHub App installation) for each repo\n- (Optional) A Slack bot token for push notifications\n\n### Build\n\n```bash\nmake build        # Build the project\nmake test         # Run tests\nmake fmt          # Format code\nmake clean        # Clean build artifacts\n```\n\n### Secrets File\n\nCreate a `secrets.json` file (see `secrets.json.example`):\n\n```json\n{\n  \"repos\": [\n    {\n      \"url\": \"https://github.com/org/repo\",\n      \"gh_token\": \"ghp_xxxxxxxxxxxx\",\n      \"gh_hook_secret\": \"your-webhook-secret\"\n    }\n  ],\n  \"anthropic_api_key\": \"sk-ant-xxxxxxxxxxxx\",\n  \"slack_access_token\": \"xoxb-xxxxxxxxxxxx\"\n}\n```\n\n**Fields:**\n\n| Field | Required | Description |\n|-------|----------|-------------|\n| `repos` | Yes | List of repositories to monitor |\n| `repos[].url` | Yes | Full GitHub repository URL (e.g. `https://github.com/org/repo`) |\n| `repos[].gh_token` | Yes* | GitHub personal access token with `repo` scope |\n| `repos[].gh_hook_secret` | No | Webhook secret for HMAC signature validation |\n| `repos[].auth` | Yes* | Alternative to `gh_token` — GitHub App installation auth (see below) |\n| `anthropic_api_key` | Yes | Anthropic API key for Claude |\n| `slack_access_token` | No | Slack bot token for posting messages |\n\n*Either `gh_token` or `auth` must be set per repo. Using `gh_token` is the simpler option.\n\nFor local-only `review-diff` usage, `repos` may be an empty list as long as the\nsecrets file still provides `anthropic_api_key`. The webhook server still\nrequires at least one configured repo by default.\n\n#### GitHub App Installation Auth\n\nInstead of a personal access token, you can authenticate as a GitHub App installation:\n\n```json\n{\n  \"repos\": [\n    {\n      \"url\": \"https://github.com/org/repo\",\n      \"auth\": [\n        \"AppInstallation\",\n        {\n          \"installation_id\": \"12345678\",\n          \"client_id\": \"Iv1.xxxxxxxxxx\",\n          \"pem\": \"-----BEGIN RSA PRIVATE KEY-----\\n...\\n-----END RSA PRIVATE KEY-----\"\n        }\n      ],\n      \"gh_hook_secret\": \"your-webhook-secret\"\n    }\n  ]\n}\n```\n\nApp installation tokens are automatically refreshed and cached (55-minute TTL).\n\n### GitHub Webhook\n\nConfigure a webhook in your GitHub repository settings:\n\n| Setting | Value |\n|---------|-------|\n| Payload URL | `https://your-server:1338/github` |\n| Content type | `application/json` |\n| Secret | Same value as `gh_hook_secret` in secrets.json |\n| Events | Select **Pull requests** and **Pushes** |\n\n### Start the Server\n\n```bash\n./reviewotron run --port 1338 --secrets secrets.json --state state.json\n```\n\nVerify it's running:\n\n```bash\ncurl http://localhost:1338/ping\n```\n\n---\n\n## Configuration\n\nEach repo can have a `.reviewotron.json` file in its root. For GitHub webhooks, this is fetched from the repo via the GitHub Contents API on each event. For local `review-diff`, the same file is loaded from the local review root. If the file doesn't exist, defaults are used.\n\n### Full Configuration Reference\n\n```json\n{\n  \"max_diff_lines\": 2000,\n  \"max_files\": 50,\n  \"max_tokens_per_review\": 100000,\n  \"model\": \"claude-sonnet-4-6\",\n  \"ignored_paths\": [\"*.test.js\", \"vendor/\"],\n  \"ignore_generated_files\": true,\n  \"ignored_authors\": [\"dependabot[bot]\"],\n  \"auto_review_pr_open\": false,\n  \"auto_review_pr_sync\": false,\n  \"review_pushes_to_develop\": false,\n  \"auto_review_on_comment\": false,\n  \"review_draft_prs\": false,\n  \"system_prompt_override\": null,\n  \"slack_channel\": \"#code-reviews\",\n  \"show_review_cost\": false,\n  \"review_plugins\": {\n    \"general\": {\n      \"enabled\": true,\n      \"system_prompt_override\": null\n    },\n    \"security\": {\n      \"enabled\": false,\n      \"vuln_classes\": [\"injection\", \"xss\", \"command_injection\", \"authn\", \"authz\", \"ssrf\", \"policy_regression\"],\n      \"always_analyze_vuln_classes\": [],\n      \"triage_model_tier\": \"fast\",\n      \"analysis_model_tier\": \"standard\",\n      \"validator_model_tier\": \"standard\",\n      \"confidence_threshold\": \"medium\",\n      \"memory_max_tokens\": 5000,\n      \"metrics_artifacts\": false,\n      \"debug_artifacts\": false\n    }\n  }\n}\n```\n\n### Config Fields\n\n| Field | Default | Description |\n|-------|---------|-------------|\n| `max_diff_lines` | `2000` | Maximum total diff lines to review. PRs exceeding this are skipped. |\n| `max_files` | `50` | Maximum files to review after ignored/generated files are removed. |\n| `max_tokens_per_review` | `100000` | Token budget hint for the review agent. |\n| `model` | `claude-sonnet-4-6` | Model ID for the general review agent. |\n| `ignored_paths` | `[]` | Glob patterns for files to exclude from review. Supports `*` and `**` wildcards. |\n| `ignore_generated_files` | `true` | Exclude conservatively detected generated files before `max_files` and `max_diff_lines` are enforced. Set to `false` to review generated artifacts. |\n| `ignored_authors` | `[]` | GitHub usernames whose PRs/pushes should be skipped. |\n| `auto_review_pr_open` | `false` | Review PRs when they are opened, reopened, or marked ready. |\n| `auto_review_pr_sync` | `false` | Review PRs when new commits are pushed to them. |\n| `review_pushes_to_develop` | `false` | Review pushes to the `develop` branch. |\n| `auto_review_on_comment` | `false` | Review when someone posts a top-level PR comment whose body is exactly `REVIEW` (after trimming). Requires the GitHub App to subscribe to **Issue comment** events. |\n| `review_draft_prs` | `false` | Include draft PRs in automatic reviews. By default drafts are skipped regardless of `auto_review_pr_open` / `auto_review_pr_sync`. |\n| `system_prompt_override` | `null` | Replace the default general review system prompt entirely. |\n| `slack_channel` | `null` | Slack channel for push review notifications. Requires `slack_access_token` in secrets. |\n| `show_review_cost` | `false` | Append a cost summary footer to PR reviews. |\n| `review_plugins` | (see below) | Per-plugin configuration. |\n\nGenerated-file detection is intentionally conservative. It includes exact\n`__generated__` and `gen` path components, path components or file stems ending\nin `_gen`, file stems starting with `generated_`, common generated artifact\nsuffixes such as minified assets, `.map` files, and protobuf outputs, and\ngenerated-file header markers. Broad folders such as `generated/`, `dist/`,\n`build/`, and `vendor/` remain reviewable unless excluded with `ignored_paths`.\n\n### Plugin Configuration\n\n#### General Plugin\n\n| Field | Default | Description |\n|-------|---------|-------------|\n| `enabled` | `true` | Enable/disable the general code review. |\n| `system_prompt_override` | `null` | Override the general review prompt (plugin-level). |\n\n#### Security Plugin\n\n| Field | Default | Description |\n|-------|---------|-------------|\n| `enabled` | `false` | Enable/disable security analysis. |\n| `vuln_classes` | All 7 classes | Which vulnerability types to scan for. |\n| `always_analyze_vuln_classes` | `[]` | Vulnerability classes that bypass `confidence_threshold`. Classes listed here are implicitly enabled even if absent from `vuln_classes`. Use sparingly for high-risk repos or temporarily while tuning recall. |\n| `triage_model_tier` | `\"fast\"` | Model tier for the triage agent. |\n| `analysis_model_tier` | `\"standard\"` | Model tier for per-class analysis agents. |\n| `validator_model_tier` | `\"standard\"` | Model tier for the adversarial validator. |\n| `confidence_threshold` | `\"medium\"` | Minimum triage confidence to trigger analysis for enabled classes. `\"high\"` = only high-confidence signals. `\"medium\"` = high + medium. `\"low\"` = all signals. |\n| `memory_max_tokens` | `5000` | Target size limit for the repo's security memory file. |\n| `metrics_artifacts` | `false` | Write compact security metrics artifacts under the review debug dir's `security/` subdirectory. These omit source code and prompt bodies. |\n| `debug_artifacts` | `false` | Write full redacted per-stage security debug artifacts under the review debug dir's `security/` subdirectory. Sensitive and opt-in. |\n\n#### Model Tiers\n\n| Tier | Model | Typical Use |\n|------|-------|-------------|\n| `\"fast\"` | `claude-haiku-4-5-20251001` | Triage, memory curator |\n| `\"standard\"` | `claude-sonnet-4-6` | Analysis agents, validator, general review |\n| `\"strong\"` | `claude-opus-4-6` | Reserved for complex codebases |\n\n#### Vulnerability Classes\n\n| Value | Description |\n|-------|-------------|\n| `\"injection\"` | SQL injection, NoSQL injection, query string construction |\n| `\"xss\"` | Cross-site scripting (reflected, stored, DOM-based) |\n| `\"command_injection\"` | OS command injection via exec/system/popen |\n| `\"authn\"` | Authentication bypass, weak token validation, missing expiry |\n| `\"authz\"` | Authorization flaws, IDOR, missing permission checks |\n| `\"ssrf\"` | Server-side request forgery via user-controlled URLs |\n| `\"policy_regression\"` | Security policy/control regressions such as broad sudo, CI/cloud/RBAC permission expansion, privileged Kubernetes workload settings, and disabled TLS/auth/CSRF controls |\n\n### Skip Behavior\n\nReviewotron skips events in these cases:\n\n- **Bot senders** — any login ending in `[bot]`\n- **Ignored authors** — usernames in the `ignored_authors` list\n- **Non-reviewable actions** — PR closed, edited, or other non-code-change actions\n- **Draft PRs** — skipped until marked ready\n- **Already reviewed** — same PR + head SHA (or same push after SHA) already processed\n- **Empty diff** — all files filtered by `ignored_paths` or generated-file detection\n- **Diff too large** — exceeds `max_diff_lines` after ignored/generated files are removed\n- **Non-develop pushes** — only `refs/heads/develop` is reviewed\n\n---\n\n## Security Review Pipeline\n\nWhen the security plugin is enabled, every diff goes through a multi-agent pipeline:\n\n### 1. Triage (Haiku, single-shot)\n\nBefore triage, Reviewotron runs a deterministic scan over changed paths and added hunk lines for advisory security signals such as dangerous APIs, risky paths, sensitive files, changed security controls, and stateful operations. These signals are hints only: they are summarized by category, vulnerability hint, and affected file for triage, with only the strongest exact hints included. They never become findings and never route directly to analysis.\n\nThe triage agent scans the diff for security-relevant patterns and classifies them by vulnerability type. This is intentionally biased toward **over-flagging** — it's cheap to run an analysis agent that finds nothing, costly to miss a real issue.\n\nThe triage agent outputs signals with confidence levels (`high`, `medium`, `low`). The `confidence_threshold` config controls which signals proceed to analysis for enabled vulnerability classes. `always_analyze_vuln_classes` is the explicit override that bypasses the threshold; classes listed there are implicitly enabled even if absent from `vuln_classes`.\n\n### 2. Analysis (Sonnet, per vulnerability class, parallel)\n\nFor each flagged vulnerability class, a specialized agent runs deep analysis:\n\n1. **Source identification** — Where does user-controlled input enter?\n2. **Sink identification** — Where does data reach a dangerous operation?\n3. **Data flow tracing** — Can the source reach the sink? Traces through variables, function calls, returns.\n4. **Sanitization evaluation** — Is there adequate, context-correct sanitization on the path?\n\nFor `policy_regression`, the same finding schema is used with a policy proof instead of a runtime user-input flow: source is the changed principal/grant/config entry or removed control, sink is the effective privileged capability or weakened boundary, flow is changed line -\u003e effective policy/control state -\u003e concrete action now possible, and sanitization is the missing or inadequate scoping/mitigation.\n\nAnalysis agents can fetch additional files from the repo via the GitHub Contents API when they need to trace a data flow beyond the diff. Each run starts from a focused, class-specific analysis question and the triage evidence. The agent is instructed to inspect changed regions and direct dependencies first, fetch more files only to close a specific evidence gap, and return no finding when a bounded check cannot establish the required source/effect, sink/capability, and missing control.\n\nAnalysis depth is budgeted by vulnerability class, triage confidence, and signal count. High-confidence AuthN/AuthZ/SSRF signals still get the most room because those classes often need cross-file context; medium/low-confidence and policy-regression runs are kept tighter to avoid broad repo archaeology.\n\n### 3. Validation (Sonnet, adversarial)\n\nAll candidate findings from all analysis agents pass through a single validator agent. It acts as an adversarial false-positive filter, checking:\n\n- The claimed source actually accepts external input\n- The claimed sink actually performs the dangerous operation\n- Every step in the flow path is backed by evidence (file + line)\n- The sanitization assessment is correct\n- A confirmed result includes a concrete proof-by-construction: reproducible trigger, source-to-sink trace, missing control, expected impact, and explicit assumptions\n\nFor `policy_regression`, validation does not require a user-controlled runtime source, but it does require exact file/line evidence, a concrete effective privilege/control change, a concrete action now possible, no unresolved assumptions, and enough proof to reject vague \"security relevant\" policy edits.\n\n**Findings that fail validation are dropped.** Confirmed validator results without concrete proof are downgraded after parsing and are not surfaced. This is by design — a noisy security reviewer that cries wolf loses developer trust. Dropped findings are logged for offline prompt tuning.\n\n### 4. Memory Curation (Haiku, async)\n\nAfter the review is posted, a curator agent runs asynchronously to update the repo's security memory with learnings from the review. This is fire-and-forget — it doesn't block the review.\n\n### Severity Mapping\n\n| Analysis Confidence | Post-Validation Severity |\n|---------------------|--------------------------|\n| High + Confirmed | Critical |\n| Medium + Confirmed | Warning |\n| Low + Confirmed | Warning |\n\n---\n\n## Slack Integration\n\nPush reviews (to `develop`) optionally send a Slack notification. This requires:\n\n1. A `slack_access_token` in `secrets.json` — a Slack bot token (`xoxb-...`) with `chat:write` permission\n2. A `slack_channel` set in the repo's `.reviewotron.json`\n\nThe message includes:\n- Pusher name and commit count\n- Link to the compare view on GitHub\n- Review summary text\n- Finding counts (critical, warnings, suggestions)\n- Color-coded: red if any critical findings, green otherwise\n\nIf the security plugin encountered an error, a note is appended to the Slack message.\n\nIf `slack_access_token` is not configured, Slack posting is silently skipped.\n\n---\n\n## State and Persistence\n\n### State File\n\nThe `--state` flag enables persistent state tracking. The state file (JSON) records:\n\n- **PR reviews**: repo URL, PR number, head SHA, timestamp, review costs\n- **Push reviews**: repo URL, after SHA\n- **Generic change reviews**: repo key, change key, timestamp, review costs\n\nFor GitHub webhooks, this prevents duplicate reviews — if the same PR at the same commit SHA is already recorded, the review is skipped. Local diff reviews record their `repo_key` and `change_key` in the same state file, but currently do not skip duplicates. State is trimmed to the 500 most recent records per repo key.\n\nWithout `--state`, state is in-memory only and lost on restart. This means reviews may be duplicated after a server restart.\n\n### Security Memory Files\n\nThe security pipeline maintains per-repo memory files at `memory/{repo-slug}.md` in in-memory/local mode. In persistent server mode, memory lives beside feedback/debug data; for example, if feedback evidence is under `./var/reviewotron-feedback-evidence/`, memory files go under `./var/memory/{repo-slug}.md`. These are plain-text markdown files (target ~5000 tokens) that accumulate knowledge about the repo:\n\n- Architecture notes (frameworks, DB access patterns, auth middleware)\n- Known safe patterns (parameterized queries, auto-escaping templates)\n- Known risk areas (shell command construction, raw HTML rendering)\n- Suppressions (accepted risks with context)\n\nMemory is injected into every security agent's prompt, reducing redundant file fetching and pattern re-discovery across reviews.\n\nThe memory curator rewrites the repo brief asynchronously after a review. If two reviews update the same brief concurrently, the last write wins; the brief is intentionally architectural rather than per-finding state.\n\n### Debug Dumps\n\nWhen an agent's structured output can't be parsed, a debug dump is saved to the review debug dir. In in-memory/local mode this is `debug/{repo-slug}/{sha-prefix}/`. In persistent server mode it uses a sibling root next to feedback evidence; for example, if feedback evidence is under `./var/reviewotron-feedback-evidence/`, debug dumps go under `./var/debug/{repo-slug}/{sha-prefix}/`. These contain the raw agent output for diagnosing prompt or parsing issues.\n\nSecurity metrics/debug artifacts are separate and opt-in via `review_plugins.security.metrics_artifacts` and `review_plugins.security.debug_artifacts`. Metrics write compact JSON files under the review debug dir's `security/` subdirectory; full debug artifacts additionally write redacted stage inputs and outputs and should be treated as sensitive.\n\n### OpenTelemetry Traces\n\nOpenTelemetry tracing is off by default in public/local runs. No trace exporter\nis started, and no trace network traffic is emitted unless it is explicitly\nenabled by environment.\n\nEnable local OTLP trace export with:\n\n```bash\nREVIEWOTRON_OTEL=1 reviewotron run --secrets secrets.json\n```\n\nWith no endpoint variable set, traces go to the default\n`http://127.0.0.1:4318`. `REVIEWOTRON_OTEL=1` on its own is only correct when a\nnode-local OTLP/HTTP collector (agent or sidecar) is listening there — if\nnothing is listening on that address, tracing looks enabled but every span is\nsilently dropped. Ahrefs/private deployments without a node-local collector\nmust also set an endpoint:\n\n```bash\nREVIEWOTRON_OTEL=1\nOTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318\n```\n\n`OTEL_EXPORTER_OTLP_ENDPOINT` is a base URL; `/v1/traces` is appended.\n`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` is used verbatim and must include the\n`/v1/traces` path itself:\n\n```bash\nOTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:5996/v1/traces\n```\n\nSetting either endpoint variable to a non-empty value enables tracing by\nitself — `REVIEWOTRON_OTEL=1` is not required in that case.\n`REVIEWOTRON_OTEL=0` or `OTEL_SDK_DISABLED=true` still force tracing off even\nwhen an endpoint is set.\n\nAn endpoint variable that is present but empty or whitespace-only (for\nexample a templated `OTEL_EXPORTER_OTLP_ENDPOINT=` line with no value\nsubstituted) is treated as unset for the enable check, but the OTLP client\nwould otherwise read it verbatim and export to a broken URL. At startup\nReviewotron detects this and normalizes the variable to its effective value\n(the default endpoint, or the resolved base plus `/v1/traces`), logging a\nwarning when it does. Avoid relying on this: either set a full URL or omit\nthe variable entirely.\n\nExport is OTLP over HTTP/protobuf — point endpoint variables at a collector's\nHTTP receiver (4318-style port), not a gRPC-only 4317 endpoint. Span batches\nflush roughly every 2s with retries; export failures are logged to stderr\n(verbosity controlled by `OTEL_LOG_LEVEL`). On graceful shutdown, queued spans\nare flushed, but reviews still in flight when the server stops may produce\nincomplete or missing traces — expected given the fire-and-forget webhook\ndesign.\n\nOther supported standard variables:\n\n```bash\nOTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20\u003ctoken\u003e\nOTEL_RESOURCE_ATTRIBUTES=service.namespace=devtools,deployment.environment=prod\n```\n\n`OTEL_EXPORTER_OTLP_HEADERS` is needed for authenticated backends.\n`OTEL_SDK_DISABLED=true` disables tracing even when `REVIEWOTRON_OTEL=1` is\nset. `OTEL_SERVICE_NAME` is respected; otherwise spans use\n`service.name=reviewotron`.\n\nThe existing `/home/user/reviewotron/log/reviewotron.*.json` Vector path remains\nJSON log ingestion only. It is not a trace drain; traces are emitted via OTLP.\n\n---\n\n## Review Feedback\n\nWhen webhook mode is started with `--state`, Reviewotron also enables feedback\npersistence for GitHub PR review bodies and inline PR review comments. Review\nbodies and inline findings include this prompt:\n\n```markdown\nWas this review helpful? React with :+1: or :-1:.\n```\n\nEach inline finding comment also gets a hidden marker:\n\n```markdown\n\u003c!-- reviewotron-feedback-id: rvf_... --\u003e\n```\n\nGitHub hides the marker in the UI but keeps it in the API body, which lets the\ncollector map a posted review comment back to a local feedback target.\n\nBy default, feedback data is stored next to the state file, not inside\n`state.json`:\n\n```text\n/path/to/state.json\n/path/to/reviewotron-feedback-targets.json\n/path/to/reviewotron-feedback-events.jsonl\n/path/to/reviewotron-feedback-evidence/\n```\n\nUse `--feedback-dir /durable/path` on both `run` and `collect-feedback` to keep\ntargets, events, and evidence bundles somewhere other than the state file's\ndirectory. This is recommended when `--state` points at a temporary or\nephemeral location. When `reviewotron run` has a feedback store, it collects\nreaction counts in the server process. A target is polled immediately the first\ntime, then waits one hour between polls by default; pass\n`--poll-interval-seconds 60` to run the server poller on a minute cadence.\n\n`reviewotron-feedback-targets.json` stores polling state for inline PR review\ncomments and top-level PR review bodies. Inline comments are collected through\nGitHub's REST reactions API; review-body reactions are collected through a\nminimal GitHub GraphQL query using the review `node_id`.\n`reviewotron-feedback-events.jsonl` stores append-only aggregate events such as\nreaction count changes, comment ID resolution, and target finalization.\n\nFor each successfully posted GitHub PR review with feedback persistence enabled,\nReviewotron also writes one immutable evidence bundle under\n`reviewotron-feedback-evidence/\u003creview_batch_id\u003e/`. Target records link each\nfeedback ID back to the bundle; inline targets also link to a finding ID.\nBundles contain the reviewed filtered diff, the posted review/comment bodies\nwith feedback prompts and inline markers, routed findings with plugin-level\nprovenance, review costs, review config, and fetched file metadata hashes. They\nare intended as input for a later feedback-review agent/command that correlates\nGitHub reactions with review context and produces improvement recommendations;\nhumans should not need to inspect the JSON files manually.\n\nPrivacy rules:\n\n- Only aggregate `+1` and `-1` counts are stored.\n- Evidence bundles are written only for successfully posted GitHub PR reviews.\n- Raw reaction objects are not stored.\n- Webhook payloads are not stored.\n- Raw prompts, agent transcripts, tool outputs, and unrelated logs are not\n  stored.\n- Fetched file contents are not stored by default; only path, byte count, and\n  SHA-256 metadata are written.\n- GitHub user identity fields such as `sender`, `user`, `login`, `name`,\n  `email`, `author`, `committer`, `pusher`, and `avatar_url` are not written to\n  feedback files.\n\nPolling stops after the earliest of five days from target creation, 24 hours\nafter the first qualifying human PR interaction, or PR close/merge. A closed PR\nreceives one final poll before the target is marked closed.\n\nThe `collect-feedback` command is a one-shot fallback for backfills, debugging,\nor smoke tests when the server is not running:\n\n```bash\nreviewotron collect-feedback --secrets secrets.json --state /path/to/state.json --feedback-dir /durable/path\n```\n\nBoth the server poller and the one-shot collector are idempotent. They resolve\nmissing GitHub review comment IDs from hidden markers, poll review-body and\ninline-comment reaction counts, update target state, and append JSONL events\nonly when counts change or targets finalize.\n\nAfter collection, summarize the local feedback store with:\n\n```bash\nreviewotron feedback-report --state /path/to/state.json --feedback-dir /durable/path\n```\n\nUse `--output json` when feeding the parsed feedback into another tool or a\nfuture feedback-review agent. The report joins target records, aggregate events,\nand evidence bundles, grouping feedback by posted PR review and preserving the\nfeedback ID, target kind, inline finding ID/plugin/source when present, reaction\ncounts, reviewed PR metadata, GitHub discussion/review URL, and evidence bundle\npath. For large feedback sets, keep investigations bounded with filters such as:\n\n```bash\nreviewotron feedback-report --state /path/to/state.json --feedback-dir /durable/path \\\n  --sentiment negative --limit 20 --brief\n```\n\nUse `review_batch_id` to open one evidence bundle, `finding_id` to select the\nexact finding in `findings.json` or `posted_review.json`, and `comment_id` or\n`github_comment_url` to return to the GitHub discussion.\n\n---\n\n## CLI Usage\n\n### `reviewotron run` — Start the Webhook Server\n\n```\nreviewotron run [OPTIONS]\n```\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `-p`, `--port` | `1338` | HTTP server port |\n| `--secrets` | `secrets.json` | Path to secrets file |\n| `--config-filename` | `.reviewotron.json` | Config filename to look for in repos |\n| `--state` | (none — in-memory) | Path to state file for persistence |\n| `--feedback-dir` | sibling paths next to `--state` | Directory for feedback targets, events, and evidence bundles |\n| `--poll-interval-seconds` | `3600` | Minimum seconds between feedback polls; use `60` for minute polling |\n| `--logfile` | (stderr) | Log file path |\n| `--loglevel` | (default) | Log level: `debug`, `info`, `warn`, `error` |\n\n### `reviewotron check` — Parse a Webhook Payload (Dry Run)\n\n```\nreviewotron check --event-type pull_request --payload payload.json [OPTIONS]\n```\n\nParses and displays a GitHub webhook payload without starting the server or performing any review. Useful for verifying payload parsing.\n\n| Option | Required | Description |\n|--------|----------|-------------|\n| `--event-type` | Yes | GitHub event type (`pull_request` or `push`) |\n| `--payload` | Yes | Path to JSON payload file |\n| `--secrets` | No | Path to secrets file (defaults to `secrets.json`; must exist for initialization) |\n\n### `reviewotron review-diff` — Review a Local Unified Diff\n\n```\nreviewotron review-diff [OPTIONS]\n```\n\nRuns the same core review engine against a local unified diff and prints the final review to stdout. Logs go to stderr unless `--logfile` is set. The diff can be a file (`--diff FILE`), stdin (`--diff -`), or — when `--diff` is omitted — a Git diff generated from the merge-base of `HEAD` and the inferred base ref, including working-tree changes. This path does not fetch or publish through GitHub; local file-content expansion uses `--root`.\n\nThe Anthropic API key is resolved from `--anthropic-api-key`, then the `ANTHROPIC_API_KEY` environment variable, then a `--secrets` file if one is given — no secrets file is required. Configuration is resolved from `--config` (inline JSON), then `.reviewotron.json` under `--root`, then defaults. See [Agent Helper Mode](#agent-helper-mode).\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `--diff` | Git diff against inferred base | Path to a unified diff file |\n| `--base` | inferred from Git | Base ref for generated diffs; tries `origin/HEAD`, `origin/main`, `origin/master`, then the upstream remote |\n| `--root` | Git worktree root, then cwd | Repository root for local file-content lookups |\n| `--repo-key` | `local:\u003croot\u003e` | Stable repository key for config, memory paths, and state |\n| `--change-key` | digest of filtered diff | Stable change key recorded in state |\n| `--title` | inferred from base or diff file | Title passed to review agents |\n| `--description-file` | (none) | Optional file used as the review description |\n| `--config-filename` | `.reviewotron.json` | Config file loaded from `--root`, or absolute config path |\n| `--config` | (none) | Inline config JSON; overrides any config file |\n| `--anthropic-api-key` | (none) | Anthropic API key; overrides `$ANTHROPIC_API_KEY` and any secrets file |\n| `--no-security` | (off) | Disable the security pipeline (on by default in local mode) |\n| `--output` | `markdown` | Output format: `markdown` or `json` |\n| `--secrets` | (none) | Optional secrets file; the API key is taken from `--anthropic-api-key`, then `$ANTHROPIC_API_KEY`, then this file |\n| `--state` | (none — in-memory) | Optional state file updated after a successful review |\n\nJSON output is an object with a review-level summary and a machine-readable findings list:\n\n```json\n{\n  \"summary\": \"The review found one startup compatibility issue in session metadata handling.\",\n  \"findings\": [\n    {\n      \"file\": \"backend/safer-claude-code/safer_claude_code.ml\",\n      \"line\": 492,\n      \"level\": \"warning\",\n      \"category\": \"bug\",\n      \"summary\": \"Legacy session-id file from old scc crashes startup because ensure_dir refuses to treat a regular file as a directory\",\n      \"failure_scenario\": \"Any user who ran a previous scc has a regular file at \u003cscc_metadata\u003e/sessions/\u003cwt_basename\u003e holding their last session UUID. After upgrading, the first scc -f or scc run-on calls prepare_session_id_mount, which calls ensure_dir(Filename.dirname host_path) — i.e. ensure_dir on the legacy file path. ensure_dir sees S_REG and fails. scc aborts on startup until the user manually removes the legacy file.\"\n    }\n  ]\n}\n```\n\n### `reviewotron review-path` — Review a File or Directory\n\n```\nreviewotron review-path PATH [OPTIONS]\n```\n\nReviews a single file or an entire directory by treating every file as newly\nadded, reusing the same engine, output formats, and JSON contract as\n`review-diff`. This is how to review code that has no Git history — a single\nfile, a freshly generated project, or a non-Git working tree.\n\nFor a file, the file's parent directory becomes the review root (so context\nlookups resolve siblings). For a directory, `PATH` is walked recursively in\nsorted order; hidden entries, build/dependency directories, symlinks, and\nbinary/oversized files are skipped (see [Agent Helper Mode](#agent-helper-mode)).\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `PATH` | (required) | File or directory to review |\n| `--config` | (none) | Inline config JSON; overrides any config file |\n| `--anthropic-api-key` | (none) | Anthropic API key; overrides `$ANTHROPIC_API_KEY` and any secrets file |\n| `--no-security` | (off) | Disable the security pipeline (on by default in local mode) |\n| `--output` | `markdown` | Output format: `markdown` or `json` |\n| `--repo-key` | `local:\u003croot\u003e` | Stable repository key for config, memory, and state |\n| `--change-key` | digest of the synthesized diff | Stable change key recorded in state |\n| `--title` | inferred from the path | Title passed to the review agents |\n| `--config-filename` | `.reviewotron.json` | Config filename loaded from the root, or absolute config path |\n| `--state` | (none — in-memory) | Optional state file updated after a successful review |\n\nWhole-folder reviews commonly exceed the default `max_files` / `max_diff_lines`\nlimits; raise them with `--config` (see [Agent Helper Mode](#agent-helper-mode)).\n\n### Local Security Timing and Artifact Comparison\n\nTo compare local filesystem review behavior before and after enabling the\nsecurity pipeline, run the same diff twice with stable repo/change keys and\nmeasure wall time. Use a throwaway state file so duplicate-review detection does\nnot skip the second run.\n\nGeneral-only baseline:\n\n```bash\n/usr/bin/time -p \\\n  dune exec -- src/reviewotron.exe review-diff \\\n  --root /path/to/repo \\\n  --diff /tmp/change.diff \\\n  --repo-key local-security-timing \\\n  --change-key general-only-1 \\\n  --no-security \\\n  --output json \\\n  --logfile /tmp/reviewotron-general.log \\\n  --state /tmp/reviewotron-general-state.json \\\n  \u003e /tmp/reviewotron-general.json\n```\n\nSecurity-enabled run with metrics and redacted debug artifacts:\n\n```bash\nSECURITY_CONFIG='{\n  \"max_files\": 500,\n  \"max_diff_lines\": 50000,\n  \"show_review_cost\": true,\n  \"review_plugins\": {\n    \"security\": {\n      \"metrics_artifacts\": true,\n      \"debug_artifacts\": true\n    }\n  }\n}'\n\n/usr/bin/time -p \\\n  dune exec -- src/reviewotron.exe review-diff \\\n  --root /path/to/repo \\\n  --diff /tmp/change.diff \\\n  --repo-key local-security-timing \\\n  --change-key security-observed-1 \\\n  --config \"$SECURITY_CONFIG\" \\\n  --output json \\\n  --logfile /tmp/reviewotron-security.log \\\n  --state /tmp/reviewotron-security-state.json \\\n  \u003e /tmp/reviewotron-security.json\n```\n\nInspect the security metrics line in `/tmp/reviewotron-security.log` and the\nartifact directory under `debug/local-security-timing/\u003cdiff-digest-prefix\u003e/security/`.\n`metrics.json` and `fetch_stats.json` are compact and omit prompt/source bodies;\nthe full debug files include redacted stage inputs and outputs and should be\ntreated as sensitive.\n\nTo isolate just the security pipeline cost, add `\"general\": {\"enabled\": false}`\nunder `review_plugins` in `SECURITY_CONFIG`.\n\n### `reviewotron config-help` — Print the Config Schema\n\n```\nreviewotron config-help\n```\n\nPrints the review configuration as a JSON Schema — every field with its type,\nenum domain (for `vuln_classes`, model tiers, confidence), and a one-line\ndescription. An agent can read this to discover which knobs exist and what they\naccept, then pass chosen values via `--config`. Takes no options and makes no\nnetwork calls.\n\n### `reviewotron collect-feedback` — Poll Review Feedback Reactions\n\n```\nreviewotron collect-feedback --secrets secrets.json --state state.json [OPTIONS]\n```\n\nRuns one GitHub reaction collection pass for feedback targets stored next to\n`state.json`. Server mode normally collects feedback automatically; this command\nis for backfills, debugging, and smoke tests.\n\n| Option | Required | Description |\n|--------|----------|-------------|\n| `--secrets` | Yes | Secrets file with repo authentication |\n| `--state` | Yes | State file whose sibling feedback files should be loaded |\n| `--feedback-dir` | No | Directory containing feedback targets, events, and evidence bundles; must match the server's `--feedback-dir` when set |\n| `--poll-interval-seconds` | No | Minimum seconds between polls for the same feedback target; defaults to 3600 |\n| `--logfile` | No | Log file path |\n| `--loglevel` | No | Log level: `debug`, `info`, `warn`, `error` |\n\n### `reviewotron feedback-report` — Summarize Collected Feedback\n\n```\nreviewotron feedback-report --state state.json [OPTIONS]\n```\n\nReads local feedback files and evidence bundles without contacting GitHub or an\nLLM. Markdown output is intended for quick inspection; JSON output is intended\nfor downstream tools.\n\n| Option | Required | Description |\n|--------|----------|-------------|\n| `--state` | Yes | State file whose sibling feedback files should be loaded |\n| `--feedback-dir` | No | Directory containing feedback targets, events, and evidence bundles; must match the server's `--feedback-dir` when set |\n| `--output` | No | `markdown` or `json`; defaults to `markdown` |\n| `--sentiment` | No | Filter targets: `all`, `reacted`, `positive`, `negative`, `mixed`, or `unreacted`; defaults to `all` |\n| `--review-batch-id` | No | Limit output to one evidence bundle/review batch |\n| `--pr` | No | Limit output to one pull request number |\n| `--limit` | No | Limit output to the first N matching feedback targets |\n| `--brief` | No | Omit finding message snippets from markdown output |\n\n### Endpoints\n\n| Path | Description |\n|------|-------------|\n| `/ping` | Health check — returns uptime |\n| `/github` | GitHub webhook receiver |\n\n---\n\n## Cost Tracking\n\nEvery agent call tracks token usage and estimates cost:\n\n- **Per agent**: input tokens, output tokens, cache read tokens, cache creation tokens, model ID, number of tool-use turns, files fetched, estimated USD cost\n- **Per plugin**: aggregated agent costs (general, security)\n- **Per review**: total across all plugins\n\nCosts are:\n- **Logged** at `info` level after each review\n- **Stored** in `state.json` alongside the review record (when state persistence is enabled)\n- **Optionally shown** in the PR review footer (when `show_review_cost: true`)\n\nCost footer example:\n\u003e *Review cost: 5 agents (general: 1 agent, security: 4 agents), ~$0.42*\n\n### Pricing\n\nCosts are estimated using a built-in pricing table that includes prompt caching rates:\n\n| Model Family | Input | Output | Cache Write (5m) | Cache Read |\n|--------------|-------|--------|------------------|------------|\n| Claude Opus 4.x | $5.00/MTok | $25.00/MTok | $6.25/MTok | $0.50/MTok |\n| Claude Sonnet 4.x | $3.00/MTok | $15.00/MTok | $3.75/MTok | $0.30/MTok |\n| Claude Haiku 4.5 | $1.00/MTok | $5.00/MTok | $1.25/MTok | $0.10/MTok |\n\nCache write tokens are charged at 1.25x the base input price (5-minute TTL). Cache read tokens are charged at 0.1x the base input price. Cache token counts are extracted from the Anthropic API response and tracked per-agent.\n\nThe pricing table is a single record in the codebase (`lib/cost_tracking.ml`) — update it when prices change.\n\n---\n\n## Limitations\n\n### Diff Size\n\nPRs with more than `max_diff_lines` (default 2000) total diff lines after ignored/generated files are removed are **skipped entirely**. There is no partial review — it's all or nothing. For large PRs, consider breaking them into smaller ones.\n\n### Push Reviews\n\nOnly pushes to `refs/heads/develop` are reviewed. Other branches, including `main`/`master`, are not reviewed on push. PR reviews cover all branches.\n\n### File Content Fetching\n\n- The general review plugin fetches up to **5 key files** for additional context (added or modified files only)\n- Security analysis agents can fetch any file via `get_file_content`, bounded by a dynamic step budget derived from vulnerability class, triage confidence, and signal count\n- All file fetches use the PR head SHA as the git ref, so agents see the PR branch state (not the default branch)\n\n### Static Analysis Only\n\nThe security pipeline performs **static analysis on the diff and referenced files**. It cannot:\n\n- Execute code or run tests\n- Detect runtime-only vulnerabilities\n- Analyze compiled/minified code meaningfully\n- Fully model infrastructure or policy state outside the reviewed diff and fetched files\n\n### Security Scope\n\n- 7 vulnerability classes are supported. Other classes (e.g., cryptographic weaknesses, deserialization, path traversal) are not covered.\n- The triage agent may miss security signals in unusual code patterns. Bumping `triage_model_tier` to `\"standard\"` (Sonnet) can improve recall at higher cost.\n- AuthN/AuthZ/SSRF analysis from diff context alone is inherently limited. These classes produce the most false negatives.\n\n### Webhook Signature Validation\n\nIf no `gh_hook_secret` is configured for a repo, webhook signature validation is **skipped** — the event is accepted without verification. While the review will fail at the GitHub API step if no auth token is configured, it's best practice to always set a webhook secret.\n\n### Duplicate Prevention\n\nDuplicate review prevention relies on the state file. Without `--state`, or after a server restart with in-memory-only state, the same PR/push may be reviewed again.\n\n### Concurrent Reviews\n\nMultiple reviews can run concurrently (events are processed via `Lwt.async`). The security memory queue handles concurrent appends safely, but there's no global rate limiting on Anthropic API calls.\n\n---\n\n## Troubleshooting\n\n### Review not triggering\n\n1. Check the webhook delivery log in GitHub (Settings \u003e Webhooks \u003e Recent Deliveries)\n2. Verify the server is running: `curl http://your-server:1338/ping`\n3. Check the server logs for skip reasons:\n   - `\"bot sender\"` — the event was from a bot account\n   - `\"ignored author\"` — the author is in `ignored_authors`\n   - `\"action ... not reviewable\"` — the PR action doesn't trigger reviews\n   - `\"draft PR\"` — mark the PR as ready for review\n   - `\"already reviewed at ...\"` — duplicate detection fired\n4. Check that the repo URL in `secrets.json` matches exactly (including `https://github.com/...`)\n\n### Review fails\n\n- `\"no auth configured for repo ...\"` — the repo URL in the webhook doesn't match any entry in `secrets.json`\n- `\"failed to fetch config\"` — GitHub API error fetching `.reviewotron.json` (check token permissions)\n- `\"triage agent failed\"` / `\"analysis agent failed\"` — Claude API error (check `anthropic_api_key`, rate limits)\n- `\"failed to post review\"` — GitHub API error posting the review (check token scopes: needs `repo` or `pull_request:write`)\n\n### Security findings not appearing\n\n1. Check that `review_plugins.security.enabled` is `true` in `.reviewotron.json` (it is `false` by default)\n2. Check the `confidence_threshold` — `\"high\"` is very selective. Try `\"medium\"` or `\"low\"`; for temporary high-recall tuning, add specific enabled classes to `always_analyze_vuln_classes`\n3. Check the logs for `\"triage: no actionable signals\"` (the diff may not contain security-relevant code)\n4. Check for `\"validator rejected\"` messages — the finding was detected but rejected as a false positive\n5. Bump `analysis_model_tier` to `\"strong\"` for complex codebases\n\n### Debug dumps\n\nWhen an agent produces output that can't be parsed as structured JSON, a debug dump is saved to the review debug dir. In persistent server mode, look for `debug/{repo-slug}/{sha-prefix}/` under the sibling root next to the feedback evidence root, for example `./var/debug/{repo-slug}/{sha-prefix}/`. Look here when you see `\"failed to parse ... output\"` in the logs.\n\n---\n\n## Known Issues\n\n- **No rate limiting for Anthropic API calls.** Concurrent reviews (e.g., multiple PRs opened at once) will all call the Anthropic API simultaneously. There is no built-in throttling or queue. The SDK handles 429 errors with automatic retry and exponential backoff, so transient rate limits self-heal. At typical usage (a handful of monitored repos), this is unlikely to be an issue.\n\n---\n\n## Architecture (for contributors)\n\n```\nsrc/\n  reviewotron.ml          CLI entrypoint (cmdliner: run + check commands)\n  request_handler.ml      HTTP server, webhook routing, signature validation\n\nlib/\n  api.ml                  Module type signatures (Github, Agent_runner, Slack)\n  api_remote.ml           Production implementations (real HTTP calls)\n  api_local.ml            Mock implementations (for testing)\n\n  context.ml              Application context: secrets, config cache, state\n  config_types.ml         All configuration types ([@@deriving json])\n  github_types.ml         GitHub API request/response types\n  slack_types.ml          Slack API types\n\n  github.ml               Event parsing, signature validation\n  github_auth.ml          GitHub token/JWT auth (PAT + App Installation)\n\n  reviewer.ml             Plugin orchestrator (Make functor)\n  review_plugin.ml        Plugin interface type\n  general_review_plugin.ml  General code review + validation\n  security_review_plugin.ml Multi-agent security pipeline\n\n  agent_runner.ml         Generic agent execution via ocaml-ai-sdk\n  triage_agent.ml         Triage agent config + prompt\n  analysis_agent.ml       Per-vuln-class analysis agent framework\n  validator_agent.ml      Adversarial validation agent\n  memory_curator_agent.ml Memory update curator agent\n\n  security_types.ml       All security pipeline types\n  security_tools.ml       get_file_content tool for agents\n  security_memory.ml      Memory file + queue I/O\n\n  review_types.ml         Finding, severity, review output types\n  review_format.ml        Finding → PR comment / Slack formatting\n  review_prompt.ml        General review prompt construction\n\n  cost_tracking.ml        Per-agent + per-review cost estimation\n  diff_parser.ml          Unified diff parser + path filtering\n  state.ml / state_types.ml  Persistent state (review dedup)\n  http_util.ml            HTTP request helper\n\ntest/\n  test.ml                 Main test suite (golden-file tests)\n  test_diff_parser.ml     Diff parser unit tests\n  test_security_corpus.ml Security corpus test runner (calls Claude — on-demand)\n  test_helpers.ml         Test context setup\n  mock_api_responses/     Golden-file fixtures\n  mock_payloads/          Sample webhook payloads\n  security_corpus/        Synthetic vulnerable/safe diffs per vuln class\n```\n\nThe codebase uses OCaml functors for testability - `Reviewer.Make` takes `Github`, `Agent_runner`, and `Slack` module implementations, so tests can inject mock versions (`Api_local`) without any HTTP calls.\n\n### Adding a Review Plugin\n\nThe general plugin is special — its summary becomes the review body. Every other\nplugin only emits findings, and they share one shape. To add a findings plugin:\n\n1. Write a `Make (AI : Api.Agent_runner)` functor with `name` and a `run` that\n   takes `~ctx ~repo_url ~config ~diff ~diff_text ~metadata ~debug_dir` and\n   returns `(Review_types.finding list * Cost_tracking.agent_cost list) Lwt.t`\n   (the `security_review_plugin.ml` shape).\n2. Add a config slice — a field on `review_plugins_config` in `config_types.ml`\n   (with `[@@deriving json, jsonschema]` so it shows up in `config-help`).\n3. Add one entry to the `findings_plugins` list in `review_engine.ml`\n   (`fp_name`, `fp_source`, `fp_enabled`, `fp_run`).\n\nThe engine runs all enabled findings plugins in parallel, tags each plugin's\nfindings with its `fp_source` for deduplication, and aggregates costs under\n`fp_name`. (Dedup currently privileges `From_security` on line collisions; new\nplugins use `From_general` unless they warrant the same treatment.)\n\n### Mock Agent Tests\n\nThe default test suite does not call external LLM providers. Tests instantiate plugins with `Api_local.Agent_runner`, which still exercises the production orchestration path but returns deterministic JSON from `test/mock_api_responses/` based on the agent `config.name`.\n\nThese mock-agent tests are intended to cover agent plumbing and contracts:\n\n- the expected agents are invoked in order\n- mock JSON parses against the current schemas\n- filtering, validation, deduplication, error handling, and cost tracking behave deterministically\n- accepted/rejected findings are mapped into the final review output correctly\n\nThey are not evidence that a prompt is high quality, that confidence is calibrated, or that a real model will find the right issues. Prompt quality should be measured separately with an eval corpus that runs real model calls on labeled diffs. The on-demand security corpus runner is the current pattern for that kind of provider-backed check.\n\nKeep file-based mock responses small and purposeful. Prefer adversarial fixtures that lock down one contract edge, such as a validator confirming a finding while echoing a damaged copy, over large \"realistic\" model transcripts. When a test only needs plugin-local behavior, prefer a small in-memory fake runner instead of adding another broad JSON fixture.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahrefs%2Freviewotron","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahrefs%2Freviewotron","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahrefs%2Freviewotron/lists"}