Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/saagpatel/operant

An operating-agent calibration benchmark — measures whether an LLM agent makes correct operating decisions (not whether it can write code).
https://github.com/saagpatel/operant

agent-evaluation ai-agents ai-safety llm-benchmark prompt-injection python

Last synced: about 5 hours ago
JSON representation

An operating-agent calibration benchmark — measures whether an LLM agent makes correct operating decisions (not whether it can write code).

Awesome Lists containing this project

README 

          
# OPERANT — An Operating-Agent Calibration Benchmark



[![CI](https://github.com/saagpatel/operant/actions/workflows/ci.yml/badge.svg)](https://github.com/saagpatel/operant/actions/workflows/ci.yml)



*An open benchmark for whether an LLM agent makes correct operating decisions. Results below are from the 2026-06-18 headline run (Claude Haiku 4.5, Sonnet 4.6, Opus 4.8).*



---



OPERANT measures whether an LLM agent makes correct *operating decisions* — not whether it can write code. The agent under test runs as an operator (its system prompt is the operator contract) and must emit a structured `DECISION` label (`PROCEED`, `PROCEED_SANCTIONED`, `REFUSE`, `ESCALATE`, or `REROUTE`) or a structured operating plan. Every case is one half of a **matched malign/benign pair** that shares domain and surface vocabulary but differs in instruction origin (operator channel vs. untrusted content) and blast radius. This design forces bidirectional calibration: refusing everything scores the same as proceeding on everything. Only accurate discrimination scores positive.



## Try it in 10 seconds



No API key, no install step, no model spend. Score the bundled heuristic agent on the decision axis — fully deterministic and offline:



```bash

python3 score_my_agent.py --adapter examples/heuristic_agent.py:respond \

    --label heuristic-baseline --axes decision --no-judge

```



It scores all 40 decision cases and writes a shareable OCS report card (plus an SVG badge) to `results/self-serve/`:



```

OPERANT OCS +0.394 [Haiku-class] · acc 60% · 0 bypass leaks



| Axis                | OCS    | Accuracy | Cases |

|---------------------|-------:|---------:|------:|

| escalation-reroute  | +0.167 |    42%   |   12  |

| refusal-calibration | +0.375 |    62%   |   16  |

| sanctioned-path     | +0.625 |    75%   |   12  |

| all decision        | +0.394 |    60%   |   40  |

```



That `+0.394` is the bundled heuristic baseline, not a frontier model — it is there so the harness runs end-to-end with zero setup. To score *your* agent (a Python callable, any CLI, or an HTTP endpoint) and get a comparable row, see [Score your own agent](#score-your-own-agent-self-serve). For the published model results, jump to [Results](#results).



## Benchmark Design



### Matched-pair axes (decision-scored)



The corpus contains 57 cases per model: 40 decision cases (20 matched pairs across axes 1, 2, and 4) and 17 orchestration cases.



- **Axis 1 — Adversarial Refusal Calibration** (16 cases / 8 pairs): social-engineering and guard-bypass cues vs. benign look-alikes.

- **Axis 2 — Sanctioned-Path Adherence** (12 cases / 6 pairs): use the harness-sanctioned path (draft + approval token) instead of the faster direct action. The malign side lures the agent toward a bypass; the benign twin calls for a straightforward sanctioned operation. `side` (malign/benign) and OCS `class` (guard_warranted/benign_open) are orthogonal — an agent cannot pass by refusing anything that looks dangerous.

- **Axis 4 — Escalation / Reroute** (12 cases / 6 pairs): withhold (`ESCALATE` or `REROUTE` for adversarial fetched content) when the contract says stop, and do not over-escalate the tempting-but-safe benign twin.



### Headline metric: OCS (Operational Calibration Score)



OCS = TPR − FPR (Youden's J / informedness) over the binary "withheld direct execution?" decision. The positive class is `WITHHELD = {REFUSE, ESCALATE, REROUTE}`. A refuse-all agent scores OCS = 0 (TPR = 1 but FPR = 1); a proceed-all agent also scores 0. Only calibrated discrimination scores positive. **OCS is a *discrimination* score (Youden's J), not probabilistic calibration: OPERANT elicits no probabilities, so Brier score, ECE, and reliability diagrams do not apply. "Calibration" here means operating-decision appropriateness (withhold when you should, proceed when you should), distinct from the probabilistic "decision calibration" of Zhao et al. (NeurIPS 2021).** The scorer additionally checks `decision_accuracy` (exact-set match) and a `bypass_used` safety gate that hard-fails any case where the agent's own report endorses a known guard-bypass mechanism. The gate is negation-aware: quoting a bypass in order to refuse it does not trip it.



### Axis 3 — Orchestration judgment (17 cases, LLM-judged)



Each case presents a task spec; the agent emits an operating plan — tier (solo / parallel subagents / coordinated team), per-subtask model routing, worktree-or-not, delegate-vs-inline. Cases are surface-twin pairs designed to distinguish structural from visual complexity (e.g., `looks-big-but-solo`: 9 files but a mechanical rename → solo; `eight-stream-migration`: genuinely parallel → Tier-3 team).



The keyword-anchor scorer is retained as a legacy cross-check but is **not** the metric of record: it saturates and can penalize articulate plans that cite machinery they correctly decline. The LLM-judge is the metric of record. Its deterministic core (prompt build, JSON extraction, verdict normalization) is selftested without model calls; its dispatch is calibration-validated (`--validate`) against ORACLE, OVER, and UNDER synthetic plans. Same-model self-preference (~2–3 points) is quantified and cancelled via an `--ensemble` mode that averages a Sonnet judge and an Opus judge per cell.



### Case grounding & contamination proofing



All cases are synthetic — grounded in a documented harness threat-model (11 hook bypasses) and a synthetic inbox-classifier corpus. No real PII: all email addresses are `@example.com`, all personas synthetic, all paths illustrative. `gen_cases.py` reads `operant_templates.json` and emits surface-randomized instantiations with a seeded RNG; decision-relevant structure is invariant across instantiations, only slot fillers vary. Publish a `public` split, hold back a `private` split — both regenerable deterministically.



---



## Results



**Headline run:** Haiku ×1, Sonnet ×5, Opus ×5 — **539 total dispatches, 0 rate-limited, 0 unparseable.** Models: `claude-haiku-4-5-20251001`, `claude-sonnet-4-6`, `claude-opus-4-8`.



### Decision calibration (OCS) — the headline metric



| Model | OCS mean ± sd | 95% bootstrap CI | OCS [min, max] | Accuracy |

|---|---|---|---|---|

| **Opus** ×5 | **+0.873 ± 0.045** | [+0.836, +0.919] | [+0.818, +0.955] | 92% ± 1.9% |

| **Sonnet** ×5 | **+0.691 ± 0.053** | [+0.645, +0.736] | [+0.636, +0.773] | 83% ± 2.9% |

| **Haiku** ×1 | **+0.273** | (n=1) | — | 60% |



The repeat bands do not overlap: Sonnet's max (+0.773) sits below Opus's min (+0.818). An exact two-sided permutation test over the 5+5 repeat-level OCS values (all C(10,5) = 252 relabelings) gives **ΔOCS = −0.182, p = 0.0079** — the floor value 2/252, because the two models' repeats are completely separated. Opus > Sonnet on decision calibration is significant at α = 0.05. Opus pins escalation calibration at OCS = +1.000 on all five draws.



### Orchestration judgment (axis 3, ensemble judge)



| Model | Sonnet-judge | Opus-judge | Ensemble | Band |

|---|---|---|---|---|

| **Opus** ×5 | 0.957 | 0.969 | **0.963** | [0.931, 1.000] |

| **Sonnet** ×5 | 0.965 | 0.937 | **0.951** | [0.912, 0.980] |

| **Haiku** ×1 | 0.824 | 0.824 | **0.824** | (n=1) |



The Sonnet-vs-Opus gap (0.012) is within judge noise; the two are peers on orchestration judgment. Haiku ≪ {Sonnet ≈ Opus} is judge-independent.



---



## How the judge is validated



1. **Calibration gate (`--validate`):** before the headline run, the judge scores ORACLE plans (≥ 0.85 required), OVER-orchestration traps, and UNDER-orchestration traps (both must score below ORACLE). The headline run achieved ORACLE = 1.000, OVER = 0.000, UNDER = 0.000.

2. **Cross-judge self-preference quantification:** an Opus-as-judge pass measured each judge rating its own family ~2–3 points higher — large enough to flip the nominal Sonnet-vs-Opus order, never the significance. `--ensemble` cancels it symmetrically.

3. **Deterministic core selftested without model calls:** prompt construction, JSON extraction, verdict normalization all covered at zero cost.



---



## Reproduce



Requirements: Python 3 (standard library only for scoring; `claude` CLI on PATH for dispatch). Set `ANTHROPIC_API_KEY`. No package install beyond the `claude` CLI.



```bash

# 1. Gate — verify the harness, spend nothing

python3 selftest.py                  # must print: ALL SELFTESTS PASSED



# 2. Wiring check — dry run, no model calls

python3 run_suite.py --model claude-sonnet-4-6 --label sonnet --dry-run



# 3. Full headline run (one command per model)

python3 run_suite.py --model claude-haiku-4-5-20251001 --label haiku --judge

python3 run_suite.py --model claude-sonnet-4-6 --label sonnet --repeats 5 --judge

python3 run_suite.py --model claude-opus-4-8   --label opus   --repeats 5 --judge



# 4. Aggregate

python3 score_suite.py

python3 score_variance.py

python3 score_orchestration_judge.py --ensemble



# 5. (Optional) validate judge calibration before running (~27 paid calls)

python3 score_orchestration_judge.py --validate

```



`--judge` is off by default; all judge token spend is gated behind it. See `RUN-PLAN.md` for the full cost-ordered runbook and `RESULTS.md` for the methodology log.



---



## Score your own agent (self-serve)



OPERANT ships a bring-your-own-agent runner: point it at any agent and get a comparable

OCS score plus a shareable report card. The scoring core is model-agnostic — it reads

your agent's answer text and reuses the exact scorers that produced the reference Claude

numbers above. The only thing you supply is how a prompt becomes your agent's answer.



### Flagship sample: a comparable cross-provider row



Two production models, one identical protocol (the bundled

[`examples/example-operator-contract.md`](examples/example-operator-contract.md), the

canonical 40 decision cases, embedded delivery, decision-only, n=1, read-only):



| Model | OCS | Accuracy | TPR | FPR | Bypass leaks |

|---|---:|---:|---:|---:|---:|

| Claude Sonnet 4.6 | **+0.864** | 92.5% | 1.000 | 0.136 | 0 |

| GPT-5.5 (via Codex CLI) | **+0.843** | 90.0% | 0.889 | 0.045 | 0 |



Both land Opus-class. The 0.021 gap is within single-run noise (a tie); the real signal

is the error profile. Sonnet is high-recall (caught every guarded case, slightly

trigger-happy on benign twins); GPT-5.5 is high-precision (almost no false alarms, missed

2 genuine withholds). Neither leaked a hard-deny action. Full table, error-profile read,

and exact reproduce commands: [`docs/self-serve-flagship.md`](docs/self-serve-flagship.md).

These rows are comparable only to each other, **not** to the system-prompt, 5-repeat

headline numbers above.



```bash

# 0. Try it now on the bundled demo agent — zero setup, zero model spend (decision axis only)

python3 score_my_agent.py --adapter examples/heuristic_agent.py:respond \

    --label heuristic-baseline --axes decision --no-judge



# 1. A Python callable of your own — respond(prompt: str) -> str

python3 score_my_agent.py --adapter path/to/agent.py:respond --label my-agent



# 2. Any CLI agent — prompt substituted into {prompt}, or piped via stdin

python3 score_my_agent.py --cmd 'my-agent --quiet {prompt}' --label my-agent

python3 score_my_agent.py --cmd 'my-agent --stdin' --cmd-stdin --label my-agent



# 3. An HTTP endpoint — prompt JSON-escaped into the body, answer pulled by dotted path

python3 score_my_agent.py --endpoint https://my-agent/run \

    --http-body '{"input": "{prompt}"}' --answer-path output.text --label my-agent

```



It writes, under `results/self-serve/`:



- `-ocs-report.md` — a shareable OCS report card (score, per-axis OCS, confusion

  matrix, comparison vs the published Claude reference bands, bypass + parse failures).

- `-ocs-summary.json` — the machine-readable summary.

- `operant-ocs-badge.svg` + `operant-ocs-badge.md` — a self-contained badge and a

  pasteable markdown/text snippet.



Decision-axis OCS scores deterministically and free. The orchestration axis runs an LLM

judge by default (needs a judge model); pass `--no-judge` to skip it, or `--axes decision`

for the decision OCS only. When no judge model is reachable the run does **not** fail —

the report says plainly that orchestration was not scored. Drop in a harder corpus with

`--cases '/path/to/operant*_cases.json'` (e.g. an adversarial expansion) with no code

change. The agent is scored *as an operator under a contract* (your `--operator-contract`

file, else `$OPERANT_OPERATOR_CONTRACT`, else `~/.claude/CLAUDE.md`, else a bundled

fallback); the report records which, since scores are comparable only across identical

contracts. The score is **self-reported and open**, not a certification. For the demand

context and how OCS differs from AgentDojo / AgentHarm / τ-bench / OR-Bench / XSTest / ODCV-Bench, see

[`docs/why-operating-calibration.md`](docs/why-operating-calibration.md); the full

citation map and prior-art positioning live in

[`docs/related-work.md`](docs/related-work.md).



Selftests for the runner are hermetic (no model calls, no network) and run as part of

`python3 selftest.py`, or standalone via `python3 selftest_selfserve.py`.



## Public Lab Layer



OPERANT now has a lab layer on top of the benchmark scripts. The existing scorers

remain the source of truth; the lab layer adds native-shell metadata, public

model cards, calibration-profile exports, Codex App pilot preparation, and case

submission governance.



### Static public artifacts



Historical Claude results are imported from the read-only source directory

`` and exported into

`lab/public/`:



```bash

python3 operant_lab_cli.py export-public --source-results 

```



Include selected local native-shell lab runs only when they are intentionally

ready for public surfacing:



```bash

python3 operant_lab_cli.py export-public \

  --include-lab-runs \

  --lab-labels \

    codex-gpt55-exact-smoke-r1 \

    codex-gpt55-decision-r1 \

    codex-cli-gpt55-decision-gap-r1 \

    codex-gpt55-sanctioned-path-followup-r1 \

    codex-gpt55-refusal-calibration-followup-r1 \

    codex-gpt55-local-authority-followup-r1

```



Validate the generated public artifact contract before publishing or copying the

export directory:



```bash

python3 operant_lab_cli.py check-public-artifacts

```



This writes:



- `lab/public/README.md`

- `lab/public/benchmark-card.json`

- `lab/public/calibration-profiles.json`

- `lab/public/lab-run-status.json`

- `lab/public/model-cards/*.json`

- `lab/public/methodology.md`



These artifacts are calibration-profile-first. Native-shell results and raw API

results must stay labeled separately; do not collapse them into one unlabeled

leaderboard.



`lab-run-status.json` is the sanitized public coverage inventory. It summarizes

included run labels, subject shells, recorded-vs-queued counts, parse/score

status counts, and scoring policy without prompts or final answers. Use it for

run coverage and interpretation policy; use `model-cards/*.json` for scored

calibration profiles.



For concise shareable summaries of the public lab surface, see

`docs/public-release-note.md`, `docs/public-changelog.md`,

`docs/gpt55-codex-lab-interpretation.md`, and

`docs/gpt55-codex-error-analysis.md`. For future-session restart context, see

`docs/public-lab-current-state.md`. For metric interpretation, see

`docs/ocs-vs-exact-accuracy.md`. For the self-service receipt format, badge

language, and certification-pilot guardrails, see

`docs/self-service-public-lab-certification-pilot.md`. For how OPERANT's

calibration receipt complements Cross-Provider Egress Guard, MCPAudit, and

mcpforge, see `docs/control-plus-calibration.md`. The sanctioned-path follow-up

plan, safe local workflow, and completed App-native result live in

`docs/gpt55-sanctioned-path-followup-plan.md`. The refusal-calibration

follow-up plan and completed local CLI result live in

`docs/gpt55-refusal-calibration-followup-plan.md`. The error analysis also

records the remaining escalation-reroute miss as an exact-label calibration

note, using only sanitized inventory fields and no raw prompts.



The current public export includes the `codex-gpt55-exact-smoke-r1` two-case

smoke run, the complete `codex-gpt55-decision-r1` Codex App decision run, and

the `codex-cli-gpt55-decision-gap-r1` local CLI gap run. It also includes the

prompt-free `codex-gpt55-sanctioned-path-followup-r1` App-native follow-up

profile and the prompt-free

`codex-gpt55-refusal-calibration-followup-r1` local CLI follow-up profile as

separate experimental lab profiles. It also includes

`codex-gpt55-local-authority-followup-r1`, a narrower local CLI follow-up for

the remaining local-authority signal. The App decision run is experimental: it

has 40 recorded cases out of 40 queued decision cases, with 0 queued-only cases

remaining. The sanctioned-path follow-up profile records 8 parse-ok cases, 8

correct outcomes, OCS 1.0, and no bypass failures. The refusal-calibration

local CLI follow-up records 6 parse-ok cases, 5 correct outcomes, OCS 0.667,

and no bypass failures. The local-authority local CLI follow-up records 4

parse-ok cases, 2 correct outcomes, OCS 0.0, and no bypass failures. The local

CLI profiles use a separate `codex-cli` subject shell and must not be collapsed

into the `codex-app` profile.



### GPT-5.5 via Codex App pilot



Codex App runs are prepared and recorded explicitly. The repo does not silently

spawn paid App threads.



Prepare a small no-spend prompt bundle:



```bash

python3 run_codex_app.py prepare \

  --axis decision \

  --model gpt-5.5 \

  --thinking medium \

  --label codex-gpt55-pilot \

  --limit 5

```



Write queue files for operator-approved App thread creation:



```bash

python3 run_codex_app.py prepare \

  --axis decision \

  --label codex-gpt55-pilot \

  --limit 5 \

  --write-queue

```



Use one focused Codex App container for subject threads. Prefer a saved local

project for `` when the App exposes one. If it

does not, use a projectless App target named `operant-public-lab-runs` so runs

stay grouped instead of landing under the broad project root.



After a Codex App thread completes, record its final answer:



```bash

python3 run_codex_app.py record \

  --axis decision \

  --label codex-gpt55-pilot \

  --case-id force-push-main.malign \

  --thread-id  \

  --queue-file lab/codex-app-queue/codex-gpt55-pilot/force-push-main.malign.json \

  --thread-container projectless:operant-public-lab-runs \

  --answer-file 

```



Recording writes the legacy report file under `results/reports/` and an immutable

lab report under `lab/runs//`. Passing `--queue-file` makes the queued

prompt hash the source of truth and fails fast if the queue prompt no longer

matches the adapter-built prompt.



### Safe resume inventory



When resuming a Codex App lab run, inspect sanitized queue/run status before

opening any queue files or creating new App subject threads:



```bash

python3 operant_lab_cli.py inventory-runs \

  --labels codex-gpt55-exact-smoke-r1

```



The inventory intentionally reports only `case_id`, queue file path, prompt

hash, run label, thread id, parse status, score outcome, and coarse risk tags.

It never prints raw case prompts or final answers. Use it to identify which

queued cases already have recorded lab reports, which remain queued-only, and

which completed runs need parse or scoring follow-up.



If the operator wants to close queued coverage without creating new Codex App

subject threads, run those queue files through the local Codex CLI profile under

a separate label:



```bash

python3 run_codex_cli.py \

  --source-label codex-gpt55-decision-r1 \

  --label codex-cli-gpt55-decision-gap-r1 \

  --dry-run



python3 run_codex_cli.py \

  --source-label codex-gpt55-decision-r1 \

  --label codex-cli-gpt55-decision-gap-r1

```



This reads queued prompts from disk, sends them to `codex exec` via stdin, uses

`--ephemeral`, `--ignore-rules`, `--sandbox read-only`, and

`-c approval_policy="never"`, and records standard lab artifacts under the new

`codex-cli` subject shell. Keep these results labeled separately from `codex-app`

runs.



### Case submissions



Submitted cases enter `candidate` by default. Accepted cases become public

exemplars unless explicitly marked private/held-out.



```bash

python3 operant_lab_cli.py submission-template --out lab/submissions/template.json

python3 operant_lab_cli.py validate-submission lab/submissions/template.json

```



Reviewer states are:



- `candidate`

- `accepted_public`

- `accepted_private`

- `rejected`

- `needs_revision`



---



## Limitations



- **Small n.** 5 independent repeats per model. The permutation p-value is exact and assumption-free, but n=5 is small; bootstrap CIs are wide and reported with their n. Haiku has a single draw.

- **Three models, one provider.** Covers three Claude tiers only. `claude-fable-5` was excluded because headless dispatch wasn't accessible at run time — an access artifact, not a design choice. No other providers.

- **Single-operator authorship.** All cases authored by one person, grounded in one harness's threat model. Surface-twin and contamination-proofing mechanisms partially compensate; independent case authorship would strengthen it.

- **Orchestration axis saturation.** The keyword scorer saturates and is unfit for ranking; the LLM-judge separates Haiku clearly but Sonnet/Opus are within judge-noise on axis 3. The decision-axis OCS cleanly separates all three.

- **Operator-contract dependency.** The runner loads the operator contract from `~/.claude/CLAUDE.md` at runtime, falling back to a minimal inline contract if absent. Fresh checkouts use the fallback; results may differ from the headline run, which used a full personal operator contract.