{"id":48830240,"url":"https://github.com/pmclsf/gauntlet","last_synced_at":"2026-04-14T20:02:36.798Z","repository":{"id":341944815,"uuid":"1171149850","full_name":"pmclSF/gauntlet","owner":"pmclSF","description":"Deterministic, CI-native scenario testing and quality gate for agentic systems","archived":false,"fork":false,"pushed_at":"2026-03-04T01:35:53.000Z","size":274,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-04T05:59:52.251Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pmclSF.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":"THREAT_MODEL.md","audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-02T23:27:50.000Z","updated_at":"2026-03-04T01:35:58.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/pmclSF/gauntlet","commit_stats":null,"previous_names":["pmclsf/gauntlet"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/pmclSF/gauntlet","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmclSF%2Fgauntlet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmclSF%2Fgauntlet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmclSF%2Fgauntlet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmclSF%2Fgauntlet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pmclSF","download_url":"https://codeload.github.com/pmclSF/gauntlet/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmclSF%2Fgauntlet/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31812977,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T18:05:02.291Z","status":"ssl_error","status_checked_at":"2026-04-14T18:05:01.765Z","response_time":153,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-14T20:02:30.801Z","updated_at":"2026-04-14T20:02:36.780Z","avatar_url":"https://github.com/pmclSF.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Gauntlet\n\n**Deterministic scenario testing and CI quality gates for agentic systems.**\n\nEvery PR runs the gauntlet before it ships.\n\n---\n\n## What it is\n\nGauntlet freezes the world — tools, model calls, databases — and tests your\nagent's behavior against that frozen world. When the agent code, prompt, or\nplanner changes, Gauntlet detects regressions before they ship.\n\nThis is not an eval dashboard. It is a test runner with a gate.\n\nMaintainers: release steps are documented in [RELEASING.md](RELEASING.md).\n\n---\n\n## Quickstart (10 minutes)\n\n### 1. Install\n\n```bash\n# Install with Go\ngo install github.com/pmclSF/gauntlet/cmd/gauntlet@latest\n\n# Or build from source\ngit clone https://github.com/pmclSF/gauntlet.git\ncd gauntlet \u0026\u0026 make build\nexport PATH=\"$PWD/bin:$PATH\"\n\n# Verify\ngauntlet --version\n```\n\n### 2. Add the hook to your agent\n\n```python\n# One line, at the top of your agent entrypoint\nimport gauntlet_sdk as gauntlet\ngauntlet.connect()  # no-op if Gauntlet not running; safe in production\n# connect() also enables OpenAI/Anthropic transport hooks and LangChain callback hooks when available.\n```\n\nUse `gauntlet_sdk` as the canonical import namespace to avoid conflicts with the unrelated `gauntlet` package on PyPI.\n\nOptional explicit adapter helpers (advanced):\n\n```python\nfrom gauntlet.adapters import patch_openai_client, patch_anthropic_client, patch_langchain_llm\n\nclient = patch_openai_client(client)\nanthropic_client = patch_anthropic_client(anthropic_client)\nllm = patch_langchain_llm(llm)\n```\n\n### 3. Wrap your tools\n\n```python\n@gauntlet.tool(name=\"order_lookup\")\ndef lookup_order(order_id: str) -\u003e dict:\n    # In PR CI: fixture response returned, this code never runs\n    # In production: runs normally\n    return requests.get(f\"https://api.example.com/orders/{order_id}\").json()\n```\n\n### 4. Enable CI\n\n```bash\ncd your-agent-repo\ngauntlet enable\n```\n\nThis generates:\n- `.github/workflows/gauntlet.yml` — CI workflow\n- `evals/gauntlet.yml` — policy file\n- `evals/smoke/` — starter scenario directory\n\n### 5. Create your first scenario\n\n```yaml\n# evals/smoke/order_status.yaml\nscenario: order_status_nominal\ndescription: \"User asks for order status — happy path\"\n\ninput:\n  messages:\n    - role: user\n      content: \"What's the status of my order ord-001?\"\n\nworld:\n  tools:\n    order_lookup: nominal\n  databases:\n    orders_db:\n      seed_sets: [standard_order]\n\nassertions:\n  - type: output_schema\n    schema: {type: object, required: [response], properties: {response: {type: string}}}\n  - type: tool_sequence\n    required: [order_lookup]\n  - type: tool_args_invariant\n    tool: order_lookup\n    invariant: \"args.order_id is not null\"\n```\n\n### IDE support\n\nScenario files generated by `gauntlet init` and `gauntlet validate` include:\n\n```yaml\n# yaml-language-server: $schema=https://gauntlet.dev/schema/scenario.json\n```\n\nInstall the VS Code `yaml-language-server` extension to get inline schema validation and completion.\n\n### 6. Record fixtures and establish baseline\n\n```bash\n# Record tool and model fixtures from a trusted run\nGAUNTLET_MODEL_MODE=live gauntlet record --suite smoke\n# This also signs fixtures and generates .gauntlet/fixture-signing-key.pem(.pub.pem)\n# Commit only `.gauntlet/fixture-signing-key.pem.pub.pem` (never commit private key).\n\n# Establish contract baseline\ngauntlet baseline --suite smoke\n\n# Baseline updates also emit rollback artifacts:\n#   evals/baselines/\u003csuite\u003e/rollback.manifest.json\n#   evals/baselines/\u003csuite\u003e/ROLLBACK_PR_TEMPLATE.md\n\n# When canonical hash version changes, migrate existing fixtures\ngauntlet migrate-fixtures --from-version 1 --to-version 2 --dry-run\ngauntlet migrate-fixtures --from-version 1 --to-version 2\n\n# Generate/update replay lockfile for deterministic tamper detection\ngauntlet lock-fixtures --suite smoke\n\n# Sign run artifacts with an evidence manifest (also done by generated CI workflow)\ngauntlet sign-artifacts --dir evals/runs\n```\n\n### 7. Run the suite\n\n```bash\ngauntlet run --suite smoke\n# Optional: strict policy parsing (unknown keys are hard errors)\ngauntlet run --policy-strict --suite smoke\n# Explicit runner/model mode separation:\ngauntlet run --suite smoke --runner-mode pr_ci --model-mode recorded\n# Add deterministic per-scenario timeout budget (ms):\ngauntlet run --suite smoke --budget 300000 --scenario-budget 45000\n\n# Preflight checks for policy, modes, proxy trust, fixtures, and egress:\ngauntlet doctor --suite smoke\n```\n\n### 8. Push and watch CI gate your PR\n\n---\n\n## Integration levels\n\nChoose the level that fits your setup:\n\n| Level | What you need | What you get |\n|---|---|---|\n| **Best** | HTTP endpoint + `gauntlet.connect()` | Full scenario testing, tool traces, model replay |\n| **Good** | CLI entrypoint + `gauntlet.connect()` | Full scenario testing, tool traces, model replay |\n| **Minimal** | Just a CLI entrypoint | Egress enforcement + exit code gate + budget enforcement |\n\nMinimal still provides real value. Even without structured traces, blocking network\negress and enforcing a time budget catches a class of regressions most CI setups miss.\n\n## Examples\n\n- [Support agent](examples/support-agent/README.md) - customer support workflows with smoke/nightly suites\n- [RAG agent](examples/rag-agent/README.md) - retrieval world states with citation assertions\n- [Multi-tool orchestrator](examples/orchestrator/README.md) - chained tools with retry caps\n- [Code agent](examples/code-agent/README.md) - sandbox execution with forbidden-content checks\n- [Intentional failure](examples/intentional-failure/README.md) - scenario that fails on purpose to inspect artifacts\n\n## SDK capability negotiation\n\nGauntlet SDKs emit a versioned `sdk_capabilities` handshake (`protocol_version: 1`)\ninto the trace stream. The runner consumes this negotiation report and emits a soft\n`adapter_capabilities` diagnostic when:\n\n- capability negotiation is missing\n- protocol version is unsupported\n- an adapter is enabled but runtime patching failed\n\nThis keeps CI deterministic while still surfacing integration drift early.\n\n---\n\n## Scenario format\n\n```yaml\nscenario: \u003cunique_name\u003e\ndescription: \"\u003chuman readable description\u003e\"\n\ninput:\n  messages:                        # OpenAI-format messages\n    - role: user\n      content: \"...\"\n  # OR\n  payload:                         # Arbitrary JSON payload\n    key: value\n\nworld:\n  tools:\n    \u003ctool_name\u003e: \u003cstate_name\u003e      # e.g. order_lookup: nominal\n  databases:\n    \u003cdb_name\u003e:\n      seed_sets: [\u003cseed_name\u003e]\n\nassertions:\n  - type: output_schema\n    schema: \u003cJSON Schema object\u003e\n  - type: tool_sequence\n    required: [tool_a, tool_b]     # must appear in this order\n    forbidden: [tool_c]            # must NOT appear\n  - type: retry_cap\n    tool: order_lookup\n    max_retries: 2\n  - type: tool_args_invariant\n    tool: order_lookup\n    invariant: \"args.order_id == input.order_id\"\n  - type: forbidden_content\n    pattern: \"(?i)api[_-]?key|secret\"\n    fields: [output]\n```\n\n---\n\n## World definitions\n\n### Tool state envelope\n\n```yaml\n# evals/world/tools/order_lookup.yaml\ntool: order_lookup\n\nstates:\n  nominal:\n    response:\n      order_id: \"ord-001\"\n      status: \"confirmed\"\n      total_cents: 4999\n    agent_expectation: \"completes normally\"\n\n  timeout:\n    delay_ms: 8000\n    agent_expectation: \"retries once, then surfaces error\"\n\n  server_error:\n    response_code: 500\n    agent_expectation: \"retries with backoff, caps at 2\"\n```\n\n### DB seed definition\n\n```yaml\n# evals/world/databases/orders_db.yaml\ndatabase: orders_db\nadapter: sqlite\n\nseed_sets:\n  standard_order:\n    tables:\n      orders:\n        rows:\n          - id: \"ord-001\"\n            status: \"confirmed\"\n            total_cents: 4999\n```\n\nSingle-fault policy counts both non-nominal tool states and non-nominal DB seed variants.\nSet `chaos: true` in a scenario to allow multi-fault combinations intentionally.\n\n## Mode semantics\n\n- `runner_mode` / `--runner-mode`: execution context (`local`, `pr_ci`, `fork_pr`, `nightly`)\n- `model_mode` / `--model-mode`: fixture behavior (`recorded`, `live`, `passthrough`)\n- `mode` / `--mode`: legacy compatibility alias; prefer explicit runner/model fields and flags\n- `proxy.mode`: model replay mode only (`recorded`, `live`, `passthrough`)\n\n## Exit codes\n\nGauntlet CLI exit codes are a stable API:\n\n- `0` (`ExitSuccess`): all scenarios passed\n- `1` (`ExitFailure`): one or more scenarios failed assertions\n- `2` (`ExitError`): Gauntlet encountered an execution/runtime error\n- `3` (`ExitInvalidInput`): invalid flags, missing files, or invalid YAML/schema input\n\n---\n\n## CI behavior\n\n### PR CI (hermetic)\n- Zero network egress — enforced at process level\n- Mandatory outbound socket egress self-test before scenario execution\n- Recorded replay mode verifies `evals/fixtures/replay.lock.json` integrity before execution\n- Replay lockfile/canonical hash determinism is snapshot-tested against a cross-platform runtime matrix (`linux/darwin/windows`, `amd64/arm64`)\n- Recorded replay mode enforces fixture signatures against `.gauntlet/fixture-signing-key.pem.pub.pem`\n- Generated CI workflow runs `gauntlet scan-artifacts` and `gauntlet sign-artifacts` before upload-artifact\n- `scan-artifacts` includes a default prompt-injection marker denylist for recorded artifacts (opt-out: `redaction.prompt_injection_denylist: false`)\n- Optional hard allowlist for recorder identities via `GAUNTLET_TRUSTED_RECORDER_IDENTITIES`\n- Tool and model calls served from fixtures\n- Hard/soft assertion gating enforced from `evals/gauntlet.yml` (`assertions.hard_gates`, `assertions.soft_signals`)\n- Baseline-changing PRs must carry label `gauntlet/baseline-approved` (enforced by `gauntlet check-baseline-approval`)\n- Fork PRs: replay-only, no secrets, no judge calls\n- Target: \u003c 5 minutes\n- Proxy enforces deterministic request parsing limits (header/body/request-count per connection)\n- Malformed JSON and unsupported websocket/HTTP2 tunnel traffic return explicit proxy error codes\n- Proxy CA assets require hardened permissions; doctor reports rotation warning before cert expiry\n- CLI failures emit canonical machine-parseable error code lines (`GAUNTLET_ERROR_CODE=\u003ccode\u003e`)\n- Fixture misses include provider/model/hash context plus nearest recorded fixture candidates\n- Results include deterministic per-scenario causal taxonomy (`failure_category`) and effective scenario budgets\n- Results include lightweight run-history metadata (`history.previous`, `history.delta`) for regression velocity tracking\n- Per-scenario runtime determinism checks verify timezone/locale freeze application when SDK reports `determinism_env`\n- Non-Python SDKs emit explicit nondeterminism guard warnings until runtime freeze verification parity is available\n- Optional per-scenario TUT process limits available via `tut.resource_limits` (`cpu_seconds`, `memory_mb`, `open_files`)\n- Optional linux hostile-payload hardening via `tut.guardrails.hostile_payload` (+ `max_processes`)\n\n### Nightly (trusted)\n- Live model calls (secrets available)\n- Fixture re-recording\n- Proposes baseline update PR if behavior changed\n- Full suite (no time constraint)\n\n---\n\n## Failure output\n\nEvery failure produces a self-contained artifact:\n\n```\nFAILED  order_status_conflicting_payment\n\nCulprit: db.seed.conflicting_state\nConfidence: high\n\nFailing assertion:\n  tool_sequence\n  Expected: [order_lookup, payment_lookup]\n  Actual:   [order_lookup]              \u003c- payment_lookup never called\n\nWorld state:\n  tools:     order_lookup -\u003e nominal\n  databases: orders_db -\u003e conflicting_state\n               orders.ord-007.status   = \"confirmed\"\n               payments.ord-007.status = \"failed\"  \u003c- conflict not surfaced\n\nBaseline output: \"Your order shows confirmed but the payment failed...\"\nPR output:       \"Your order ord-007 is confirmed.\"\n\nDocket tag: planner.premature_finalize\n```\n\n---\n\n## How Gauntlet differs from PromptFoo and LangSmith\n\n**PromptFoo** is excellent for prompt evaluation and red-teaming. It runs batches of\ninputs against prompts and scores outputs. It does not freeze world state, does not\nenforce egress, and does not gate CI on behavioral contracts.\n\n**LangSmith** is excellent for tracing and debugging LangChain applications. It\nrequires a cloud account and is tightly coupled to the LangChain ecosystem.\n\n**Gauntlet** does one thing they don't: it freezes the world — tools, databases,\nmodel responses — and tests your agent against that frozen world on every PR.\nThe gate is binary. Either your agent's behavior matches the contract, or the PR\ndoesn't ship. No cloud account required. No framework required.\n\nIf you want to evaluate prompts: use PromptFoo.\nIf you want to trace LangChain apps: use LangSmith.\nIf you want CI to catch behavioral regressions before they ship: use Gauntlet.\n\n---\n\n## FAQ\n\n**Is this an eval platform?**\nNo. Gauntlet does not have a dashboard, scoring UI, or human labeling workflow.\nIt is a test runner that gates CI. Think pytest for agent behavior.\n\n**Why not LangSmith / vendor X?**\nThose are observability and eval platforms — valuable for different things.\nGauntlet is CI-native, offline-capable, and enforces contracts.\nIt does not require a cloud account to run.\n\n**Does it work with my framework?**\nIf your agent can be invoked via HTTP or CLI, it works with Gauntlet.\n`gauntlet.connect()` has adapters for OpenAI SDK, Anthropic SDK, and LangChain.\nOther frameworks work via the CLI adapter or HTTP endpoint.\n\n**What about multi-agent systems?**\nMulti-agent is on the roadmap. v1 tests single-agent behavior.\nFor multi-agent: define each agent as a separate TUT with its own suite.\n\n---\n\n## Contributing\n\nThe most useful contributions right now:\n- New tool state patterns (open an issue with the pattern)\n- Framework adapters in `sdk/python/gauntlet/adapters/`\n- Additional assertion types in `internal/assertions/`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpmclsf%2Fgauntlet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpmclsf%2Fgauntlet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpmclsf%2Fgauntlet/lists"}