{"id":30910244,"url":"https://github.com/geminimir/promptproof-action","last_synced_at":"2026-04-07T04:31:27.066Z","repository":{"id":309455774,"uuid":"1035952764","full_name":"geminimir/promptproof-action","owner":"geminimir","description":"Deterministic LLM contract checks for CI. Replays recorded fixtures, enforces schema/regex/budget rules, generates HTML/JUnit/JSON reports, and comments on PRs. Zero live model calls in CI.","archived":false,"fork":false,"pushed_at":"2025-09-02T03:55:36.000Z","size":259,"stargazers_count":10,"open_issues_count":9,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-09T18:46:43.838Z","etag":null,"topics":["budgets","ci","fixtures","github-action","json-schema","llm","redaction","regex","security","testing"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/geminimir.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-11T10:43:55.000Z","updated_at":"2025-09-04T22:28:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"1b99bdc6-c6e7-4e15-aae8-596f7ee612cb","html_url":"https://github.com/geminimir/promptproof-action","commit_stats":null,"previous_names":["geminimir/promptproof-action"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/geminimir/promptproof-action","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geminimir%2Fpromptproof-action","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geminimir%2Fpromptproof-action/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geminimir%2Fpromptproof-action/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geminimir%2Fpromptproof-action/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/geminimir","download_url":"https://codeload.github.com/geminimir/promptproof-action/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geminimir%2Fpromptproof-action/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31500397,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T03:10:19.677Z","status":"ssl_error","status_checked_at":"2026-04-07T03:10:13.982Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["budgets","ci","fixtures","github-action","json-schema","llm","redaction","regex","security","testing"],"created_at":"2025-09-09T17:03:26.898Z","updated_at":"2026-04-07T04:31:27.060Z","avatar_url":"https://github.com/geminimir.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PromptProof GitHub Action\n\n[![CI](https://img.shields.io/github/actions/workflow/status/geminimir/promptproof-action/ci.yml?branch=main)](https://github.com/geminimir/promptproof-action/actions)\n[![Marketplace](https://img.shields.io/badge/Marketplace-PromptProof%20Eval-blue?logo=github)](https://github.com/marketplace/actions/promptproof-eval)\n[![Website](https://img.shields.io/badge/Website-promptproof.io-darkgreen)](https://promptproof.io)\n[![Node](https://img.shields.io/badge/node-%3E%3D18-brightgreen)](https://nodejs.org)\n[![version](https://img.shields.io/github/v/tag/geminimir/promptproof-action?label=version)](https://github.com/geminimir/promptproof-action/releases)\n[![View sample report](https://img.shields.io/badge/Report-Sample-green)](https://geminimir.github.io/promptproof-action/reports/after.html)\n\nDeterministic LLM testing in your CI/CD pipeline. This action evaluates recorded LLM outputs against defined contracts and fails PRs when violations are detected.\n\n## Features\n\n- **Zero network calls** - Tests run on recorded fixtures\n- **Rich reporting** - HTML, JUnit, JSON output formats\n- **PR comments** - Automatic violation summaries\n- **Budget tracking** - Cost and latency monitoring\n- **Flexible checks** - JSON schema, regex, numeric bounds, string contains/equals, list/set equality, file diff, custom functions\n\n## See it in action\n\n- Regression fail PR · Cost gate PR · Assertion fail PR\n  \n  [Links to live PRs and GIFs to be inserted after publishing]\n\n### Report preview\n\nBelow are example screenshots of the HTML report generated by this action.\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd width=\"50%\"\u003e\n      \u003cstrong\u003eBefore\u003c/strong\u003e\u003cbr/\u003e\n      \u003cimg src=\"assets/report-before.png\" alt=\"Report - Before\" width=\"100%\" /\u003e\n    \u003c/td\u003e\n    \u003ctd width=\"50%\"\u003e\n      \u003cstrong\u003eAfter\u003c/strong\u003e\u003cbr/\u003e\n      \u003cimg src=\"assets/report-after.png\" alt=\"Report - After\" width=\"100%\" /\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003c/table\u003e\n\n## Quick Start\n\n```yaml\nname: PromptProof\non: [pull_request]\n\njobs:\n  eval:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: geminimir/promptproof-action@v0\n        with:\n          config: promptproof.yaml\n          baseline-ref: origin/main\n          runs: 3\n          seed: 1337\n          max-run-cost: 2.50\n          report-artifact: promptproof-report\n          mode: gate\n```\n\n## Inputs\n\n| Input | Description | Default |\n|-------|-------------|---------|\n| `config` | Path to promptproof.yaml | `promptproof.yaml` |\n| `baseline-ref` | Git ref to load baseline snapshot from (e.g., `origin/main`) |  |\n| `runs` | Number of runs for flake control |  |\n| `seed` | Seed for flake control determinism |  |\n| `max-run-cost` | Maximum total cost for this run (USD) |  |\n| `report-artifact` | Name of uploaded report artifact | `promptproof-report` |\n| `mode` | `gate` (fail) or `report-only` (warn). Defaults to config. |  |\n| `format` | Output format (`html`|`junit`|`json`|`console`|`sarif`) | `html` |\n| `regress` | Also compare to local baseline | `false` |\n| `node-version` | Node.js version | `20` |\n| `snapshot-on-success` | Create snapshot after successful run | `false` |\n| `snapshot-promote-on-main` | Promote snapshot to baseline on main | `false` |\n| `snapshot-tag` | Optional snapshot tag |  |\n\n## Outputs\n\n| Output | Description |\n|--------|-------------|\n| `violations` | Number of violations found |\n| `passed` | Number of fixtures that passed |\n| `failed` | Number of fixtures that failed |\n| `failed-tests` | Alias for `failed` |\n| `total-cost` | Total cost (USD) of this evaluation |\n| `regressions` | New failures vs baseline (when regression comparison is enabled) |\n| `report-path` | Path to generated report |\n\n## Configuration\n\nCreate a `promptproof.yaml` file in your repository:\n\n```yaml\nschema_version: pp.v1\nfixtures: fixtures/outputs.jsonl\nchecks:\n  - id: no_pii\n    type: regex_forbidden\n    target: text\n    patterns:\n      - \"[A-Z0-9._%+-]+@[A-Z0-9.-]+\\\\.[A-Z]{2,}\"\nbudgets:\n  cost_usd_per_run_max: 0.50\n  latency_ms_p95_max: 2000\nmode: fail\n```\n\n## Permissions\n\nWhen using `format: sarif`, ensure your workflow grants Code Scanning upload permissions:\n\n```yaml\npermissions:\n  contents: read\n  security-events: write\n```\n\n## Examples\n\n### Advanced usage (baseline/regress + flake control + cost gate)\n\n```yaml\n- uses: geminimir/promptproof-action@v0\n  with:\n    config: promptproof.yaml\n    baseline-ref: origin/main   # pull last green snapshot from main\n    regress: true               # also compare with any local baseline\n    runs: 5                     # flake control\n    seed: 42                    # deterministic nondeterminism\n    max-run-cost: 1.75          # cost gate for the entire suite\n    format: junit               # emit JUnit XML for test tab\n    mode: gate                  # fail on violations\n```\n\n### Emit SARIF for Code Scanning\n\n```yaml\n- uses: geminimir/promptproof-action@v0\n  with:\n    config: promptproof.yaml\n    format: sarif\n```\n\n### Gate on cost via branch rules (report-only mode)\n\n```yaml\n- uses: geminimir/promptproof-action@v0\n  with:\n    config: promptproof.yaml\n    max-run-cost: 1.00\n    mode: report-only           # never fail directly\n```\n\nThen in Branch protection, require the \"PromptProof\" check so the PR is blocked when the budget is exceeded.\n\n### Custom Output Format\n\n```yaml\n- uses: geminimir/promptproof-action@v0\n  with:\n    config: promptproof.yaml\n    format: junit\n    report-artifact: promptproof-report\n    snapshot-on-success: true\n    snapshot-promote-on-main: true\n    snapshot-tag: nightly\n```\n\n### Zero-network Quickstart (fixtures only)\n\nNo API keys required. Use sample fixtures to see a green run:\n\n```yaml\nname: PromptProof\non: [pull_request]\njobs:\n  eval:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: geminimir/promptproof-action@v0\n        with:\n          config: example/promptproof.yaml\n          format: html\n          mode: report-only\n```\n\nThis uses recorded fixtures under `example/fixtures/` so CI makes no network calls.\n\n### Make it a required check\n\n1. Settings → Branches → Branch protection rules → Add rule\n2. Branch name pattern = `main`\n3. Enable \"Require status checks to pass\" → select \"PromptProof\"\n4. Save\n \n\n### With Matrix Testing\n\n```yaml\nstrategy:\n  matrix:\n    suite: [support, sales, docs]\nsteps:\n  - uses: geminimir/promptproof-action@v0\n    with:\n      config: promptproof-${{ matrix.suite }}.yaml\n```\n\n## PR Comments\n\nThe action automatically comments on PRs with:\n- Violation summary grouped by check type\n- Key metrics (cost, latency, pass/fail counts)\n- Expandable details for each violation type\n- A permalink to the run artifacts\n\n## Artifacts\n\nReports are uploaded as artifacts and retained for 30 days:\n- HTML report for human review\n- JSON report for programmatic access\n- JUnit XML for test result visualization\n- SARIF report for Code Scanning\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeminimir%2Fpromptproof-action","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeminimir%2Fpromptproof-action","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeminimir%2Fpromptproof-action/lists"}