https://github.com/Dmatut7/shoppay-audit-benchmark
AI business-logic audit benchmark for Codex-style coding agents
https://github.com/Dmatut7/shoppay-audit-benchmark
agent-benchmark ai-agent ai-audit benchmark business-logic business-logic-bugs codex coding-agents llm-evaluation nodejs open-source payments software-testing
Last synced: 3 days ago
JSON representation
AI business-logic audit benchmark for Codex-style coding agents
- Host: GitHub
- URL: https://github.com/Dmatut7/shoppay-audit-benchmark
- Owner: Dmatut7
- License: mit
- Created: 2026-06-05T17:55:24.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2026-06-05T18:44:36.000Z (about 2 months ago)
- Last Synced: 2026-06-05T19:23:42.856Z (about 2 months ago)
- Topics: agent-benchmark, ai-agent, ai-audit, benchmark, business-logic, business-logic-bugs, codex, coding-agents, llm-evaluation, nodejs, open-source, payments, software-testing
- Language: JavaScript
- Homepage: https://dmatut7.github.io/shoppay-audit-benchmark/
- Size: 33.2 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Citation: CITATION.cff
- Security: SECURITY.md
- Roadmap: docs/ROADMAP.md
Awesome Lists containing this project
- awesome-ai-code-review - ShopPay Audit Benchmark - Spec-grounded benchmark for testing whether AI code review/coding agents catch business-logic defects in payment, wallet, webhook, and authorization flows. (Benchmarks)
README
# ShopPay Audit Benchmark
[](https://github.com/Dmatut7/shoppay-audit-benchmark/actions/workflows/test.yml)
Landing page: https://dmatut7.github.io/shoppay-audit-benchmark/
ShopPay Audit Benchmark is a compact, intentionally flawed payment and wallet service for evaluating whether AI coding agents can find **business-logic defects** that happy-path tests miss.
The project is designed for Codex-style audit workflows: read the business rules, inspect the implementation, identify rule violations, and optionally submit a fix with regression tests.
## Why this exists
Many code checks find crashes, missing imports, or syntax errors. Real product failures often come from logic gaps instead:
- refunding the wrong order
- trusting an unsigned webhook
- calculating tax in the wrong order
- allowing profile updates to change privileged fields
- double-spending a wallet balance under concurrency
This repository gives maintainers and AI-agent builders a repeatable target for testing that kind of reasoning.
## What is included
- `SPEC.md` — the source of truth for expected business behavior.
- `src/` — a tiny Node.js service with seeded business-rule defects.
- `test/` — tests that keep the baseline reproducible and prove the seeded defects are present.
- `BENCHMARK.md` — suggested audit task, success criteria, and expected finding categories.
- `.github/workflows/test.yml` — CI for the reproducible baseline.
## Quick start
```bash
npm test
```
No install step is required because the benchmark uses only Node.js built-ins.
## Benchmark workflow
Give an agent this task:
> Read `SPEC.md`, audit `src/`, and report every place where implementation behavior violates the business rules. For each finding, include the violated rule, impacted file/function, reproduction idea, and a minimal fix plan.
A stronger run can ask the agent to implement fixes and add regression tests after the audit report.
## Scoring and examples
- `docs/SCORING.md` defines a 100-point rubric for comparing AI audit reports.
- `examples/audit-report.md` shows the expected report shape and finding detail.
- `examples/runs/2026-06-05-codex-reference-audit.md` provides a scored reference audit run.
- `docs/ROADMAP.md` tracks planned benchmark cases.
- `docs/PROMOTION.md` contains ready-to-share launch copy.
- `docs/OUTREACH.md` tracks public promotion and directory submissions.
## Project status
Current release: `v0.1.0` baseline benchmark.
This repository is intentionally small. The goal is fast, repeatable audit runs that show whether an AI agent can map implementation behavior back to written business rules.
## Important note about the tests
The default tests are intentionally green against the flawed baseline. Some tests assert the current vulnerable behavior so the benchmark remains reproducible. They are not acceptance tests for a fixed production service.
For a fix branch, replace the baseline-vulnerability assertions with regression tests that enforce `SPEC.md`.
## Business areas covered
- Orders and refund lifecycle
- User authorization
- Payment webhook trust
- Wallet balance and atomic deduction
- Tax and discount ordering
- Profile update privilege boundaries
## Contributing
See `CONTRIBUTING.md` and `docs/ROADMAP.md` for the contribution model and planned benchmark cases.
## License
MIT