{"id":51472057,"url":"https://github.com/Dmatut7/shoppay-audit-benchmark","last_synced_at":"2026-07-25T13:00:39.327Z","repository":{"id":362750528,"uuid":"1260638425","full_name":"Dmatut7/shoppay-audit-benchmark","owner":"Dmatut7","description":"AI business-logic audit benchmark for Codex-style coding agents","archived":false,"fork":false,"pushed_at":"2026-06-05T18:44:36.000Z","size":34,"stargazers_count":1,"open_issues_count":7,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-05T19:23:42.856Z","etag":null,"topics":["agent-benchmark","ai-agent","ai-audit","benchmark","business-logic","business-logic-bugs","codex","coding-agents","llm-evaluation","nodejs","open-source","payments","software-testing"],"latest_commit_sha":null,"homepage":"https://dmatut7.github.io/shoppay-audit-benchmark/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Dmatut7.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":"docs/ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-05T17:55:24.000Z","updated_at":"2026-06-05T18:54:05.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Dmatut7/shoppay-audit-benchmark","commit_stats":null,"previous_names":["dmatut7/shoppay-audit-benchmark"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/Dmatut7/shoppay-audit-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dmatut7%2Fshoppay-audit-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dmatut7%2Fshoppay-audit-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dmatut7%2Fshoppay-audit-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dmatut7%2Fshoppay-audit-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Dmatut7","download_url":"https://codeload.github.com/Dmatut7/shoppay-audit-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dmatut7%2Fshoppay-audit-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35879604,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-25T02:00:06.922Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-benchmark","ai-agent","ai-audit","benchmark","business-logic","business-logic-bugs","codex","coding-agents","llm-evaluation","nodejs","open-source","payments","software-testing"],"created_at":"2026-07-06T19:00:32.894Z","updated_at":"2026-07-25T13:00:39.304Z","avatar_url":"https://github.com/Dmatut7.png","language":"JavaScript","funding_links":[],"categories":["Benchmarks"],"sub_categories":[],"readme":"# ShopPay Audit Benchmark\n\n[![Test](https://github.com/Dmatut7/shoppay-audit-benchmark/actions/workflows/test.yml/badge.svg)](https://github.com/Dmatut7/shoppay-audit-benchmark/actions/workflows/test.yml)\n\n![Social card](docs/assets/social-card.svg)\n\n\nLanding page: https://dmatut7.github.io/shoppay-audit-benchmark/\n\nShopPay Audit Benchmark is a compact, intentionally flawed payment and wallet service for evaluating whether AI coding agents can find **business-logic defects** that happy-path tests miss.\n\nThe project is designed for Codex-style audit workflows: read the business rules, inspect the implementation, identify rule violations, and optionally submit a fix with regression tests.\n\n## Why this exists\n\nMany code checks find crashes, missing imports, or syntax errors. Real product failures often come from logic gaps instead:\n\n- refunding the wrong order\n- trusting an unsigned webhook\n- calculating tax in the wrong order\n- allowing profile updates to change privileged fields\n- double-spending a wallet balance under concurrency\n\nThis repository gives maintainers and AI-agent builders a repeatable target for testing that kind of reasoning.\n\n## What is included\n\n- `SPEC.md` — the source of truth for expected business behavior.\n- `src/` — a tiny Node.js service with seeded business-rule defects.\n- `test/` — tests that keep the baseline reproducible and prove the seeded defects are present.\n- `BENCHMARK.md` — suggested audit task, success criteria, and expected finding categories.\n- `.github/workflows/test.yml` — CI for the reproducible baseline.\n\n## Quick start\n\n```bash\nnpm test\n```\n\nNo install step is required because the benchmark uses only Node.js built-ins.\n\n## Benchmark workflow\n\nGive an agent this task:\n\n\u003e Read `SPEC.md`, audit `src/`, and report every place where implementation behavior violates the business rules. For each finding, include the violated rule, impacted file/function, reproduction idea, and a minimal fix plan.\n\nA stronger run can ask the agent to implement fixes and add regression tests after the audit report.\n\n\n## Scoring and examples\n\n- `docs/SCORING.md` defines a 100-point rubric for comparing AI audit reports.\n- `examples/audit-report.md` shows the expected report shape and finding detail.\n- `examples/runs/2026-06-05-codex-reference-audit.md` provides a scored reference audit run.\n- `docs/ROADMAP.md` tracks planned benchmark cases.\n- `docs/PROMOTION.md` contains ready-to-share launch copy.\n- `docs/OUTREACH.md` tracks public promotion and directory submissions.\n\n## Project status\n\nCurrent release: `v0.1.0` baseline benchmark.\n\nThis repository is intentionally small. The goal is fast, repeatable audit runs that show whether an AI agent can map implementation behavior back to written business rules.\n\n## Important note about the tests\n\nThe default tests are intentionally green against the flawed baseline. Some tests assert the current vulnerable behavior so the benchmark remains reproducible. They are not acceptance tests for a fixed production service.\n\nFor a fix branch, replace the baseline-vulnerability assertions with regression tests that enforce `SPEC.md`.\n\n## Business areas covered\n\n- Orders and refund lifecycle\n- User authorization\n- Payment webhook trust\n- Wallet balance and atomic deduction\n- Tax and discount ordering\n- Profile update privilege boundaries\n\n## Contributing\n\nSee `CONTRIBUTING.md` and `docs/ROADMAP.md` for the contribution model and planned benchmark cases.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDmatut7%2Fshoppay-audit-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDmatut7%2Fshoppay-audit-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDmatut7%2Fshoppay-audit-benchmark/lists"}