{"id":51505794,"url":"https://github.com/maxgfr/ultraeval","last_synced_at":"2026-07-08T00:03:40.359Z","repository":{"id":370010117,"uuid":"1292805327","full_name":"maxgfr/ultraeval","owner":"maxgfr","description":"Evaluate a skill or codebase with a multi-agent workflow, ground every finding in a real file:line, and generate an AI-exploitable TDD fix backlog. A skills.sh agent skill.","archived":false,"fork":false,"pushed_at":"2026-07-07T21:36:45.000Z","size":106,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-07T23:12:21.749Z","etag":null,"topics":["agent-skill","ai-agents","code-review","evaluation","llm","skills-sh","static-analysis","subagents","tdd","workflow"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maxgfr.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-07T21:36:15.000Z","updated_at":"2026-07-07T21:36:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/maxgfr/ultraeval","commit_stats":null,"previous_names":["maxgfr/ultraeval"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/maxgfr/ultraeval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxgfr%2Fultraeval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxgfr%2Fultraeval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxgfr%2Fultraeval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxgfr%2Fultraeval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maxgfr","download_url":"https://codeload.github.com/maxgfr/ultraeval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxgfr%2Fultraeval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35246577,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-skill","ai-agents","code-review","evaluation","llm","skills-sh","static-analysis","subagents","tdd","workflow"],"created_at":"2026-07-08T00:03:39.601Z","updated_at":"2026-07-08T00:03:40.350Z","avatar_url":"https://github.com/maxgfr.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ultraeval\n\n[![CI](https://github.com/maxgfr/ultraeval/actions/workflows/ci.yml/badge.svg)](https://github.com/maxgfr/ultraeval/actions/workflows/ci.yml)\n\n\u003e Evaluate a **skill or codebase** with a multi-agent workflow, ground every finding in a real `file:line`, and get back **AI-exploitable fix docs** — a prioritized backlog plus per-fix **TDD cards** a model can implement red→green→refactor.\n\nultraeval is an [Agent Skill](https://www.skills.sh/) (the open agent-skills ecosystem). A tiny zero-dependency engine scaffolds the run, **generates the workflow + subagent contracts**, and enforces a grounding gate; the AI does the research, judgment, and writing. It is the method productized: the same one used to audit a whole family of skills, packaged so you can replay it on any target.\n\n## Install\n\n```bash\nnpx skills add maxgfr/ultraeval        # into the current project (committed, team-shared)\nnpx skills add -g maxgfr/ultraeval     # globally\n```\n\nNo `npm install`, no API keys — the engine is a single committed `.mjs` bundle.\n\n## What it does\n\n```\ninit → plan → run(research → test-plan → execute+gates → judge → results) → verify → backlog(TDD) → render\n```\n\n- **`plan`** generates `eval.workflow.mjs` — a ready-to-launch multi-agent Workflow parameterized to your target — plus `agents/*.md` dispatch contracts. This is the \"generate the workflow and subagents\" part.\n- Every finding must resolve to a real `file:line` in the target (or a produced run-log line). **`check` rejects a hallucinated or stale citation**; **`verify`** adversarially confirms the cited content actually supports the claim.\n- **`backlog --tdd`** turns confirmed findings into `BACKLOG.json` (machine-readable, priority-ordered) and one `fixes/FIX-*.md` **TDD card** per finding (RED failing-test-first → GREEN change → VERIFY).\n\n## What it produces\n\n```\n\u003crun\u003e/\n  eval.config.json         # target, kind, category, scored dimensions\n  eval.workflow.mjs        # the generated multi-agent Workflow\n  agents/*.md              # subagent dispatch contracts\n  research/\u003cdim\u003e.md        # cited methodology per dimension\n  TEST-PLAN.md             # every functionality + gate to test\n  runs/core.md, live.md    # deterministic + live evidence (cited by findings)\n  findings.json            # grounded findings (the gate enforces file:line resolution)\n  VERIFY.todo.json/.json   # adversarial claim↔evidence verdicts\n  RESULTS.md / SUMMARY.md  # scored report (claims cite [F#])\n  BACKLOG.json             # priority-ordered fix tasks\n  fixes/FIX-*.md           # per-fix TDD cards\n  REMEDIATION.md           # the human-readable plan\n  index.html / index.md    # dashboard\n```\n\n## Standalone CLI (the engine)\n\n```bash\nENGINE=node scripts/ultraeval.mjs\n$ENGINE init --target ../my-skill --out /tmp/eval --category \"agent skill\"\n$ENGINE plan --run /tmp/eval                       # generate the workflow + agents\n$ENGINE check --run /tmp/eval                       # grounding gate (exit 1 on a hallucinated citation)\n$ENGINE verify --run /tmp/eval                      # write the adversarial worklist\n$ENGINE verify --run /tmp/eval --apply verdicts.json\n$ENGINE check --run /tmp/eval --semantic --require-verify   # exit gate\n$ENGINE backlog --run /tmp/eval --tdd               # BACKLOG.json + fixes/FIX-*.md\n$ENGINE render --run /tmp/eval                      # index.html + index.md\n$ENGINE clean --run /tmp/eval                       # remove derived artifacts (keeps deliverables)\n```\n\nRun `node scripts/ultraeval.mjs --help` for the full flag surface. The grounding contract, orchestration, gate rules, and TDD-card format are documented under [`skills/ultraeval/references/`](./skills/ultraeval/references/).\n\n## Why the gate matters\n\nThe failure mode of every \"AI evaluates X\" tool is confident, ungrounded findings. ultraeval makes that structurally hard: `check` opens each cited `file:line` in the target and fails if it does not exist or is out of range; `verify` then asks a skeptic whether the content actually supports the claim, and `check --semantic --require-verify` is the exit gate. A fix backlog you cannot trace back to real code is worse than none.\n\n## Development\n\n```bash\npnpm install\npnpm run build        # tsup -\u003e scripts/ultraeval.mjs, mirrored into skills/ultraeval/scripts/\npnpm test             # vitest\npnpm run eval         # RED/GREEN gate probe against the shipped bundle\npnpm run check:build  # bundle is reproducible + install-bundle shape is valid\n```\n\nThe engine source is `src/*.ts`; the shipped bundle is committed so the skill installs with zero dependencies. Keep the two engine copies byte-identical (`check:build` enforces it).\n\n## Security\n\nultraeval only **reads** the evaluated target and writes under the run dir; it never executes the target's code. The `executor` subagent may run the target's *own* commands (its tests/gates) — sandbox untrusted repos.\n\n## License\n\nMIT © maxgfr\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxgfr%2Fultraeval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxgfr%2Fultraeval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxgfr%2Fultraeval/lists"}