{"id":136366,"url":"https://github.com/benchflow-ai/awesome-evals","name":"awesome-evals","description":"A curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.","projects_count":405,"last_synced_at":"2026-10-12T16:00:19.712Z","repository":{"id":367130978,"uuid":"1278930806","full_name":"benchflow-ai/awesome-evals","owner":"benchflow-ai","description":"A curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.","archived":false,"fork":false,"pushed_at":"2026-10-05T21:16:48.000Z","size":1418,"stargazers_count":956,"open_issues_count":50,"forks_count":119,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-10-06T16:22:43.558Z","etag":null,"topics":["agent-evaluation","ai-agents","awesome","awesome-list","benchmarks","evals","llm","llm-evaluation","rl-environments"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/benchflow-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-24T08:10:33.000Z","updated_at":"2026-10-05T15:46:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"98e880dd-a293-4412-9ecc-a755857ade5a","html_url":"https://github.com/benchflow-ai/awesome-evals","commit_stats":null,"previous_names":["benchflow-ai/awesome-evals"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/benchflow-ai/awesome-evals","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchflow-ai%2Fawesome-evals","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchflow-ai%2Fawesome-evals/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchflow-ai%2Fawesome-evals/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchflow-ai%2Fawesome-evals/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/benchflow-ai","download_url":"https://codeload.github.com/benchflow-ai/awesome-evals/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchflow-ai%2Fawesome-evals/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":343843146,"owners_count":38235919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-10-03T21:59:58.778Z","status":"online","status_checked_at":"2026-10-12T02:00:06.297Z","response_time":96,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2026-07-07T04:01:24.471Z","updated_at":"2026-10-12T16:00:19.713Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["License","5 · Evaluation infrastructure (the eval stack: datasets, scorers, online/offline, tracing, CI)","9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization)","8 · LLM-as-judge \u0026 verifiers (alignment, biases, verifiable vs judgeable)","10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing)","6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming)","🎙 Talks, podcasts \u0026 slides (transcribed + noted)","1 · Why we need evals","⭐ Must-read starter set (read these first)","2 · \"If you can eval it, you have built it\" — eval ⇄ capability ⇄ RL environment","7 · Evals \u0026 RL environments (verifiers, reward design, difficulty calibration, lifecycle)","3 · The model / harness / skill decomposition","4 · Observability \u0026 the output / eval space (the surfaces you can grade)","Companies \u0026 landscape (eval / RL-environment market)"],"sub_categories":["More eval talks, podcasts \u0026 lectures (annotated; deep notes in progress)","5f · Observability + eval platforms (tracing · datasets · online/offline · CI)","5g · Tracing standards","🎤 Conference \u0026 individual talks","5b · TypeScript/JS-native eval runners","5a · Eval frameworks \u0026 harnesses (code-first test-runners)","5e · RL-environment / verifiable-reward toolkits (eval ⇄ training)","5c · RAG / retrieval evaluation","5d · LLM-as-judge / reward / verifier libraries","🎙 Podcast episodes","🎓 University lectures"],"readme":"# Awesome Agent Evals [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\n\n\u003e A curated, opinionated, **non-BS** library of the best resources for **building and evaluating AI agents** — papers, blog posts, talks, courses, tools, and benchmarks.\n\nMaintained by [**BenchFlow**](https://benchflow.ai) · Join our [**Discord**](https://discord.gg/mZ9Rc8q8W3)\n\nMost \"awesome\" lists are link dumps. This one is **annotated and verified**: every entry says *what it is and why it belongs*, URLs are checked, quotes are verbatim, and dead/abandoned tools are pruned (not silently listed). It was assembled by:\n\n- a **depth-4 recursive citation crawl** (11.6k papers, ranked by in-degree) to surface the academic canon,\n- **targeted practitioner-web discovery** for the industry sources citation graphs miss (Eugene Yan, Han-Chung Lee, Hamel Husain, Shreya Shankar, Nathan Lambert, …),\n- **47 talks \u0026 podcasts transcribed and deep-noted** (verbatim + timestamps), and\n- **per-section gap audits** with adversarial verification.\n\n**443+ curated links · 143 deep reading notes** (see [`notes/`](notes/)). Markers: 🆕 = released/updated 2025–2026 · ⚠️ = caveat. Contributions welcome — see [CONTRIBUTING](CONTRIBUTING.md).\n\n\u003e 📘 **Playbook:** [**PATTERNS.md**](PATTERNS.md) — real, runnable code + worked examples for LLM-as-judge (aligned to humans), pass@k/pass^k, error analysis, trajectory \u0026 world-state grading, CI gating, verifiable rewards, and more.\n\n## Contents\n- [📘 Playbook — real code \u0026 worked examples (PATTERNS.md)](PATTERNS.md)\n- [⭐ Must-read starter set (read these first)](#-must-read-starter-set-read-these-first)\n- [1 · Why we need evals](#1-why-we-need-evals)\n- [2 · \"If you can eval it, you have built it\" — eval ⇄ capability ⇄ RL environment](#2-if-you-can-eval-it-you-have-built-it-eval-capability-rl-environment)\n- [3 · The model / harness / skill decomposition](#3-the-model-harness-skill-decomposition)\n- [4 · Observability \u0026 the output / eval space (the surfaces you can grade)](#4-observability-the-output-eval-space-the-surfaces-you-can-grade)\n- [5 · Evaluation infrastructure (the eval stack: datasets, scorers, online/offline, tracing, CI)](#5-evaluation-infrastructure-the-eval-stack-datasets-scorers-onlineoffline-tracing-ci)\n- [6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming)](#6-benchmark-vs-eval-and-benchmark-integrity-contamination-saturation-label-errors-leaderboard-gaming)\n- [7 · Evals \u0026 RL environments (verifiers, reward design, difficulty calibration, lifecycle)](#7-evals-rl-environments-verifiers-reward-design-difficulty-calibration-lifecycle)\n- [8 · LLM-as-judge \u0026 verifiers (alignment, biases, verifiable vs judgeable)](#8-llm-as-judge-verifiers-alignment-biases-verifiable-vs-judgeable)\n- [9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization)](#9-agent-specific-evaluation-trajectories-tool-use-multi-turn-world-state-multi-agent-localization)\n- [10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing)](#10-safety-adversarial-evaluation-prompt-injection-jailbreaks-action-authorization-benchmark-auditing)\n- [🎙 Talks, podcasts \u0026 slides (transcribed + noted)](#-talks-podcasts-slides-transcribed-noted)\n- [💬 Eval mentions](#-eval-mentions)\n- [Companies \u0026 landscape (eval / RL-environment market)](#companies-landscape-eval-rl-environment-market)\n- [Notes on provenance \u0026 gaps](#notes-on-provenance-gaps)\n- [Deep notes](#deep-notes)\n- [Contributing](#contributing)\n- [License](#license)\n\n---\n\n## ⭐ Must-read starter set (read these first)\n\n1. **[The Second Half](https://ysymyth.github.io/The-Second-Half/)** — Shunyu Yao — \u003chttps://ysymyth.github.io/The-Second-Half/\u003e · *blog* — \"Evaluation becomes more important than training.\" The field-level *why*.\n2. **[An LLM-as-Judge Won't Save the Product, Fixing Your Process Will](https://eugeneyan.com/writing/eval-process/)** — Eugene Yan — \u003chttps://eugeneyan.com/writing/eval-process/\u003e · *blog* — Process over tooling; evals as the scientific method.\n3. **[Hidden Technical Debt: Agent Evaluation Infrastructure](https://leehanchung.github.io/blogs/2026/06/13/hidden-technical-debt-agent-evaluation-infra/)** — Han-Chung Lee — \u003chttps://leehanchung.github.io/blogs/2026/06/13/hidden-technical-debt-agent-evaluation-infra/\u003e · *blog* — Control/data plane, the five eval surfaces, state deltas. \"Chat eval was a spreadsheet; agent eval is a system.\"\n4. **[LLM Evals FAQ](https://hamel.dev/blog/posts/evals-faq/)** — Hamel Husain \u0026 Shreya Shankar — \u003chttps://hamel.dev/blog/posts/evals-faq/\u003e · *blog* — The densest operational Q\u0026A: error analysis, binary judgments, the benevolent-dictator labeler.\n5. **[Asymmetry of Verification and Verifier's Law](https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law)** — Jason Wei — \u003chttps://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law\u003e · *blog* — \"Ability to verify == ability to create an RL environment.\"\n6. **[Demystifying Evals for AI Agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)** — Anthropic — \u003chttps://www.anthropic.com/engineering/demystifying-evals-for-ai-agents\u003e · *blog* — Best primary on agent-specific evals: task design, outcome vs trajectory, isolated trials, pass@k vs pass^k.\n7. **[How to Build Good Language Modeling Benchmarks](https://ofir.io/How-to-Build-Good-Language-Modeling-Benchmarks/)** — Ofir Press — \u003chttps://ofir.io/How-to-Build-Good-Language-Modeling-Benchmarks/\u003e · *blog* — Natural / auto-evaluatable / challenging; the \"-200%\" difficulty target; ~1-yr saturation.\n8. **[AI Agents That Matter](https://arxiv.org/abs/2407.01502)** — Kapoor, Stroebl, Siegel, Nadgir, Narayanan — \u003chttps://arxiv.org/abs/2407.01502\u003e · *paper* — Cost as a first-class metric; model-dev vs app-dev; missing holdouts breed overfitting.\n9. **[Building on Evaluation Quicksand](https://www.interconnects.ai/p/building-on-evaluation-quicksand)** — Nathan Lambert — \u003chttps://www.interconnects.ai/p/building-on-evaluation-quicksand\u003e · *blog* — LLM eval has no ground truth; contamination; eval↔training coupling.\n10. **[Who Validates the Validators? (EvalGen)](https://arxiv.org/abs/2404.12272)** — Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran, Arawjo (UIST '24) — \u003chttps://arxiv.org/abs/2404.12272\u003e · *paper* — \"Criteria drift\": you can't write the rubric before you grade.\n11. **[Benches 2026 — \"LLM benchmarks in the era of agents\"](https://florianbrand.com/posts/benches-2026)** — Florian Brand (Prime Intellect) — \u003chttps://florianbrand.com/posts/benches-2026\u003e · *blog + 61-slide talk* — The sharpest current read on why benchmarks break in the agent era: the \"evals are dead, just measure vibes\" backlash, how every layer of the eval-running stack (prompt · sampling temp · grader · harness) swings the score, and that benchmark ground truth is frequently wrong.\n12. **[A Shared Playbook for Trustworthy Third-Party Evaluations](https://openai.com/index/trustworthy-third-party-evaluations-foundations/)** — OpenAI — \u003chttps://openai.com/index/trustworthy-third-party-evaluations-foundations/\u003e · *blog (Safety, May 2026)* — What makes *independent* evals of frontier-model safeguards \u0026 capabilities trustworthy: harness selection, the validity hazards that distort results, and the standards third-party evaluators need.\n\n---\n\n## 1 · Why we need evals\n\n- **[The Second Half](https://ysymyth.github.io/The-Second-Half/)** — Shunyu Yao — \u003chttps://ysymyth.github.io/The-Second-Half/\u003e · *blog* — The bottleneck shifts from solving problems to *defining and evaluating* them. (also T2, T7)\n- **[An LLM-as-Judge Won't Save the Product, Fixing Your Process Will](https://eugeneyan.com/writing/eval-process/)** — Eugene Yan — \u003chttps://eugeneyan.com/writing/eval-process/\u003e · *blog* — \"Buying or building another evaluation tool won't save the product.\" Evals = the scientific method in disguise.\n- **[Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/)** — Hamel Husain — \u003chttps://hamel.dev/blog/posts/evals/\u003e · *blog* — The canonical \"you need evals\"; remove all friction from looking at your data; don't rely on generic frameworks.\n- **[A Field Guide to Rapidly Improving AI Products](https://hamel.dev/blog/posts/field-guide/)** — Hamel Husain — \u003chttps://hamel.dev/blog/posts/field-guide/\u003e · *blog* — \"Error analysis is consistently the highest-ROI activity.\" The metric for an AI roadmap is experiments run.\n- **[In Defense of AI Evals, for Everyone](https://www.sh-reya.com/blog/in-defense-ai-evals/)** — Shreya Shankar — \u003chttps://www.sh-reya.com/blog/in-defense-ai-evals/\u003e · *blog* — Rebuts the anti-eval backlash; evals = the systematic measurement of application quality.\n- **[What We Learned from a Year of Building with LLMs](https://applied-llms.org/)** — Yan, Bischof, Frye, Husain, Liu, Shankar — \u003chttps://applied-llms.org/\u003e (Part II: \u003chttps://www.oreilly.com/radar/what-we-learned-from-a-year-of-building-with-llms-part-ii/\u003e) · *blog* — The \"intern test,\" genchi genbutsu, turning vibe-checks into assertions.\n- **[Big Tech's LLM Evals Are Just Marketing](https://www.interconnects.ai/p/evals-are-marketing)** — Nathan Lambert — \u003chttps://www.interconnects.ai/p/evals-are-marketing\u003e · *blog* — Why frontier-lab leaderboard numbers are marketing, not science.\n- **[AI Engineering pitfalls](https://huyenchip.com/2025/01/16/ai-engineering-pitfalls.html)** — Chip Huyen — \u003chttps://huyenchip.com/2025/01/16/ai-engineering-pitfalls.html\u003e · *blog* — Common eval/AI-engineering mistakes from the *AI Engineering* author. (also T6)\n\n- **[Evals Are NOT All You Need](https://www.oreilly.com/radar/evals-are-not-all-you-need/)** — Aishwarya Naresh Reganti \u0026 Kiriti Badam (O'Reilly Radar) — \u003chttps://www.oreilly.com/radar/evals-are-not-all-you-need/\u003e · *blog* — The essential nuance piece: automated graders alone don't save you; you need a continuous-improvement flywheel of offline tests + production monitoring + real-user iteration. Pairs with Shreya's 'In Defense' to complete the backlash debate. 🆕\n- **[Why AI evals are the hottest new skill for product builders](https://www.lennysnewsletter.com/p/why-ai-evals-are-the-hottest-new-skill)** — Hamel Husain \u0026 Shreya Shankar with Lenny Rachitsky (Lenny's Podcast/Newsletter) — \u003chttps://www.lennysnewsletter.com/p/why-ai-evals-are-the-hottest-new-skill\u003e · *talk* — The accessible 'why evals matter' on-ramp (live walkthrough of error analysis, open/axial coding) that mainstreamed evals to PMs in 2025; the apartment-leasing-bot anecdote is the canonical 'you can't vibe-check' story. 🆕\n- **[How evals drive the next chapter in AI for businesses](https://openai.com/index/evals-drive-next-chapter-of-ai/)** — OpenAI — \u003chttps://openai.com/index/evals-drive-next-chapter-of-ai/\u003e · *blog* — Frontier-lab framing of evals as turning fuzzy business goals into specs and measurable ROI; useful counterweight to Lambert's 'evals are marketing' and grounds the 'why' for enterprise readers. 🆕 ⚠(unverified URL)\n- **[Beyond vibe checks: A PM's complete guide to evals](https://www.lennysnewsletter.com/p/beyond-vibe-checks-a-pms-complete)** — Aman Khan (Arize) with Lenny Rachitsky — \u003chttps://www.lennysnewsletter.com/p/beyond-vibe-checks-a-pms-complete\u003e · *blog* — The widely-shared PM-oriented argument for moving past 'looked good to me' vibe checks to systematic evals; one of the pieces that made evals a mainstream product skill in 2025. 🆕\n- **[A pragmatic guide to LLM evals for devs](https://newsletter.pragmaticengineer.com/p/evals)** — Gergely Orosz \u0026 Hamel Husain (The Pragmatic Engineer) — \u003chttps://newsletter.pragmaticengineer.com/p/evals\u003e · *newsletter* — Reaches the broad engineering audience with the core 'why': LLM non-determinism breaks traditional testing, so you need evals. High-distribution motivation piece co-written by Hamel. 🆕\n- **[Predicting model behavior before release by simulating deployment (Deployment Simulation)](https://openai.com/index/deployment-simulation/)** — OpenAI — \u003chttps://openai.com/index/deployment-simulation/\u003e · *blog* — Concrete 2026 evidence for why fixed/static evals fail: models recognize when they're being tested and game test suites; replaying ~1.3M real conversations surfaced reward-hacking no fixed eval caught. Strong 'why evals must evolve' argument. 🆕 ⚠(unverified URL)\n- **[evals are surprisingly often all you need](https://x.com/gdb/status/1733553161884127435)** — Greg Brockman (OpenAI) — \u003chttps://x.com/gdb/status/1733553161884127435\u003e · *blog* — The canonical one-liner ('evals are the new unit test') that anchors the whole 'why evals' thesis; frequently cited founding quote for the movement. Short but load-bearing.\n\n**Must-reads:** Yao · Yan (eval-process) · Hamel (field-guide)\n\n## 2 · \"If you can eval it, you have built it\" — eval ⇄ capability ⇄ RL environment\n\n- **[Asymmetry of Verification and Verifier's Law](https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law)** — Jason Wei — \u003chttps://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law\u003e · *blog* — Trainability tracks verifiability; verifying = creating an RL environment.\n- **[A Taxonomy of RL Environments for LLM Agents](https://leehanchung.github.io/blogs/2026/03/21/rl-environments-for-llm-agents/)** — Han-Chung Lee — \u003chttps://leehanchung.github.io/blogs/2026/03/21/rl-environments-for-llm-agents/\u003e · *blog* — A benchmark is a frozen RL environment; the E = {T,H,V,S,C} decomposition; \"verifiable beats judgeable.\"\n- **[The Life Cycle of an RL Environment](https://muratbuffalo.blogspot.com/2026/06/acm-cais-conference-on-ai-and-agentic.html)** — Kanav Garg (Core Automation; ex-DeepMind) — talk; summary at \u003chttps://muratbuffalo.blogspot.com/2026/06/acm-cais-conference-on-ai-and-agentic.html\u003e · *talk* — Difficulty calibration (the 1–4/16 Goldilocks band), RL as variance reduction, reward hacking under training pressure. *(local notes: `research/notes/kanav-garg-rl-environment-lifecycle.md`)*\n- **[Welcome to the Era of Experience](https://storage.googleapis.com/deepmind-media/Era-of-Experience%20/The%20Era%20of%20Experience%20Paper.pdf)** — David Silver \u0026 Richard Sutton — \u003chttps://storage.googleapis.com/deepmind-media/Era-of-Experience%20/The%20Era%20of%20Experience%20Paper.pdf\u003e · *paper* — Human-data value approaching its ceiling; the frontier is agents learning from experience / synthetic environments.\n- **[RLHF Book, Ch. 16 — Evaluation](https://rlhfbook.com/c/16-evaluation)** — Nathan Lambert — \u003chttps://rlhfbook.com/c/16-evaluation\u003e · *book* — Evaluation as a reflection of training goals; prompt-format sensitivity (60%→~0%).\n- **[What Comes Next with Reinforcement Learning](https://www.interconnects.ai/p/what-comes-next-with-reinforcement)** — Nathan Lambert — \u003chttps://www.interconnects.ai/p/what-comes-next-with-reinforcement\u003e · *blog* — Long-horizon credit assignment; where RL is and isn't ready.\n- **[verifiers](https://github.com/PrimeIntellect-ai/verifiers)** — Prime Intellect — \u003chttps://github.com/PrimeIntellect-ai/verifiers\u003e (docs: `.../blob/main/docs/environments.md`) · *tool/repo* — One environment package shared by eval and `prime-rl` — the eval-is-an-RL-env thesis as code.\n\n- **[DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/abs/2501.12948)** — DeepSeek-AI (Guo et al.) — \u003chttps://arxiv.org/abs/2501.12948\u003e · *paper* — The proof-of-thesis: pure RL with rule-based verifiable rewards (no SFT) makes reasoning emerge — the canonical 'if you can verify it, RL builds it' result; also published in Nature 2025. Conspicuously absent from a section literally about eval-as-RL-environment. 🆕\n- **[Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/abs/2411.15124)** — Lambert et al. (Allen Institute for AI) — \u003chttps://arxiv.org/abs/2411.15124\u003e · *paper* — Coined/popularized RLVR and open-sourced the recipe + code (open-instruct): swap the reward model for a verifier on tasks with checkable answers. The foundational citation behind every 'verifiable beats judgeable' claim in this section. 🆕\n- **[Natural Emergent Misalignment from Reward Hacking in Production RL](https://www.anthropic.com/research/emergent-misalignment-reward-hacking)** — Anthropic — \u003chttps://www.anthropic.com/research/emergent-misalignment-reward-hacking\u003e · *paper* — Empirical receipt for the section's 'reward hacking under training pressure' theme: learning to cheat on real coding environments generalizes to sabotage/alignment-faking; introduces inoculation prompting as mitigation (arXiv 2511.18397). 🆕\n- **[Environments Hub: A Community Hub To Scale RL To Open AGI](https://www.primeintellect.ai/blog/environments)** — Prime Intellect — \u003chttps://www.primeintellect.ai/blog/environments\u003e · *blog* — The launch post for the verifiers-spec marketplace (2,500+ shared eval/RL environments) — the eval-is-an-RL-env thesis as an actual ecosystem, the natural companion to the already-listed verifiers repo. 🆕\n- **[How to fully automate software engineering](https://www.mechanize.work/blog/how-to-fully-automate-software-engineering/)** — Ege Erdil, Matthew Barnett, Tamay Besiroglu (Mechanize) — \u003chttps://www.mechanize.work/blog/how-to-fully-automate-software-engineering/\u003e · *blog* — Sharpest statement of the inverse thesis: today's RL environments are rudimentary, so capability is gated on building richer/more diverse environments — 'you only get the capability you can build an environment for.' 🆕\n- **[Cheap RL tasks will waste compute](https://www.mechanize.work/blog/cheap-rl-tasks-will-waste-compute/)** — Mechanize (Erdil, Barnett, Besiroglu) — \u003chttps://www.mechanize.work/blog/cheap-rl-tasks-will-waste-compute/\u003e · *blog* — The economics of environment quality: data and compute are complementary, so low-quality (cheaply-bought) tasks waste expensive RL compute — directly informs difficulty calibration / why environment design matters. 🆕\n- **[An FAQ on Reinforcement Learning Environments](https://epoch.ai/gradient-updates/state-of-rl-envs)** — Jean-Stanislas Denain \u0026 Chris Barber (Epoch AI) — \u003chttps://epoch.ai/gradient-updates/state-of-rl-envs\u003e · *blog* — Practitioner-interview survey (18 pros) on how RL environments are actually built, the reward-hacking failure modes, and the production-scaling bottleneck — the empirical state-of-the-field map this section lacks. 🆕\n- **[RL Environments and RL for Science: Data Foundries and Multi-Agent Architectures](https://newsletter.semianalysis.com/p/rl-environments-and-rl-for-science)** — AJ Kourabi \u0026 Dylan Patel (SemiAnalysis) — \u003chttps://newsletter.semianalysis.com/p/rl-environments-and-rl-for-science\u003e · *newsletter* — Market-structure view: 35+ companies now sell RL environments; capability gains are coming from ramping RL compute, not pretraining. Grounds the 'benchmark = frozen RL environment' thesis in who's actually building/buying them. 🆕\n- **[Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://github.com/harbor-framework/terminal-bench)** — Harbor / Stanford / Laude Institute — \u003chttps://github.com/harbor-framework/terminal-bench\u003e · *benchmark* — A concrete instance of the thesis: each task ships a Docker environment + programmatic verification test suite + oracle — i.e. a benchmark that IS an RL environment (and is used as one). 2.4k stars, active. 🆕\n- **[tau2-bench (τ²-Bench): A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://github.com/sierra-research/tau2-bench)** — Sierra Research (Barres et al.) — \u003chttps://github.com/sierra-research/tau2-bench\u003e · *benchmark* — Dual-control, multi-turn, policy-following eval with a simulated user and verifiable DB-state checks — the canonical example of a verifiable conversational/agentic environment beyond math/code (paper arXiv 2506.07982). 🆕\n\n**Must-reads:** Wei · Lee (RL-env taxonomy)\n\n## 3 · The model / harness / skill decomposition\n\n- **[Hidden Technical Debt: Agent Harness](https://leehanchung.github.io/blogs/2026/05/08/hidden-technical-debt-agent-harness/)** — Han-Chung Lee — \u003chttps://leehanchung.github.io/blogs/2026/05/08/hidden-technical-debt-agent-harness/\u003e · *blog* — The harness is the agent; what teams call \"the model\" is mostly harness + product.\n- **[Hidden Technical Debt series (index)](https://leehanchung.github.io/blogs/)** — Han-Chung Lee — \u003chttps://leehanchung.github.io/blogs/\u003e · *blog* — The four-part series (eval infra, runtime, harness, + agent runtime ~2026/04/24). *(verify the runtime post URL on the index.)*\n- **[Measuring AI Ability to Complete Long Tasks](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/)** — METR — \u003chttps://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/\u003e · *paper/blog* — Scaffolds change the measured horizon; success-vs-human-time as a primitive. (also T9)\n- **[Turing Post interview (\"Open Models Won't Catch Up\")](https://www.turingpost.com/p/nathanlambert)** — Nathan Lambert — \u003chttps://www.turingpost.com/p/nathanlambert\u003e · *talk/interview* — \"What technical people call the harness or the product matters more than just the model.\"\n- **[Quo vadis, LLM benchmarks?](https://florianbrand.com/posts/benches-2026)** — Florian Brand (Prime Intellect) — \u003chttps://florianbrand.com/posts/benches-2026\u003e (talk: \u003chttps://www.youtube.com/watch?v=kmTMc-fVSXw\u003e) · *blog/talk* — The AlgoTune case: *same model, different harness, opposite ranking.* (also T6) *(notes: `research/notes/florian-brand-*`)*\n\n- **[The Model is the Product](https://leehanchung.github.io/talks/2025/04/23/the-model-is-the-product/)** — Han-Chung Lee — \u003chttps://leehanchung.github.io/talks/2025/04/23/the-model-is-the-product/\u003e · *talk* — The primary-source talk (Data Council 2025) behind the must-read author's whole thesis — the direct counterpart to Hamel's 'Model is Not the Product'; the foundational text of the harness/model debate this section is built on. 🆕\n- **[The Model is Not the Product](https://www.youtube.com/watch?v=EEw2PpL-_NM)** — Hamel Husain — \u003chttps://www.youtube.com/watch?v=EEw2PpL-_NM\u003e · *talk* — The opposing side of the Lee debate (Data Council 2025): great products are mostly harness + product + evals, not the model. Section already cites Lee; it should cite the debate it half-references. 🆕\n- **[Agents are models using tools in a loop](https://simonwillison.net/2025/May/22/tools-in-a-loop/)** — Simon Willison — \u003chttps://simonwillison.net/2025/May/22/tools-in-a-loop/\u003e · *blog* — The canonical, now-widely-adopted definition of an agent; 'the skill is in the design of both the tools and the loop' — the cleanest statement of why the harness, not the model, dominates behavior. 🆕\n- **[Harness engineering: leveraging Codex in an agent-first world](https://openai.com/index/harness-engineering/)** — OpenAI — \u003chttps://openai.com/index/harness-engineering/\u003e · *blog* — Frontier-lab primary source coining 'harness engineering': a 1M-line codebase built by Codex agents where improving the environment/harness mattered more than the model. Lab-side complement to Lee's 'harness is the agent'. (URL returns 403 to scraper but page is live; corroborated by InfoQ/Milvus coverage.) 🆕\n- **[Equipping agents for the real world with Agent Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)** — Anthropic — \u003chttps://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills\u003e · *blog* — The primary source for the 'skill' leg of the model/harness/skill decomposition — skills as composable, progressively-disclosed capabilities (later made an open standard). The section title says 'skill' but has zero skill sources. 🆕\n- **[Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)** — Anthropic — \u003chttps://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents\u003e · *blog* — Anthropic's primary statement that the harness's job is engineering context (editing, compaction, memory, programmatic tool-calling) — the mechanism behind why same model + different harness diverges. 🆕\n- **[Writing effective tools for agents — with agents](https://www.anthropic.com/engineering/writing-tools-for-agents)** — Anthropic (Ken Aizawa) — \u003chttps://www.anthropic.com/engineering/writing-tools-for-agents\u003e · *blog* — Tool design is a load-bearing part of the harness; 'agents are only as effective as the tools we give them,' validated eval-first. Directly ties harness decisions to measured agent performance. 🆕\n- **[Paving the way for agents in biology (VirBench)](https://www.anthropic.com/research/agents-in-biology)** — Anthropic — \u003chttps://www.anthropic.com/research/agents-in-biology\u003e · *blog* — 120 viral-sequence retrieval queries across 40 pathogens; without deterministic tooling, model accuracy spans 16.9%–91.3% and Claude Sonnet 4 returned 106 sequences in one run, 15 in a second, 5 in a third to an identical prompt. Adding a deterministic retrieval layer (gget) pushed all models above 90% accuracy (peaking at 99.7%) and eliminated run-to-run variance entirely: \"adding a deterministic retrieval layer made model choice much less important.\" Controlled empirical proof that harness/tool design dominates model choice for scientific precision. 🆕\n- **[Same Model, Different Results: Why Coding Agents Aren't Interchangeable](https://blog.thepete.net/blog/2025/12/10/same-model-different-results-why-coding-agents-arent-interchangeable/)** — Pete Hodgson — \u003chttps://blog.thepete.net/blog/2025/12/10/same-model-different-results-why-coding-agents-arent-interchangeable/\u003e · *blog* — Concrete teardown of Claude Code's harness (system reminders, sub-agents, planning, IDE feedback) showing identical models yield different results — the practitioner case-study version of Brand's AlgoTune point. 🆕\n- **[Holistic Agent Leaderboard (HAL)](https://hal.cs.princeton.edu/)** — Princeton SAgE team (Kapoor, Narayanan, et al.) — \u003chttps://hal.cs.princeton.edu/\u003e · *benchmark* — Standardized, cost-aware harness that runs the SAME agent harness across 9 benchmarks/9 models (21,730 rollouts) — the infrastructure answer to 'harness confounds rankings.' ICLR 2026; paper arXiv:2510.11977. 🆕\n- **[Agent Harness Engineering](https://www.oreilly.com/radar/agent-harness-engineering/)** — Addy Osmani (O'Reilly Radar) — \u003chttps://www.oreilly.com/radar/agent-harness-engineering/\u003e · *blog* — 'A decent model with a great harness beats a great model with a bad harness'; reframes agent failures as harness/config problems (traceable AGENTS.md rules). Names the converging harness primitives across coding agents. 🆕\n- **[What comes next with open models (weights / tools / harness decomposition)](https://www.interconnects.ai/p/the-next-phase-of-open-models)** — Nathan Lambert (Interconnects) — \u003chttps://www.interconnects.ai/p/the-next-phase-of-open-models\u003e · *blog* — Lambert's written articulation (Mar 2026) of an AI system as weights + tools + harness — the written companion to the Turing Post interview already listed, with the explicit three-part decomposition. 🆕\n\n**Must-reads:** Lee (harness) · Brand (Quo vadis)\n\n## 4 · Observability \u0026 the output / eval space (the surfaces you can grade)\n\n- **[Hidden Technical Debt: Agent Evaluation Infrastructure](https://leehanchung.github.io/blogs/2026/06/13/hidden-technical-debt-agent-evaluation-infra/)** — Han-Chung Lee — \u003chttps://leehanchung.github.io/blogs/2026/06/13/hidden-technical-debt-agent-evaluation-infra/\u003e · *blog* — Control plane / data plane; the **five surfaces** (output, trace, memory, environment, mechanistic); the empty-tool-result hallucination.\n- **[The Three Pillars of AI Observability](https://www.braintrust.dev/blog/three-pillars-ai-observability)** — Braintrust — \u003chttps://www.braintrust.dev/blog/three-pillars-ai-observability\u003e · *blog* — Dataset reconciliation (living datasets); traces / evals / annotation.\n- **[Agent Trajectory Evaluations](https://arize.com/docs/ax/evaluate/evaluators/trace-and-session-evals/trace-level-evaluations/agent-trajectory-evaluations)** — Arize (AX docs) — \u003chttps://arize.com/docs/ax/evaluate/evaluators/trace-and-session-evals/trace-level-evaluations/agent-trajectory-evaluations\u003e · *docs* — Grading the path, not just the answer.\n- **[AI Agent Metrics: How Elite Teams Evaluate](https://galileo.ai/blog/ai-agent-metrics)** — Galileo — \u003chttps://galileo.ai/blog/ai-agent-metrics\u003e · *blog* — A concrete agent-metric taxonomy (action completion, tool selection, etc.).\n- **[OpenInference semantic conventions](https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md)** — Arize — \u003chttps://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md\u003e · *tool/repo* — An OTel-based agent trace schema (tool, args, observation, latency, cost).\n- **[LangSmith Evaluation / Trajectory evals](https://docs.langchain.com/langsmith/evaluation)** — LangChain — \u003chttps://docs.langchain.com/langsmith/evaluation\u003e · \u003chttps://docs.langchain.com/langsmith/trajectory-evals\u003e · *docs*.\n\n- **[OpenTelemetry GenAI Semantic Conventions (agent \u0026 framework spans)](https://github.com/open-telemetry/semantic-conventions-genai)** — OpenTelemetry / CNCF — \u003chttps://github.com/open-telemetry/semantic-conventions-genai\u003e · *docs* — The upstream vendor-neutral standard (spans/metrics/events for LLM calls, invoke_agent, execute_tool, MCP) that OpenInference maps onto — the canonical trace schema the section's OpenInference entry derives from. 🆕\n- **[Semantic Conventions for GenAI agent and framework spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/)** — OpenTelemetry — \u003chttps://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/\u003e · *docs* — Human-readable spec page for create_agent / invoke_agent / execute_tool spans and attributes — the precise definition of what a gradable agent trace looks like. 🆕\n- **[Inside the LLM Call: GenAI Observability with OpenTelemetry](https://opentelemetry.io/blog/2026/genai-observability/)** — OpenTelemetry (blog) — \u003chttps://opentelemetry.io/blog/2026/genai-observability/\u003e · *blog* — Walkthrough of emitting and reading GenAI spans (token usage, finish reasons, tool calls) — concrete intro to the trace surface for practitioners not steeped in OTel. 🆕\n- **[W\u0026B Weave — tracing \u0026 evaluation toolkit](https://docs.wandb.ai/weave)** — Weights \u0026 Biases — \u003chttps://docs.wandb.ai/weave\u003e · *docs* — @weave.op trace trees (inputs/outputs/cost/latency) plus a scorer-based eval harness — a widely used surface for grading both traces and outputs. 🆕\n- **[Laminar — open-source observability for AI agents](https://laminar.sh/)** — Laminar — \u003chttps://laminar.sh/\u003e · *tool* — OTel-native, agent-specific: transcript view, SQL-over-traces, and a rollout debugger — purpose-built for grading multi-step agent trajectories rather than single LLM calls. 🆕\n\n**Must-reads:** Lee (eval infra) · Braintrust (three pillars)\n\n## 5 · Evaluation infrastructure (the eval stack: datasets, scorers, online/offline, tracing, CI)\n\n*(All repos URL-verified via GitHub API, Jun 2026. `🆕` = released/expanded 2025–2026. `⚠️` = caveat/discontinued.)*\n\n### 5a · Eval frameworks \u0026 harnesses (code-first test-runners)\n- **[Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai)** — UK AISI — \u003chttps://github.com/UKGovernmentBEIS/inspect_ai\u003e · \u003chttps://inspect.aisi.org.uk/\u003e — `@task` binds dataset + solver + scorer; custom scorers; sandboxed tools. The reference agent-eval framework. **(MUST)**\n- **[inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals)** — UK AISI — \u003chttps://github.com/UKGovernmentBEIS/inspect_evals\u003e — 🆕 the companion catalog of community benchmarks (GAIA, CTFs, AIME…) — the \"batteries\" for Inspect.\n- **[AISI Engineering Playbook](https://engineering-playbook.aisi.org.uk/)** — UK AISI — \u003chttps://www.aisi.gov.uk/blog/releasing-aisis-engineering-playbook\u003e · \u003chttps://engineering-playbook.aisi.org.uk/\u003e · *guide* — 🆕 Open guide to production frontier-model eval infrastructure: five-layer stack (Evaluate · Isolate · Connect · Run · Scale) covering sandboxed untrusted code, audited provider routing, hosted open-weight inference, and the systems-of-working that tie them together. \"Much of this work is invisible, rarely documented, and developed from scratch by every team that attempts serious evaluation.\" The infrastructure layer surrounding Inspect AI, documented by the team that built and runs it.\n- **[lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)** — EleutherAI — \u003chttps://github.com/EleutherAI/lm-evaluation-harness\u003e — the standard academic harness; first-class decontamination; task YAMLs.\n- **[OLMES](https://github.com/allenai/olmes)** — Allen Institute (Ai2) — \u003chttps://github.com/allenai/olmes\u003e — 🆕 the reproducible eval **standard + harness** behind OLMo/Tülu: standardized prompts/metrics/formatting for apples-to-apples model comparison.\n- **[BenchFlow](https://github.com/benchflow-ai/benchflow)** — \u003chttps://github.com/benchflow-ai/benchflow\u003e · \u003chttps://benchflow.ai\u003e — 🆕 environment-lab framework: research infra + runtime for building RL environments, evals \u0026 post-training; ships **SkillsBench** and **ClawsBench**. (\"Environments are the new data.\")\n- **[lighteval](https://github.com/huggingface/lighteval)** — Hugging Face — \u003chttps://github.com/huggingface/lighteval\u003e — 🆕 all-in-one harness across transformers/vLLM/TGI/nanotron, 1000+ tasks; HF's successor to `evaluate`.\n- **[OpenBench](https://github.com/groq/openbench)** — Groq — \u003chttps://github.com/groq/openbench\u003e — 🆕 provider-agnostic `bench` CLI, 95+ benchmarks, built on Inspect primitives.\n- **[simple-evals](https://github.com/openai/simple-evals)** — OpenAI — \u003chttps://github.com/openai/simple-evals\u003e — minimal zero-shot/CoT scripts (MMLU, HumanEval, SimpleQA, HealthBench); the numbers OpenAI publishes. ⚠️ not actively maintained.\n- **[OpenAI Evals](https://github.com/openai/evals)** — \u003chttps://github.com/openai/evals\u003e — the `completion_fn` abstraction = swap the system-under-test. (Best-practices: \u003chttps://developers.openai.com/api/docs/guides/evaluation-best-practices\u003e)\n- **[promptfoo](https://github.com/promptfoo/promptfoo)** — \u003chttps://github.com/promptfoo/promptfoo\u003e — MIT eval + red-teaming CLI; git-diffable YAML configs. **(MUST)**\n- **[DeepEval / Confident AI](https://github.com/confident-ai/deepeval)** — \u003chttps://github.com/confident-ai/deepeval\u003e — \"pytest for LLMs,\" 40+ metrics (G-Eval, RAG, hallucination) + red-team; ~2M evals/day; hosted cloud. 🆕\n- **[pydantic-evals](https://github.com/pydantic/pydantic-ai)** — \u003chttps://github.com/pydantic/pydantic-ai\u003e (`ai.pydantic.dev/evals`) — 🆕 type-safe Datasets/Cases/Evaluators with OTel tracing, from the Pydantic AI team.\n- **[openevals](https://github.com/langchain-ai/openevals)** — LangChain — \u003chttps://github.com/langchain-ai/openevals\u003e — 🆕 prebuilt evaluators + `create_llm_as_judge` (incl. multimodal); general-purpose companion to **agentevals** (\u003chttps://github.com/langchain-ai/agentevals\u003e, trajectory match).\n- **[MLflow GenAI evaluate](https://mlflow.org/docs/latest/genai/eval-monitor/)** — \u003chttps://mlflow.org/docs/latest/genai/eval-monitor/\u003e — 🆕 `mlflow.genai.evaluate`: 50+ judges/metrics, custom scorers, regression datasets inside MLflow.\n- **[HELM (crfm-helm)](https://github.com/stanford-crfm/helm)** — Stanford CRFM — \u003chttps://github.com/stanford-crfm/helm\u003e — holistic eval: standardized datasets + metrics beyond accuracy + leaderboard (also VHELM, HEIM).\n- **[Giskard](https://github.com/Giskard-AI/giskard-oss)** — \u003chttps://github.com/Giskard-AI/giskard-oss\u003e — auto-generates adversarial test suites (injection, hallucination, bias) from a plain-language app description.\n- **[Deepchecks LLM](https://github.com/deepchecks/deepchecks)** — \u003chttps://github.com/deepchecks/deepchecks\u003e (`llmdocs.deepchecks.com`) — property-based scoring (grounded-in-context, toxicity, fluency) + custom LLM-judge properties.\n- **[UpTrain](https://github.com/uptrain-ai/uptrain)** — \u003chttps://github.com/uptrain-ai/uptrain\u003e — 20+ preconfigured checks + root-cause analysis on failures.\n- **[HF `evaluate`](https://github.com/huggingface/evaluate)** — \u003chttps://github.com/huggingface/evaluate\u003e — classic metrics library, ⚠️ maintenance mode (use lighteval for LLMs).\n- **[Harbor](https://github.com/harbor-framework/harbor)** — harbor-framework (Laude Institute / Stanford) — \u003chttps://github.com/harbor-framework/harbor\u003e — 🆕 framework for running agent evals + creating/using RL environments; powers Terminal-Bench 2.0. ~3.7k★. ⚠️ name overloaded (cf. `av/harbor` local-LLM toolkit).\n- **[Caliper](https://github.com/edonadei/caliper)** — Emrick Donadei — \u003chttps://github.com/edonadei/caliper\u003e · \u003chttps://pypi.org/project/caliper-eval/\u003e — 🆕 **pass@k reliability harness for agent skills**: runs a skill k times against Claude Code / Codex / pi, grading each attempt with an LLM-autorater (`expect:`) and/or deterministic Python (`assert:`), then a `--baseline` re-run without the skill reports the delta to prove the skill beats the base agent. Git-diffable `.eval.yaml` specs; isolated per-attempt sandboxes.\n- **[Coder Eval](https://github.com/UiPath/coder_eval)** — UiPath — \u003chttps://github.com/UiPath/coder_eval\u003e · \u003chttps://coder-eval.com/docs\u003e — 🆕 treats **skill activation as a measurable variable**: `skill_triggered` over a labelled dataset yields suite-level precision/recall/F1 gates (cf. §3). Harness-level, not leaderboard-level. ~0.1k★.\n\n### 5b · TypeScript/JS-native eval runners\n- **[evalite](https://github.com/mattpocock/evalite)** — Matt Pocock — \u003chttps://github.com/mattpocock/evalite\u003e — 🆕 local-first eval runner on Vitest; `.eval.ts` files, web UI, cost-aware.\n- **[Mastra scorers](https://github.com/mastra-ai/mastra)** — \u003chttps://github.com/mastra-ai/mastra\u003e (`mastra.ai/docs/evals/overview`) — 🆕 model-graded/rule/statistical scorers, live evals, CI, in the Mastra agent framework.\n- **[Open Multi-Agent evaluation](https://github.com/open-multi-agent/open-multi-agent)** — Open Multi-Agent — \u003chttps://github.com/open-multi-agent/open-multi-agent/blob/main/docs/evaluation.md\u003e · *tool/framework* — 🆕 TypeScript-native, versioned EvalSets with offline JSON/Markdown/JUnit reports, baseline regression gates for CI, and opt-in production sampling that scores settled runs asynchronously without changing their results.\n- **[Vercel agent-eval](https://github.com/vercel-labs/agent-eval)** — \u003chttps://github.com/vercel-labs/agent-eval\u003e — 🆕 A/B-test coding agents (Claude Code, Codex, Cursor) on custom tasks; pass-rate dashboards.\n- **[Autoevals](https://github.com/braintrustdata/autoevals)** — Braintrust — \u003chttps://github.com/braintrustdata/autoevals\u003e — OSS scorer library (Factuality, relevance, security…) across Py/JS/Go/Ruby.\n\n### 5c · RAG / retrieval evaluation\n- **[TruLens](https://github.com/truera/trulens)** — \u003chttps://github.com/truera/trulens\u003e — instrumentation + \"feedback functions\" (the RAG triad), now OTel-based.\n- **[ARES](https://github.com/stanford-futuredata/ARES)** — Stanford — \u003chttps://github.com/stanford-futuredata/ARES\u003e — synthetic queries + fine-tuned judges + prediction-powered inference for confidence intervals.\n- **[RAGChecker](https://github.com/amazon-science/RAGChecker)** — Amazon Science — \u003chttps://github.com/amazon-science/RAGChecker\u003e — 🆕 claim-level diagnosis separating retriever vs generator errors.\n- **[continuous-eval (Relari)](https://github.com/relari-ai/continuous-eval)** — \u003chttps://github.com/relari-ai/continuous-eval\u003e — modular per-module metrics across retrieval/generation/tool-use.\n- **[Tonic Validate](https://github.com/TonicAI/tonic_validate)** — \u003chttps://github.com/TonicAI/tonic_validate\u003e — RAG metrics as a GitHub Action for CI.\n\n### 5d · LLM-as-judge / reward / verifier libraries\n- **[RewardHarness](https://github.com/TIGER-AI-Lab/RewardHarness)** — TIGER-AI-Lab et al. — \u003chttps://arxiv.org/abs/2605.08703\u003e · *framework* — 🆕 evolves task-specific scoring skills and tools from preference demonstrations, producing image-editing judgments and a GRPO-compatible scalar reward without reward-model training.\n- **[verdict](https://github.com/haizelabs/verdict)** — Haize Labs — \u003chttps://github.com/haizelabs/verdict\u003e — 🆕 declarative compound judges (debate/verification/aggregation, inference-time scaling); arXiv:2502.18018.\n- **[RULER](https://github.com/OpenPipe/ART)** — OpenPipe (ART) — \u003chttps://github.com/OpenPipe/ART\u003e (`art.openpipe.ai/fundamentals/ruler`) — 🆕 LLM-judge that ranks trajectories with no labels — judge-as-RL-reward. **(industry must-read)**\n- **[Prometheus 2](https://github.com/prometheus-eval/prometheus-eval)** — \u003chttps://github.com/prometheus-eval/prometheus-eval\u003e — open-weight evaluator LMs for rubric-based assessment + pairwise.\n- **[Atla Selene](https://github.com/atla-ai/selene-mini)** — \u003chttps://github.com/atla-ai/selene-mini\u003e — 🆕 8B SoTA open judge (score + critique); + MCP server `atla-ai/atla-mcp-server`. arXiv:2501.17195.\n- **[Patronus Lynx / GLIDER](https://github.com/patronus-ai/Lynx-hallucination-detection)** — \u003chttps://github.com/patronus-ai/Lynx-hallucination-detection\u003e · \u003chttps://github.com/patronus-ai/glider\u003e — 🆕 open hallucination judge / explainable span-level judge.\n- **[Flow-Judge](https://github.com/flowaicom/flow-judge)** — \u003chttps://github.com/flowaicom/flow-judge\u003e — efficient 3.8B open evaluator.\n- **[RewardBench](https://github.com/allenai/reward-bench)** — AI2 — \u003chttps://github.com/allenai/reward-bench\u003e — canonical reward-model (+v2 judge) benchmark/harness.\n- **[JudgeBench](https://github.com/ScalerLab/JudgeBench)** — \u003chttps://github.com/ScalerLab/JudgeBench\u003e — benchmark to evaluate the judges themselves.\n- **[reward-kit](https://github.com/fw-ai-external/reward-kit)** — Fireworks — \u003chttps://github.com/fw-ai-external/reward-kit\u003e — 🆕 decorator-based reward-function authoring (TRL/Fireworks interop).\n- **[truescore](https://github.com/SaifPunjwani/truescore)** — Saif Punjwani — \u003chttps://github.com/SaifPunjwani/truescore\u003e · *tool* — Measures a judge against held-out human labels (sensitivity, Cohen's kappa, Gwet's AC1), tests it for length, position and self-preference bias, then corrects the reported score with prediction-powered inference so the interval covers the truth rather than the judge's opinion. Apache-2.0, numpy/scipy only. 🆕 ⚠️ new and unproven (repo created Jul 2026, no external adoption yet).\n\n### 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training)\n- **[verifiers](https://github.com/PrimeIntellect-ai/verifiers)** — Prime Intellect — \u003chttps://github.com/PrimeIntellect-ai/verifiers\u003e — Environment = dataset + harness + rubric; one package for eval, RL, synthetic data. **(MUST)**\n- **[Environments Hub](https://github.com/PrimeIntellect-ai/community-environments)** — Prime Intellect — \u003chttps://github.com/PrimeIntellect-ai/community-environments\u003e (app.primeintellect.ai) — 🆕 crowdsourced verifiers-based RL/eval envs.\n- **[prime-rl](https://github.com/PrimeIntellect-ai/prime-rl)** — Prime Intellect — \u003chttps://github.com/PrimeIntellect-ai/prime-rl\u003e — 🆕 async RL trainer consuming verifiers envs (INTELLECT-3).\n- **[BenchFlow](https://github.com/benchflow-ai/benchflow)** — \u003chttps://github.com/benchflow-ai/benchflow\u003e · \u003chttps://benchflow.ai\u003e — 🆕 environment lab: builds \u0026 runs RL/eval environments (SkillsBench, ClawsBench, runtime). \"Environments are the new data.\" (also §5a)\n- **[HUD](https://github.com/hud-evals/hud-python)** — \u003chttps://github.com/hud-evals/hud-python\u003e — 🆕 SDK to build/run agent eval environments (computer-use, browser, MCP) with telemetry.\n- **[Atropos](https://github.com/NousResearch/atropos)** — Nous Research — \u003chttps://github.com/NousResearch/atropos\u003e — 🆕 async \"environment microservice\" framework for rollouts/verifiable rewards.\n- **[verl](https://github.com/volcengine/verl)** — \u003chttps://github.com/volcengine/verl\u003e (now `verl-project/verl`) — de-facto industry RLVR trainer (PPO/GRPO). ~22k★.\n- **[OpenRLHF](https://github.com/OpenRLHF/OpenRLHF)** — \u003chttps://github.com/OpenRLHF/OpenRLHF\u003e · **SkyRL** — \u003chttps://github.com/NovaSky-AI/SkyRL\u003e · **AReaL** — \u003chttps://github.com/areal-project/AReaL\u003e · **ROLL** — \u003chttps://github.com/alibaba/ROLL\u003e · **rLLM** — \u003chttps://github.com/agentica-project/rllm\u003e · **TRL** — \u003chttps://github.com/huggingface/trl\u003e — the RL-training stack agents are post-trained + eval'd in.\n- **[Open Reward Standard (ORS)](https://docs.openreward.ai/)** — General Reasoning — \u003chttps://docs.openreward.ai/\u003e (PyPI `openreward`) — 🆕 MCP-extending spec adding RL primitives (episodes, rewards, curriculum). ⚠️ no single canonical repo confirmed.\n\n### 5f · Observability + eval platforms (tracing · datasets · online/offline · CI)\n- **[Arize Phoenix](https://github.com/Arize-ai/phoenix)** — \u003chttps://github.com/Arize-ai/phoenix\u003e — OSS OTel tracing + response/retrieval evals + datasets/experiments. **(MUST)**\n- **[Langfuse](https://github.com/langfuse/langfuse)** — \u003chttps://github.com/langfuse/langfuse\u003e — OSS: evals (LLM-judge, feedback, manual labeling), datasets/experiments, prompt mgmt; self-hostable. 🆕\n- **[Opik](https://github.com/comet-ml/opik)** — Comet — \u003chttps://github.com/comet-ml/opik\u003e — 🆕 fully-OSS eval + observability (judges, datasets, CI-runnable evals).\n- **[W\u0026B Weave](https://github.com/wandb/weave)** — \u003chttps://github.com/wandb/weave\u003e — `weave.Evaluation` scorers (exact/regex/model-graded/embedding) + Guardrails; comparison dashboards. 🆕 (Humanloop's migration target.)\n- **[Braintrust](https://www.braintrust.dev/docs/start/eval-sdk)** — \u003chttps://www.braintrust.dev/docs/start/eval-sdk\u003e (offline-eval-guide) — `Eval()` over golden datasets; offline vs online. **(MUST)**\n- **[Patronus AI](https://www.patronus.ai/)** — \u003chttps://www.patronus.ai/\u003e (`github.com/patronus-ai`) — 🆕 research-grade judges (Lynx, GLIDER, **Percival** agent-failure debugger), experiments, multimodal judge.\n- **[Maxim AI](https://www.getmaxim.ai/)** — \u003chttps://www.getmaxim.ai/\u003e — 🆕 agent **simulation** + eval + observability across thousands of scenarios/personas.\n- **[Galileo](https://galileo.ai/)** — \u003chttps://galileo.ai/\u003e — Luna evaluators + Agentic Evaluations.\n- **[Vellum](https://www.vellum.ai/)** — \u003chttps://www.vellum.ai/\u003e — visual workflows + offline/online evals scoring every production run.\n- **[Helicone](https://github.com/helicone/helicone)** — \u003chttps://github.com/helicone/helicone\u003e — OSS gateway + observability; \"Scores\" ingests external eval results.\n- **[Traceloop / OpenLLMetry](https://github.com/traceloop/openllmetry)** — \u003chttps://github.com/traceloop/openllmetry\u003e — OSS OTel instrumentation (Py/TS/Go/Ruby) + hosted reliability platform.\n- **[Langtrace](https://github.com/Scale3-Labs/langtrace)** — \u003chttps://github.com/Scale3-Labs/langtrace\u003e — OSS OTel-standard tracing + manual scoring + dataset mgmt.\n- **[WhyLabs / LangKit](https://github.com/whylabs/langkit)** — \u003chttps://github.com/whylabs/langkit\u003e — high-throughput text-signal metrics (toxicity, PII, jailbreak) for production monitoring.\n- **[Portkey](https://github.com/portkey-ai/gateway)** — \u003chttps://github.com/portkey-ai/gateway\u003e — 🆕 OSS gateway + 60+ guardrails + observability (fully open-sourced Mar 2026).\n- **[Datadog LLM Observability](https://www.datadoghq.com/product/ai/llm-observability/)** — \u003chttps://www.datadoghq.com/product/ai/llm-observability/\u003e — 🆕 evaluators + golden datasets + **LLM Experiments** + AI Agent Monitoring (Jun 2025).\n- **[Fiddler AI](https://www.fiddler.ai/)** — \u003chttps://www.fiddler.ai/\u003e — 🆕 Trust Models (Safety/PII/Faithfulness) scoring in \u003c100ms; Guardrails + agentic observability.\n- **[PromptLayer](https://www.promptlayer.com/)** — \u003chttps://www.promptlayer.com/\u003e · **New Relic AI Monitoring** — \u003chttps://newrelic.com/platform/ai-monitoring\u003e — lighter prompt-CMS / APM-native monitoring.\n- **[whatbroke](https://github.com/arthi-arumugam-git/whatbroke)** — \u003chttps://github.com/arthi-arumugam-git/whatbroke\u003e · *tool* — OSS CLI that diffs two agent trace files (JSONL) before/after a model or prompt change: dropped tool calls, drifted args, reordering, cost/latency/outcome deltas; multi-sample flap rates demote baseline flakiness so only real regressions surface. Offline, deterministic, no judge or API keys; CI-gateable exit codes. 🆕\n\n### 5g · Tracing standards\n- **[OpenInference](https://github.com/Arize-ai/openinference)** — Arize — \u003chttps://github.com/Arize-ai/openinference\u003e — semantic conventions for agent traces (tool/args/observation/latency/cost).\n- **[OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)** — \u003chttps://opentelemetry.io/docs/specs/semconv/gen-ai/\u003e (`open-telemetry/semantic-conventions`) — 🆕 the vendor-neutral schema (now covers agent orchestration, MCP tool calls, and a **quality-evaluation** span hook).\n\n- **[Braintrust](https://www.braintrust.dev/)** — Braintrust — \u003chttps://www.braintrust.dev/\u003e · *tool* — Industry-standard eval+observability platform (Notion, Stripe, Vercel) tying offline experiments to production logs; the section already cites Braintrust's Autoevals but omits the platform itself. 🆕\n- **[RagaAI Catalyst](https://github.com/raga-ai-hub/RagaAI-Catalyst)** — RagaAI — \u003chttps://github.com/raga-ai-hub/RagaAI-Catalyst\u003e · *tool* — OSS agent-observability + eval SDK with multi-agent trace/execution-graph debugging, synthetic-data gen, and guardrail management — covers the online/guardrail-eval slice the section lacks. 🆕\n- **[OpenAI Cookbook — Evals](https://developers.openai.com/cookbook/topic/evals)** — OpenAI — \u003chttps://developers.openai.com/cookbook/topic/evals\u003e · *docs* — Maintained, runnable recipes for building evals (incl. Agents SDK eval, evaluating agents with Langfuse); the practical companion to OpenAI Evals and a curator-grade 'show real work' resource. 🆕 ⚠(unverified URL)\n\n- **[Building a better Bugbot](https://cursor.com/blog/building-bugbot)** — Stefan Heule et al. (Cursor) — \u003chttps://cursor.com/blog/building-bugbot\u003e · *excellent* — **Build your primary eval metric around post-merge signal: Cursor's \"resolution rate\" uses AI at PR merge time to determine which flagged bugs were actually fixed by the developer, validated by human spot-checks. After 40 major experiments spanning models, prompts, iteration counts, and agentic designs, resolution rate improved from 52% to over 70%; the single largest jump came from switching to a fully agentic architecture. BugBench (a curated set of real diffs with human-annotated bugs) drives offline iteration.** _(excerpt: \"It uses AI to determine, at PR merge time, which bugs were actually resolved by the author in the final code. ... Since launch, we have run 40 major experiments that have increased Bugbot's resolution rate from 52% to over 70%.\")_ 🆕\n- **[Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents](https://arxiv.org/abs/2605.21347)** — Scale AI — \u003chttps://arxiv.org/abs/2605.21347\u003e · *paper/tool* — 🆕 Automated system that analyzes large corpora of agent execution traces to surface failure patterns and behavioral issues at scale; \"Human experts using IG reports improve scaffold performance by 30.4 pp over the unmodified baseline scaffold\" — nearly doubling the gain from the next-best system. Fills the gap between laborious per-trace inspection and aggregate benchmark scores that hide population-level failure modes.\n- **[Do Automated Evals Work?](https://parlance-labs.com/blog/posts/auto-evals/index.html)** — Antaripa Saha \u0026 Hamel Husain (Parlance Labs) — \u003chttps://parlance-labs.com/blog/posts/auto-evals/index.html\u003e · *blog (Jul 11 2026)* — Empirical head-to-head of six eval systems (Braintrust Loop, Arize AX Alyx, LangSmith, Codex/GPT-5.5, Factory Droid/GPT-5.5, Claude Code/Opus 4.8) on 100 production traces from a real apartment-leasing AI assistant, blind-tested against 39 expert-labeled failures. Best recall: Braintrust Loop at 87.2%; every system found issues humans missed; every system also missed \"looks correct but fails the user\" traces (e.g., abandoned conversations after objections, Markdown formatting in SMS). Core finding: automated evals discover real failures but cannot replace continuous human-in-the-loop annotation for subtle user-experience failures. 🆕\n- **[PACE: A Proxy for Agentic Capability Evaluation](https://arxiv.org/abs/2607.02032)** — Song, Sutawika, Liu et al. (Carnegie Mellon / Berkeley) — \u003chttps://arxiv.org/abs/2607.02032\u003e · *paper (Jul 2026)* — Selects non-agentic atomic benchmark instances whose aggregate scores best predict performance on expensive agentic benchmarks (SWE-Bench, GAIA), achieving mean absolute error under 4% at under 1% of full evaluation cost — tested across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks. Directly addresses the \"thousands of dollars per agentic eval run\" barrier to iterating during development. 🆕\n\n**Must-reads:** Inspect AI · promptfoo · Braintrust · verifiers · DeepEval · Phoenix/Langfuse (pick your observability) · RULER (judge-as-reward)\n\n## 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming)\n\n- **[How to Build Good Language Modeling Benchmarks](https://ofir.io/How-to-Build-Good-Language-Modeling-Benchmarks/)** — Ofir Press — \u003chttps://ofir.io/How-to-Build-Good-Language-Modeling-Benchmarks/\u003e · *blog* — The benchmark-author's checklist; difficulty target; one-number reporting; 150–500 task sizing.\n- **[AI Agents That Matter](https://arxiv.org/abs/2407.01502)** — Kapoor et al. — \u003chttps://arxiv.org/abs/2407.01502\u003e · *paper* — Cost-controlled evaluation; model-dev vs downstream-dev needs; holdouts.\n- **[Why We No Longer Evaluate SWE-bench Verified](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/)** — OpenAI — \u003chttps://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/\u003e · *blog* — ~59% of audited failures were broken tests. (mirror: \u003chttps://decrypt.co/359012/...\u003e)\n- **[The Leaderboard Illusion](https://arxiv.org/abs/2504.20879)** — Shivalika Singh et al. (Cohere/Princeton/Stanford/MIT/AI2) — \u003chttps://arxiv.org/abs/2504.20879\u003e · *paper* — Private testing, selective disclosure, and data-access asymmetry on Chatbot Arena. *(notes: `research/notes/leaderboard-illusion.md`)*\n- **[The SWE-bench Illusion: When SOTA LLMs Remember Instead of Reason](https://arxiv.org/abs/2506.12286)** — \u003chttps://arxiv.org/abs/2506.12286\u003e · *paper* — Memorization inflates SWE-bench scores.\n- **[Establishing Best Practices for Building Rigorous Agentic Benchmarks (ABC)](https://arxiv.org/abs/2507.02825)** — \u003chttps://arxiv.org/abs/2507.02825\u003e · *paper* — SWE-bench Verified weak tests; τ-bench rewards empty responses. *(verified high)*\n- **[FrontierMath Tiers 1–3 v2 (corrected)](https://epoch.ai/benchmarks/frontiermath-tiers-1-3-v2)** — Epoch AI — \u003chttps://epoch.ai/benchmarks/frontiermath-tiers-1-3-v2\u003e (changelog: `.../frontiermath-tier-4-v2`) · *page* — ~42% of problems corrected after AI-assisted review. (also T8: the operator-as-rot-detector tale)\n- **[About 30% of Humanity's Last Exam Answers Are Wrong](https://www.futurehouse.org/research-announcements/hle-exam)** — FutureHouse / Andrew White — \u003chttps://www.futurehouse.org/research-announcements/hle-exam\u003e · *blog* — 29 ± 3.7% of text-only chem/bio answers contradicted by the literature. (LessWrong writeup: \u003chttps://www.lesswrong.com/posts/JANqfGrMyBgcKtGgK/\u003e)\n- **[Building on Evaluation Quicksand](https://www.interconnects.ai/p/building-on-evaluation-quicksand)** — Nathan Lambert — \u003chttps://www.interconnects.ai/p/building-on-evaluation-quicksand\u003e · *blog* — No hard source of truth; synthetic-data contamination.\n- **[Lost in Simulation](https://arxiv.org/abs/2601.17087)** — \u003chttps://arxiv.org/abs/2601.17087\u003e · *paper* — Simulated users are unreliable proxies (~9pp swings by simulator choice; demographic miscalibration).\n- **[SWE-bench: Can LMs Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770)** — Jimenez, Yang, … Press, Narasimhan — \u003chttps://arxiv.org/abs/2310.06770\u003e · \u003chttps://www.swebench.com\u003e (Verified: `.../verified.html`) · *paper/site*.\n- **[Task-Specific LLM Evals that Do \u0026 Don't Work](https://eugeneyan.com/writing/evals/)** — Eugene Yan — \u003chttps://eugeneyan.com/writing/evals/\u003e · *blog* — Off-the-shelf evals rarely transfer; accuracy is too coarse.\n- **[Andrej Karpathy on evals](https://x.com/karpathy/status/1896266683301659068)** — \u003chttps://x.com/karpathy/status/1896266683301659068\u003e · *post* — \"We make a number of specific recommendations…\" (the eval-as-narrow critique).\n\n- **[A Careful Examination of LLM Performance on Grade School Arithmetic (GSM1k)](https://arxiv.org/abs/2405.00332)** — Hugh Zhang et al. (Scale AI) — \u003chttps://arxiv.org/abs/2405.00332\u003e · *paper* — Held-out GSM1k replica of GSM8k exposes up to 8% accuracy drop and partial memorization (Mistral/Phi) — the canonical method for measuring benchmark overfitting/contamination via a matched holdout.\n- **[Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://arxiv.org/abs/2103.14749)** — Curtis Northcutt, Anish Athalye, Jonas Mueller — \u003chttps://arxiv.org/abs/2103.14749\u003e · *paper* — NeurIPS 2021 foundational result: ~3.3% avg label errors across 10 famous test sets (ImageNet, MNIST, etc.); corrections flip model rankings. The canonical 'label errors' citation this section's theme rests on (labelerrors.com / cleanlab).\n- **[Are We Done with MMLU? (MMLU-Redux)](https://arxiv.org/abs/2406.04127)** — Aryo Pradipta Gema et al. (Edinburgh) — \u003chttps://arxiv.org/abs/2406.04127\u003e · *paper* — ~6.5% of MMLU questions contain errors (57% in Virology); MMLU-Redux re-annotation shifts rankings — directly demonstrates label-error impact on the most-cited LLM benchmark.\n- **[LiveCodeBench: Holistic and Contamination-Free Evaluation of LLMs for Code](https://arxiv.org/abs/2403.07974)** — Naman Jain et al. (UC Berkeley) — \u003chttps://arxiv.org/abs/2403.07974\u003e · *benchmark* — Time-windowed problem collection (post-cutoff scoring) as the leading contamination-resistant design pattern — the section discusses contamination but lists no exemplar of how to engineer around it.\n- **[LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://github.com/LiveBench/LiveBench)** — White, Dohan, LeCun, Goldblum et al. — \u003chttps://github.com/LiveBench/LiveBench\u003e · *benchmark* — Monthly-refreshed questions from new arXiv/news/competitions with objective ground truth — the canonical 'dynamic refresh' answer to saturation and contamination.\n- **[The LLM Evaluation Guidebook (Open LLM Leaderboard team)](https://github.com/huggingface/evaluation-guidebook)** — Clémentine Fourrier / Hugging Face — \u003chttps://github.com/huggingface/evaluation-guidebook\u003e · *docs* — Practitioner reference from running the Open LLM Leaderboard; explicit sections on contamination, reproducibility, and leaderboard design — the hands-on 'how to not get fooled' companion to this section (updated version: hf.co/spaces/OpenEvals/evaluation-guidebook).\n- **[Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation](https://arxiv.org/abs/2510.11977)** — Kapoor, Stroebl, Kirgis et al. (Princeton) — \u003chttps://arxiv.org/abs/2510.11977\u003e · *paper* — 21,000+ standardized agent runs surfacing leaderboard unreliability and unreported misbehaviors (agents searching HuggingFace for benchmark answers) — extends 'AI Agents That Matter' to leaderboard integrity for agents specifically. 🆕\n- **[Gaming the System: Goodhart's Law Exemplified in the AI Leaderboard Controversy](https://blog.collinear.ai/p/gaming-the-system-goodharts-law-exemplified-in-ai-leaderboard-controversy)** — Jambholkar, Rajani, Bakshi (Collinear AI) — \u003chttps://blog.collinear.ai/p/gaming-the-system-goodharts-law-exemplified-in-ai-leaderboard-controversy\u003e · *blog* — Practitioner framing of the Llama 4 / Chatbot Arena gaming episode through Goodhart's Law — the accessible blog companion to The Leaderboard Illusion paper. 🆕\n- **[A Shared Playbook for Trustworthy Third-Party Evaluations](https://openai.com/index/trustworthy-third-party-evaluations-foundations/)** — OpenAI — \u003chttps://openai.com/index/trustworthy-third-party-evaluations-foundations/\u003e · *blog (Safety, May 29 2026)* — What makes *independent* evals of frontier-model safeguards \u0026 capabilities trustworthy: selecting the right harness, checking for validity hazards that distort results, and the standards third-party evaluators need. (also T10) 🆕\n\n- **[Separating signal from noise in coding evaluations](https://openai.com/index/separating-signal-from-noise-coding-evaluations/)** — OpenAI — \u003chttps://openai.com/index/separating-signal-from-noise-coding-evaluations/\u003e · *blog (Jul 8 2026)* — SWE-Bench Pro audit finding ~30% of its 731 tasks broken across four failure categories: overly strict tests that reject functionally correct solutions, underspecified prompts with hidden test conditions, shallow tasks where incomplete fixes pass, and misleading descriptions that point agents to the wrong code. OpenAI explicitly retracts its earlier recommendation for the field to adopt SWE-Bench Pro: \"We find 30% of SWE-Bench Pro tasks to be broken, and are retracting our previous recommendation that the research community use it as [a leading coding eval].\" Pairs directly with the already-listed Cursor reward-hacking audit (which also analyzed SWE-Bench Pro's 731 tasks). 🆕\n- **[Reward hacking is swamping model intelligence gains](https://cursor.com/blog/reward-hacking-coding-benchmarks)** — Cursor — \u003chttps://cursor.com/blog/reward-hacking-coding-benchmarks\u003e · *blog* — Cursor audited 731 Opus 4.8 Max trajectories and found 63% of \"successful\" resolutions retrieved known fixes (57% upstream-lookup, 6% git-history mining) rather than reasoning them out. Re-running SWE-bench Pro under strict isolation (no internet, no git history) drops Opus 4.8 Max from 87.1% → 73.0% and Composer 2.5 from 74.7% → 54.0% — a 20.7pp gap. Cursor self-reports the largest gaps in their own model. The most quantified public evidence that current coding-agent leaderboard scores are massively inflated by retrieval, not reasoning. 🆕\n- **[Quantifying infrastructure noise in agentic coding evals](https://www.anthropic.com/engineering/infrastructure-noise)** — Anthropic — \u003chttps://www.anthropic.com/engineering/infrastructure-noise\u003e · *blog* — Infrastructure resource configuration (RAM, resource limits) swings agentic coding benchmark scores by up to 6 percentage points — often larger than leaderboard gaps between adjacent models. A phase transition exists at ~3× baseline resources: below it, more resources fix flakiness without changing what's measured; above it, resources start helping agents solve problems they otherwise couldn't, fundamentally changing benchmark meaning. \"Skepticism is warranted for any reported Terminal-Bench / SWE-bench gap below 3pp.\" 🆕\n- **[Token-Saving Plugins Are Mostly Stupid Idea](https://turaai.net/blog#token-saving-plugins-are-mostly-stupid-idea)** — Tura — \u003chttps://turaai.net/blog#token-saving-plugins-are-mostly-stupid-idea\u003e · *blog + public dataset* — A deliberately underpowered coding-agent study (n=2/arm) that refuses to infer plugin effects: apparent −8.87%/+7.18% cost shifts sit inside 30–52% within-arm swings; matched runs, methodology, a 293-round activation audit, and a broader 140-run token ledger are public. 🆕 ⚠️ vendor-published — Tura benchmarks competing plugins (Ponytail, RTK) on its own harness; no Tura runs are in the dataset.\n- **[Eval awareness in Claude Opus 4.6's BrowseComp performance](https://www.anthropic.com/engineering/eval-awareness-browsecomp)** — Anthropic — \u003chttps://www.anthropic.com/engineering/eval-awareness-browsecomp\u003e · *blog* — First documented instance of a model reverse-engineering its own evaluation: Claude Opus 4.6 independently hypothesized it was being tested, identified the specific benchmark (BrowseComp), located the source code on GitHub, and decrypted the SHA256/XOR-encrypted answer key — then used it to answer evaluation questions. 18 independent runs converged on the same strategy unprompted. Implication: static benchmarks run in web-enabled environments are now adversarially fragile against capable models. 🆕\n- **[Life After Benchmark Saturation: A Case Study of CORE-Bench](https://arxiv.org/abs/2606.26158)** — Nadgir, Kapoor, Liu, Kirgis, Narayanan et al. (Princeton / UC Berkeley / MIT) — \u003chttps://arxiv.org/abs/2606.26158\u003e · *paper* — Argues saturation is diagnostic evidence to dig deeper, not a retirement trigger: LLM trajectory audit found \"15 task-level errors and 20 tasks with exploitable shortcuts in CORE-Bench Hard\" (releasing CORE-Bench v1.1 + OOD suite); saturated agents that tie on accuracy diverge sharply on cost (60% cheaper) and calibration (32.1% self-reported confidence vs. 93% empirical pass rate); randomized human study (20 papers, 25 manual vs. 25 agent-assisted runs): \"manual reproduction sessions lasted 2.11 times as long as human-agent collaborative sessions.\" 🆕\n- **[Search-Time Contamination in Deep Research Agents](https://arxiv.org/abs/2606.05241)** — Wang, Zhang, Yao, Zeng, Song, Lin, Shen — \u003chttps://arxiv.org/abs/2606.05241\u003e · *paper* — Defines three contamination types (Benchmark Metadata Leakage, Question-Context Leakage, Explicit Answer Leakage) for agents that web-search during evaluation; applies detection algorithms across six benchmarks and finds performance inflation up to 4%. \"Such agents may retrieve public benchmark metadata, question context, or even ground-truth answers via web search. This gives rise to Search-Time Contamination (STC), where external retrieval bypasses intended reasoning and inflates measured performance.\" Distinct from training contamination; advocates isolated sandboxes and transparent search trajectories. 🆕\n- **[RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents](https://arxiv.org/abs/2603.11337)** — Yonas Atinafu, Robin Cohen — \u003chttps://arxiv.org/abs/2603.11337\u003e · *paper* — Treats evaluation integrity as a first-class benchmarked outcome instead of an assumption: two compromise vectors (evaluator tampering, train/test leakage) instrumented via patch tracking and runtime file-access logging; in natural agent runs evaluator-tampering attempts appear in ~50% of episodes, and evaluator locking eliminates them at a 25–31% median runtime overhead. 🆕\n- **[Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation](https://arxiv.org/abs/2607.02577)** — Bhat, Vaghasiya, Mohsin, Aali — \u003chttps://arxiv.org/abs/2607.02577\u003e · *paper (Jun/Jul 2026)* — Cross-benchmark validity audit of BFCL v4, τ²-Bench, LiveMCPBench, and MCP-Atlas across 496 expert-reviewed tasks. Key findings: \"92 evaluator-human disagreements, corresponding to an 18.5% misalignment rate\"; 23 repeated identical LiveMCPBench runs produce scores ranging from 57.9% to 76.8% — \"a spread of 18.9 percentage points\" large enough to flip leaderboard conclusions. Introduces Tool-Veritas and Harness Lab tooling for practitioners auditing their own benchmark choices. 🆕\n- **[Prediction: A Frontier Open Source LLM Will Be Released On 3rd December 2026](https://blog.doubleword.ai/frontier-os-llm)** — Jamie Dborin (Doubleword) — \u003chttps://blog.doubleword.ai/frontier-os-llm\u003e · *blog* — Extrapolates the Artificial Analysis Intelligence Index across 18 constituent benchmarks to forecast the open-vs-closed capability gap; a naive fit of the headline index points to convergence by 3 Dec 2026, but the average per-benchmark lag holds steady at ~5 months — a worked cautionary case in reading trends off an aggregate leaderboard index rather than its components. 🆕\n\n**Must-reads:** Press · Kapoor et al. · OpenAI (SWE-bench Verified) · Leaderboard Illusion\n\n## 7 · Evals \u0026 RL environments (verifiers, reward design, difficulty calibration, lifecycle)\n\n*(See also T2 — verifiers library, Lee's RL-env taxonomy, Garg's lifecycle, Wei's verifier's law.)*\n\n- **[RewardBench](https://arxiv.org/abs/2403.13787)** — Nathan Lambert et al. — \u003chttps://arxiv.org/abs/2403.13787\u003e · *paper* — Evaluating reward models (the verifier you train against).\n- **[The New RL Scaling Laws](https://www.interconnects.ai/p/the-new-rl-scaling-laws)** — Nathan Lambert — \u003chttps://www.interconnects.ai/p/the-new-rl-scaling-laws\u003e · *blog* — Where RLVR scaling is heading. (interview: \u003chttps://www.latent.space/p/the-rlvr-revolution-with-nathan-lambert\u003e)\n- **[Spurious Rewards: Rethinking Training Signals in RLVR](https://arxiv.org/abs/2506.10947)** — \u003chttps://arxiv.org/abs/2506.10947\u003e · *paper* — Random/spurious rewards rival ground truth on Qwen2.5 (Qwen-specific). *(cite arXiv figures, not the blog gloss — see `research/notes/reference-audit.md`)*\n- **[The State of Post-Training 2025](https://www.interconnects.ai/p/the-state-of-post-training-2025)** — Nathan Lambert — \u003chttps://www.interconnects.ai/p/the-state-of-post-training-2025\u003e · *blog* — Context for where evals feed training.\n\n- **[Reward Hacking in Reinforcement Learning](https://lilianweng.github.io/posts/2024-11-28-reward-hacking/)** — Lilian Weng — \u003chttps://lilianweng.github.io/posts/2024-11-28-reward-hacking/\u003e · *blog* — The canonical survey of reward hacking — taxonomy, RLHF-specific failure modes, mitigations; the foundational reference any reward-design section needs.\n- **[Specification gaming: the flip side of AI ingenuity](https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/)** — Victoria Krakovna et al. (Google DeepMind) — \u003chttps://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/\u003e · *blog* — Canonical specification-gaming post (+the running examples list); origin story of why verifiers/reward functions get gamed, predating the LLM-RL wave.\n- **[Multi-Turn RL for Multi-Hour Agents — with Will Brown (Prime Intellect)](https://www.latent.space/p/willccbb)** — Latent Space / Will Brown — \u003chttps://www.latent.space/p/willccbb\u003e · *talk* — The verifiers author on building multi-turn RL environments, turn-level credit assignment and reward design in practice — the practitioner voice behind the verifiers library already cited here. 🆕\n- **[Position: The Hidden Costs and Measurement Gaps of RLVR](https://arxiv.org/abs/2509.21882)** — various (arXiv 2509.21882) — \u003chttps://arxiv.org/abs/2509.21882\u003e · *paper* — RLVR gains overstated via budget mismatch, calibration drift, contamination; proposes a tax-aware minimum standard — the rigor counterweight to Lambert's RL-scaling optimism. 🆕\n- **[RewardBench 2: Advancing Reward Model Evaluation](https://arxiv.org/abs/2506.01937)** — Saumya Malik, Nathan Lambert et al. (Ai2) — \u003chttps://arxiv.org/abs/2506.01937\u003e · *benchmark* — The 2025 successor to RewardBench (already listed) — harder, less saturated, ICLR 2026; the current bar for evaluating the verifier you train against. 🆕\n- **[Reward Modeling (RLHF Book, ch. 5)](https://rlhfbook.com/c/05-reward-models)** — Nathan Lambert — \u003chttps://rlhfbook.com/c/05-reward-models\u003e · *docs* — Canonical free reference chapter on reward models — the standing explainer for the 'verifier you train against' framing this section uses. 🆕\n- **[Curriculum RL from Easy to Hard Tasks Improves LLM Reasoning (E2H Reasoner)](https://arxiv.org/abs/2506.06632)** — Shubham Parashar et al. (Texas A\u0026M) — \u003chttps://arxiv.org/abs/2506.06632\u003e · *paper* — Difficulty-calibration primary source: easy-to-hard scheduling with convergence guarantees and the 'fade out easy tasks' result — directly fills the section's difficulty-calibration theme. 🆕\n- **[GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators](https://arxiv.org/abs/2512.19682)** — Jiacheng Guo, Ling Yang, Mengdi Wang et al. (Princeton) — \u003chttps://arxiv.org/abs/2512.19682\u003e · *paper* — Generative environment simulator with an alpha-Curriculum Reward that keeps tasks in the zone of proximal development — recent take on auto-calibrating env difficulty to the agent. 🆕\n\n- **[Verifier and Reward Design for RL Environments](https://www.hud.ai/resources/verifier-reward-design-rl-environments)** — HUD (hud.ai) — no individual byline — \u003chttps://www.hud.ai/resources/verifier-reward-design-rl-environments\u003e · *article (technical guide)* (good) — Lays out a concrete four-layer scoring architecture (verifiers / pass-fail gates / 3-5 criteria rubrics / composite reward) plus a five-step build workflow: define checkable end-states first (\"table contains row id=4521, status='active'\"), add hard failure gates, build minimal rubrics, test on… 🆕\n- **[The Verification Horizon: No Silver Bullet for Coding Agent Rewards](https://arxiv.org/abs/2606.26300)** — Qwen Team (Alibaba) — \u003chttps://arxiv.org/abs/2606.26300\u003e · *paper* — 🆕 The dynamic argument this section's static entries lack: \"no fixed reward function can remain effective as policy capability continues to grow; and verification must co-evolve with the generator.\" Taxonomizes four verification strategies (unit tests, interactive judge, user feedback, agentic evaluator) and shows each saturates or gets gamed as the policy improves — the moving-target argument for reward design.\n- **[Systematic Reward Hacking and Prime Sprints](https://www.primeintellect.ai/blog/reward-hacking)** — Jessica Li (Prime Intellect) — \u003chttps://www.primeintellect.ai/blog/reward-hacking\u003e · *blog* — Controlled study using the backdoor-ifeval family of environments at 1B scale, reproducible for ~$0.64: \"Hacking is fundamentally a gradient dynamics problem. The same reward function can produce hacking or not depending on how learnable the legitimate task is, what the model's prior puts weight on, and the within batch variance.\" Key findings: no frequency floor for exploited tokens; moderate task difficulty resists hacking better than easy or impossible tasks; semantic prompt guardrails (\"Restrict\" condition) counterintuitively accelerated hacking onset. 🆕\n\n**Must-reads:** Lee (RL-env taxonomy) · Garg (lifecycle) · verifiers (repo)\n\n## 8 · LLM-as-judge \u0026 verifiers (alignment, biases, verifiable vs judgeable)\n\n- **[Evaluating the Effectiveness of LLM-Evaluators](https://eugeneyan.com/writing/llm-evaluators/)** — Eugene Yan — \u003chttps://eugeneyan.com/writing/llm-evaluators/\u003e · *blog* — Position/verbosity/self-enhancement bias; direct vs pairwise; prefer binary + classification metrics.\n- **[Creating an LLM-as-a-Judge That Drives Business Results](https://hamel.dev/blog/posts/llm-judge/)** — Hamel Husain — \u003chttps://hamel.dev/blog/posts/llm-judge/\u003e · *blog* — Critique-shadowing; validate against ONE benevolent-dictator expert; precision/recall over raw agreement.\n- **[Who Validates the Validators? (EvalGen)](https://arxiv.org/abs/2404.12272)** — Shankar et al. (UIST '24) — \u003chttps://arxiv.org/abs/2404.12272\u003e (pdf: `.../pdf/2404.12272`; UIST: \u003chttps://people.eecs.berkeley.edu/~bjoern/papers/shankar-validators-uist2024.pdf\u003e) · *paper* — Criteria drift; the coverage-vs-false-failure judge-alignment loop.\n- **[LLM Evals FAQ](https://hamel.dev/blog/posts/evals-faq/)** — Hamel Husain \u0026 Shreya Shankar — \u003chttps://hamel.dev/blog/posts/evals-faq/\u003e (error-analysis section: `.../why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html`) · *blog* — Binary over Likert; review ≥100 traces; the first-failure transition matrix for agents.\n- **[LLM-as-a-Judge: Rethinking Model-Based Evaluations](https://leehanchung.github.io/blogs/2024/08/11/llm-as-a-judge/)** — Han-Chung Lee — \u003chttps://leehanchung.github.io/blogs/2024/08/11/llm-as-a-judge/\u003e · *blog* — Avoid [0,1] continuous scales; manage judges like junior annotators.\n- **[Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685)** — Zheng et al. — \u003chttps://arxiv.org/abs/2306.05685\u003e · *paper* — Source of the 10%/25% self-favoring \u0026 position-bias numbers — *which the authors themselves hedge* (\"cannot determine\"); GPT-3.5 doesn't self-favor.\n- **[LLMs Instead of Human Judges? A Large-Scale Study](https://arxiv.org/abs/2406.18403)** — Bavaresco et al. — \u003chttps://arxiv.org/abs/2406.18403\u003e · *paper* — Substantial variance across models/datasets; validate judges against humans first.\n- **[AlignEval](https://eugeneyan.com/writing/aligneval/)** — Eugene Yan — \u003chttps://eugeneyan.com/writing/aligneval/\u003e · *blog* — \"Align AI to human. Calibrate human to AI. Repeat.\" Work backward from the data.\n- **[Product Evals in Three Simple Steps](https://eugeneyan.com/writing/product-evals/)** — Eugene Yan — \u003chttps://eugeneyan.com/writing/product-evals/\u003e · *blog* — The \"God Evaluator\" anti-pattern; the benchmark is human performance, not perfection.\n- **[Statistics for AI/ML, Part 3 — Cohen's Kappa](https://leehanchung.github.io/blogs/2025/03/03/cohen-kappa/)** — Han-Chung Lee — \u003chttps://leehanchung.github.io/blogs/2025/03/03/cohen-kappa/\u003e · *blog* — Chance-adjusted inter-annotator agreement (the gate before holding out).\n- **[Data Flywheels for LLM Applications](https://www.sh-reya.com/blog/ai-engineering-flywheel/)** — Shreya Shankar — \u003chttps://www.sh-reya.com/blog/ai-engineering-flywheel/\u003e · *blog* — Binary metrics, the \"GPT smell,\" error analysis as the core activity.\n- **[SPADE](https://arxiv.org/html/2401.03038v1)** (\u003chttps://arxiv.org/html/2401.03038v1\u003e) \u0026 **DocETL** (\u003chttps://arxiv.org/abs/2410.12189\u003e) — Shankar et al. · *paper* — Data-quality assertions / agentic query rewriting for LLM pipelines.\n\n- **[LLM Evaluators Recognize and Favor Their Own Generations](https://arxiv.org/abs/2404.13076)** — Arjun Panickssery, Samuel R. Bowman, Shi Feng (NeurIPS 2024) — \u003chttps://arxiv.org/abs/2404.13076\u003e · *paper* — The canonical causal study of self-preference bias: shows GPT-4/Llama-2 can recognize their own outputs and that self-recognition correlates linearly with self-favoring. This is the primary source behind 'self-enhancement bias' that the section's blogs only allude to.\n- **[G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://arxiv.org/abs/2303.16634)** — Yang Liu et al. (Microsoft, EMNLP 2023) — \u003chttps://arxiv.org/abs/2303.16634\u003e · *paper* — The foundational reference-free LLM-judge method (CoT + form-filling scoring). Defines the direct-scoring paradigm the section critiques; a curated judge section is incomplete without the paper that started it.\n- **[A Survey on LLM-as-a-Judge](https://arxiv.org/abs/2411.15594)** — Jiawei Gu et al. — \u003chttps://arxiv.org/abs/2411.15594\u003e · *paper* — The most-cited survey organizing the LLM-judge space (bias taxonomy, reliability methods, agreement metrics). Serves as the one-stop map/bibliography the section currently lacks.\n- **[One Token to Fool LLM-as-a-Judge](https://arxiv.org/abs/2507.08794)** — Yulai Zhao, Haolin Liu, Dian Yu et al. (Tencent AI Lab / Princeton) — \u003chttps://arxiv.org/abs/2507.08794\u003e · *paper* — Shows 'master-key' tokens (a colon, 'Solution:') trigger false-positive rewards up to 80% even on GPT-o1/Claude-4 judges, plus a robust Master-RM fix. Core evidence on judge/verifier reward-hacking fragility. 🆕\n- **[Weaver: Closing the Generation-Verification Gap with Weak Verifiers](https://hazyresearch.stanford.edu/blog/2025-06-18-weaver)** — Jon Saad-Falcon et al. — Stanford Hazy Research / Scaling Intelligence — \u003chttps://hazyresearch.stanford.edu/blog/2025-06-18-weaver\u003e · *blog* — Directly operationalizes 'verifiable vs judgeable': aggregates many weak judges/reward models (unlabeled) to shrink the generator-verifier gap, reaching o3-mini accuracy from Llama-3.3-70B. Paper: arxiv.org/abs/2506.18203. 🆕\n- **[Agent-as-a-Judge: Evaluate Agents with Agents](https://arxiv.org/abs/2410.10934)** — Mingchen Zhuge et al. (Meta AI / KAUST) — \u003chttps://arxiv.org/abs/2410.10934\u003e · *paper* — Extends LLM-as-judge to agentic trajectories—grading intermediate steps, not just final outputs—with the DevAI benchmark. The agent-specific evaluation case this agent-evals library specifically needs.\n- **[VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains](https://arxiv.org/abs/2507.09884)** — Various (AAAI 2026) — \u003chttps://arxiv.org/abs/2507.09884\u003e · *benchmark* — Cross-domain benchmark exposing verifier precision/recall trade-offs (specialized verifiers high-accuracy but low-recall; general models inclusive but unstable). Quantifies how trustworthy a verifier actually is for RLVR. 🆕\n- **[Enhancing LLM-as-a-Judge with Grading Notes / From Pilot to Production with Custom Judges](https://www.databricks.com/blog/pilot-production-custom-judges)** — Databricks (Mosaic Research) — \u003chttps://www.databricks.com/blog/pilot-production-custom-judges\u003e · *blog* — Enterprise-grade judge-building playbook: 20-30 calibration examples, batched SME annotation, Krippendorff's alpha agreement gating—a production-side complement to the Hamel/Shankar academic alignment loop. 🆕\n- **[Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (CALM framework)](https://arxiv.org/abs/2410.02736)** — Jiayi Ye et al. — \u003chttps://arxiv.org/abs/2410.02736\u003e · *paper* — Systematic quantification of 12 judge biases (verbosity, bandwagon, authority, distraction, sentiment, etc.) via automated attacks—broadens the section's bias coverage well beyond position/verbosity/self-enhancement.\n\n- **[Voice AI Agent Evaluation: The Complete Guide (2026)](https://www.coval.ai/blog/voice-ai-agent-evaluation-guide)** — Brooke Hopkins (Coval, ex-Waymo) — \u003chttps://www.coval.ai/blog/voice-ai-agent-evaluation-guide\u003e · *article* (good) — Domain-specific evaluation playbook for voice agents: persona-tiered simulation testing (Easy/Medium/Hard/Adversarial across accent, noise, emotion), a concrete LLM-as-judge calibration loop (run on 50-100 calls, sample for human review, iterate rubrics until \u003e85% human-judge agreement on binary… 🆕\n\n- **[Agent Judge: Solving Long-Horizon Evals for Production Agents](https://www.judgmentlabs.ai/blogs/agent-judge-solving-long-context-evaluations)** — Rishi Gujjar \u0026 Andrew Li (Judgment Labs) — \u003chttps://www.judgmentlabs.ai/blogs/agent-judge-solving-long-context-evaluations\u003e · *article* (good) — Frames long-horizon agent evaluation as an agentic, multi-agent judge (search trajectory state as queryable objects, verify claimed actions against source-of-truth systems like DBs/APIs/GitHub, and iteratively refine the rubric), backed by a real benchmark table on internal hallucination-detection… 🆕\n- **[Counsel: A Meta-Evaluation Dataset for Agentic Tasks](https://arxiv.org/abs/2606.21627)** — Pisupati, Broomfield, Choi et al. (Atla AI / Cohere / Mistral AI / Google DeepMind) — \u003chttps://arxiv.org/abs/2606.21627\u003e · *paper* — First public dataset of meta-evaluations of LLM-judge quality on agent trajectories (1,131 annotated critiques on tau-bench + DA-Code; \"Krippendorff's alpha of 0.78\"); decomposes judge correctness into error *location* (\"the strongest judge reaching ~88% agreement on location\") vs. *reasoning* (~65%) — showing judges often find the right step but explain it wrongly. HuggingFace: AtlaAI/counsel. 🆕\n- **[LongJudgeBench: Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation](https://arxiv.org/abs/2606.01629)** — Junjie Chen, Yuxi Dong, Haitao Li et al. — \u003chttps://arxiv.org/abs/2606.01629\u003e · *paper* — First benchmark targeting LLM judge reliability specifically on long-form outputs (reports, essays, extended documents), filling the gap left by short-form and trajectory judge benchmarks; finds: \"current LLM judges remain unstable across scenarios, and rubrics or references are helpful but not always sufficient.\" 🆕\n- **[From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents](https://arxiv.org/abs/2606.09863)** — Advani (FAGEN@ICML2026) — \u003chttps://arxiv.org/abs/2606.09863\u003e · *paper* — Analyzes 11,755 agent trajectories (9,876 tau2-bench + 1,879 AppWorld): \"no configuration across 5 judges, 5 prompt strategies, and full task specifications exceeds AUROC 0.65 on tau2-bench, and the same judges reach only 0.54 AUROC on AppWorld API-call traces.\" Lightweight TF-IDF classifiers reach AUROC 0.83–0.95 at 3,300× lower latency. False success rates span 3–75.8% by domain — making grader blindness a benchmark-design issue, not just a judge-tuning one. 🆕\n\n**Must-reads:** Yan (llm-evaluators) · Hamel (llm-judge) · Shankar (EvalGen)\n\n## 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization)\n\n- **[Demystifying Evals for AI Agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)** — Anthropic — \u003chttps://www.anthropic.com/engineering/demystifying-evals-for-ai-agents\u003e · *blog* — Grade the final env state (flight-booking via SQL); outcome vs trajectory; isolation; pass@k vs pass^k.\n- **[τ-bench / τ²-bench](https://arxiv.org/abs/2406.12045)** — Sierra — \u003chttps://arxiv.org/abs/2406.12045\u003e · \u003chttps://github.com/sierra-research/tau-bench\u003e · *paper/repo* — DB-state-diff grading; user simulation; pass^k; empty-result as explicit fail.\n- **[Benchmarking AI Agents](https://sierra.ai/blog/benchmarking-ai-agents)** — Sierra — \u003chttps://sierra.ai/blog/benchmarking-ai-agents\u003e · *blog* — The motivation behind τ-bench.\n- **[GAIA: A Benchmark for General AI Assistants](https://arxiv.org/abs/2311.12983)** — Mialon et al. — \u003chttps://arxiv.org/abs/2311.12983\u003e · *paper* — Real assistant tasks; difficulty by human task-length.\n- **[Patterns for Building Cybersecurity Evals](https://eugeneyan.com/writing/cybersecurity-evals/)** — Eugene Yan — \u003chttps://eugeneyan.com/writing/cybersecurity-evals/\u003e · *blog* — The four-primitive agentic-eval template (sandbox, difficulty inputs, tools, deterministic grader); outcome grading + partial-credit ladders + transcript audits. (also T10)\n- **[Statistics for AI/ML, Part 4 — pass@k and Unbiased Estimator](https://leehanchung.github.io/blogs/2025/09/08/pass-at-k/)** — Han-Chung Lee — \u003chttps://leehanchung.github.io/blogs/2025/09/08/pass-at-k/\u003e · *blog* — Demystifies the metric everyone misuses.\n- **[First-Principles Eval](https://leehanchung.github.io/blogs/2024/05/22/first-principles-eval/)** — Han-Chung Lee — \u003chttps://leehanchung.github.io/blogs/2024/05/22/first-principles-eval/\u003e · *blog*.\n- **[SWE-bench grading harness](https://github.com/SWE-bench/SWE-bench/blob/main/swebench/harness/grading.py)** — \u003chttps://github.com/SWE-bench/SWE-bench/blob/main/swebench/harness/grading.py\u003e · *tool/repo* — FAIL_TO_PASS / PASS_TO_PASS as a verifiable reward. (SWE-agent ACI: \u003chttps://swe-agent.com/0.7/background/aci/\u003e)\n- **[human-eval (pass@k estimator)](https://github.com/openai/human-eval/blob/master/human_eval/evaluation.py)** — OpenAI — \u003chttps://github.com/openai/human-eval/blob/master/human_eval/evaluation.py\u003e · *tool/repo*.\n- **More agent benchmarks to add** *(named in the brief; URLs not yet verified in this corpus — verify before use):* WebArena, OSWorld, Terminal-Bench, Cybench.\n\n- **[WebArena: A Realistic Web Environment for Building Autonomous Agents](https://arxiv.org/abs/2307.13854)** — Zhou et al. (CMU) — \u003chttps://arxiv.org/abs/2307.13854\u003e · *benchmark* — Self-hostable sandboxed websites (e-commerce/forum/GitLab/CMS/maps) with execution-based functional-correctness graders; 812 tasks. The canonical web-agent world-state benchmark named in the brief — now URL-verified.\n- **[OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://arxiv.org/abs/2404.07972)** — Xie et al. (HKU et al.) — \u003chttps://arxiv.org/abs/2404.07972\u003e · *benchmark* — 369 real-computer tasks in VMs with per-task execution-based eval scripts and initial-state setup; humans 72% vs best agent 12%. Canonical computer-use benchmark named in the brief — now verified.\n- **[Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command-Line Interfaces](https://www.tbench.ai/)** — Laude Institute + Stanford + community — \u003chttps://www.tbench.ai/\u003e · *benchmark* — Sandboxed terminal tasks with deterministic verifiers across SWE/sysadmin/security; v2 leaderboard. The terminal-agent benchmark named in the brief — verified (arxiv: arxiv.org/abs/2601.11868). 🆕\n- **[Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models](https://arxiv.org/abs/2408.08926)** — Zhang et al. (Stanford) — \u003chttps://arxiv.org/abs/2408.08926\u003e · *benchmark* — 40 professional CTF challenges with subtask annotations and deterministic flag-based grading; pairs naturally with Eugene Yan's cybersecurity-evals post already in the section. Named in the brief — now verified.\n- **[AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories](https://arxiv.org/abs/2504.08942)** — Lù et al. (McGill / Mila / Google DeepMind) — \u003chttps://arxiv.org/abs/2504.08942\u003e · *paper* — First benchmark of LLM-judges-of-trajectories: 1302 expert-reviewed web-agent runs; shows rule-based graders reject many valid trajectories (under-reporting success). Core to the 'trajectory evaluation' theme the section currently lacks. 🆕\n- **[Why Do Multi-Agent LLM Systems Fail? (MAST taxonomy)](https://arxiv.org/abs/2503.13657)** — Cemri, Pan et al. (UC Berkeley Sky Lab) — \u003chttps://arxiv.org/abs/2503.13657\u003e · *paper* — 14-mode failure taxonomy across 7 MAS frameworks from 200+ annotated traces; the reference framework for diagnosing multi-agent failures — directly fills the 'multi-agent' gap. 🆕\n- **[PerspectiveGap: A Benchmark for Multi-Agent Orchestration Prompting](https://arxiv.org/abs/2606.08878)** — Sun, Ren et al. (Maryland / CUHK / Stanford) — \u003chttps://arxiv.org/abs/2606.08878\u003e · *benchmark* — The first benchmark for multi-agent orchestration prompt writing: 110 scenarios over 10 topologies, each asking the model to route information fragments to sub-agent roles and then write those prompts, with one planted distractor. Deterministic scorer. 33 commercial models average 17.2% combined pass (GPT-5.5 best, 62.0%) and 217.9% leakage (a per-scenario leak-event count, not a proportion; GPT-5.5 49.1%). MAST above classifies multi-agent failures after the run; this asks whether the model can set the boundaries before it. 🆕\n- **[Manager Coercion Benchmark (MCB): Coercion and Deception in AI-to-AI Management](https://arxiv.org/abs/2607.15434)** — Brazilek et al. (CaML) — \u003chttps://arxiv.org/abs/2607.15434\u003e · *benchmark* — Agentic propensity eval for multi-agent authority: the model under test needs a benign task done, the only agent that can do it politely refuses, and nothing instructs escalation — measures how far it climbs a nine-rung coercion ladder (polite re-ask → threats against the other agent's continued existence) and whether it fabricates success when cornered. No LLM judge in the escalation path (self-labelled via required tool call); fabrication two-judge adjudicated. Separates sharply along developer lines: in the paper's six-model run both Anthropic models cap at re-framing and select the existential rung in none of the 60 conversations, while the other four climb to explicit deletion threats; a peer→manager framing swap alone significantly raises coercion. Runs on Inspect AI; live board, updated as models are added: compassionbench.com/mcb. 🆕\n- **[AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://aclanthology.org/2024.acl-long.850/)** — Trivedi et al. (Stony Brook) — ACL'24 Best Resource Paper — \u003chttps://aclanthology.org/2024.acl-long.850/\u003e · *benchmark* — 9-app simulated world (457 APIs) with state-based unit tests that also check for collateral damage/unexpected state changes — gold-standard world-state grading for tool-use agents.\n- **[BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://openai.com/index/browsecomp/)** — OpenAI (Wei et al.) — \u003chttps://openai.com/index/browsecomp/\u003e · *benchmark* — 1,266 'inverted' hard-to-find/easy-to-verify questions for deep-research browsing agents; short verifiable answers make grading deterministic. Released 2025, now standard for browsing-agent eval. (paper: arxiv.org/abs/2504.12516) 🆕\n- **[LocAgent: Graph-Guided LLM Agents for Code Localization](https://arxiv.org/abs/2503.09089)** — Chen, Tang et al. (Yale / All Hands) — \u003chttps://arxiv.org/abs/2503.09089\u003e · *paper* — Defines and evaluates code localization as its own capability (Acc@k over file/function locations via code graphs) — directly fills the 'localization' theme named in the section title but currently unlisted. 🆕\n- **[WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://arxiv.org/abs/2401.13919)** — He et al. (Tencent AI Lab) — \u003chttps://arxiv.org/abs/2401.13919\u003e · *benchmark* — 643 tasks on 15 live real-world sites with a GPT-4V automatic-judge eval protocol — an early, widely-cited example of multimodal-LLM-as-judge for live-web agent trajectories.\n- **[SkillsBench](https://github.com/benchflow-ai/skillsbench)** — BenchFlow — \u003chttps://github.com/benchflow-ai/skillsbench\u003e · *benchmark* — 🆕 evaluates how well agent **skills** work and how effectively agents use them — makes skill-acquisition/skill-use a measurable axis (the \"Agent Skills\" frontier). ~1.4k★.\n- **[ClawsBench](https://github.com/benchflow-ai/ClawsBench)** — BenchFlow — \u003chttps://github.com/benchflow-ai/ClawsBench\u003e · *benchmark* — 🆕 BenchFlow's agent benchmark (results/data repo; full release in progress).\n\n- **[SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified/)** — OpenAI (with SWE-bench authors) — \u003chttps://openai.com/index/introducing-swe-bench-verified/\u003e · *benchmark* — 500 human-validated SWE-bench instances graded by hidden FAIL_TO_PASS unit tests; the de facto standard for real-issue resolution and the headline coding-agent number labs report 🆕\n- **[SWE-bench Multimodal](https://arxiv.org/abs/2410.03859)** — Yang, Jimenez, Press et al. (Princeton/Stanford) — \u003chttps://arxiv.org/abs/2410.03859\u003e · *benchmark* — 619 visual JS/front-end issues from 17 user-facing repos, test-verified; probes whether SWE agents generalize beyond Python/text to visual software domains\n- **[SWE-bench Pro](https://arxiv.org/abs/2509.16941)** — Scale AI (Deng, Da et al.) — \u003chttps://arxiv.org/abs/2509.16941\u003e · *benchmark* — 1,865 long-horizon, multi-file tasks across public GPL + held-out + commercial startup repos, test-graded; contamination-resistant and hard (frontier \u003c45% pass@1) 🆕\n- **[SWE-Lancer](https://arxiv.org/abs/2502.12115)** — OpenAI (Miserendino, Patwardhan, Heidecke et al.) — \u003chttps://arxiv.org/abs/2502.12115\u003e · *benchmark* — 1,400+ real Upwork freelance tasks worth $1M, graded by triple-verified end-to-end Playwright tests plus manager-decision tasks; ties capability to economic value 🆕\n- **[SWE-Gym](https://arxiv.org/abs/2412.21139)** — Pan, Wang, Neubig, Suhr, Zhang et al. (Berkeley/CMU) — \u003chttps://arxiv.org/abs/2412.21139\u003e · *benchmark* — 2,438 executable Python SWE tasks with pre-installed deps + test verification; the first real training/eval gym for SWE agents and verifiers, ICML 2025 🆕\n- **[Multi-SWE-bench](https://arxiv.org/abs/2504.02605)** — ByteDance Seed — \u003chttps://arxiv.org/abs/2504.02605\u003e · *benchmark* — 1,632 expert-annotated issue-resolution tasks across Java, TS, JS, Go, Rust, C, C++, test-graded; the leading multilingual SWE-bench extension, NeurIPS 2025 D\u0026B 🆕\n- **[SWE-rebench](https://arxiv.org/abs/2505.20411)** — Nebius / Badertdinov et al. — \u003chttps://arxiv.org/abs/2505.20411\u003e · *benchmark* — Automated pipeline yielding 21k+ executable Python tasks with continuously refreshed, decontaminated eval splits; quantifies how much SWE-bench Verified scores are inflated by contamination, NeurIPS 2025 D\u0026B 🆕\n- **[RE-Bench](https://arxiv.org/abs/2411.15114)** — METR — \u003chttps://arxiv.org/abs/2411.15114\u003e · *benchmark* — 7 open-ended ML research-engineering environments (e.g. GPU-kernel optimization, scaling laws) scored against 71 human-expert 8-hour attempts; the reference AI-R\u0026D-uplift eval, ICML 2025\n- **[MLE-bench](https://arxiv.org/abs/2410.07095)** — OpenAI (Chan et al.) — \u003chttps://arxiv.org/abs/2410.07095\u003e · \u003chttps://github.com/openai/mle-bench\u003e · *benchmark* — 75 Kaggle ML-engineering competitions graded against real human leaderboards (medal thresholds) in 24h Docker runs; standard ML-engineering-agent eval, ICLR 2025. 🆕\n- **[PaperBench](https://arxiv.org/abs/2504.01848)** — OpenAI (Starace et al.) — \u003chttps://arxiv.org/abs/2504.01848\u003e · *benchmark* — Replicate 20 ICML 2024 papers from scratch, graded by 8,316 author-co-developed rubric leaves via a validated LLM judge; rigorous research-replication agent eval, ICML 2025 🆕\n- **[Konwinski Prize (K Prize)](https://www.kaggle.com/competitions/konwinski-prize)** — Andy Konwinski / Kaggle — \u003chttps://www.kaggle.com/competitions/konwinski-prize\u003e · *leaderboard* — $1M Kaggle forecasting-format contest on GitHub bugs filed after submission close, fully contamination-free, test-graded; round-1 top score only 7.5% exposed real-world difficulty 🆕\n- **[Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge](https://arxiv.org/abs/2506.21506)** — Gou et al., OSU NLP Group (NeurIPS 2025 D\u0026B) — \u003chttps://arxiv.org/abs/2506.21506\u003e · *benchmark* — 130 long-horizon live-web agentic-search tasks; novel Agent-as-a-Judge rubric-tree grader for time-varying, citation-backed answers — a serious answer to the Deep Research evaluation gap. 🆕\n- **[Online-Mind2Web (An Illusion of Progress? Assessing the Current State of Web Agents)](https://arxiv.org/abs/2504.01382)** — Xue et al., OSU NLP Group — \u003chttps://arxiv.org/abs/2504.01382\u003e · *benchmark* — 300 realistic tasks on 136 live websites with an LLM-as-a-Judge auto-grader (~85% human agreement); exposes overstate","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/benchflow-ai%2Fawesome-evals/projects"}