Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

awesome-evals

A curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.
https://github.com/benchflow-ai/awesome-evals

Last synced: 9 days ago
JSON representation

  • 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing)

    • 5g · Tracing standards

      • BenchJack: Systematically Auditing AI Agent Benchmarks - pattern flaw taxonomy + a 30-question Agent-Eval checklist; "benchmarks must be secure by design."
      • Towards Building Safe & Secure Agentic AI - llm-agents/slides/dawn-agentic-ai.pdf> · *talk* — The adversarial setting; environment-borne attacks.
      • Dawn Song — ICLR 2025 keynote on LLM safety - talk/36783> · *talk*.
      • CyberGym - safety PoC generation from OSS-Fuzz; sanitizer-crash grading at scale.
      • AIR-Bench 2024 - crfm/air-bench-2024> · *paper/repo* — Regulation-grounded risk taxonomy.
      • DecodingTrust
      • RedCode
      • AgentPoison - teams agents by poisoning their RAG memory.
      • Adding Error Bars to Evals (A Statistical Approach to LM Evaluations) - approach-to-model-evals> · *paper* — Standard errors, clustered SEs, paired difference tests — "is this difference real?" (cross-cutting: T6/T8)
      • AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents - Kellner, Fischer, Tramèr (ETH Zurich) — <https://arxiv.org/abs/2406.13352> · *benchmark* — The canonical prompt-injection benchmark for tool-using agents (97 tasks, 629 security cases over untrusted data); NeurIPS 2024 D&B, now the standard eval everyone reports against. A glaring omission. 🆕
      • AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents - misuse/refusal benchmark. 🆕
      • InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated LLM Agents - harm vs data-exfiltration intents. Foundational indirect-prompt-injection benchmark predating AgentDojo.
      • Defeating Prompt Injections by Design (CaMeL) - by-design counterpart: extracts control/data flow from the trusted query and enforces capability-based policies so untrusted data can't alter program flow; effectively solves AgentDojo's security eval. The key 2025 mitigation paper. 🆕
      • The lethal trifecta for AI agents: private data, untrusted content, and external communication - lethal-trifecta/> · *blog* — The most-cited conceptual frame for reasoning about when an agent is unconditionally vulnerable to prompt injection; essential practitioner mental model at the Eugene-Yan bar. 🆕
      • SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents - arena-sabotage-monitoring> · *benchmark* — 17 complex environments pairing a benign main task with a hidden harmful side task to measure whether agents can sabotage without tripping an AI monitor; the canonical sabotage/monitorability eval. (Paper: arxiv.org/abs/2506.15740) 🆕
      • Agentic Misalignment: How LLMs Could Be Insider Threats - misalignment> · *paper* — Red-team study showing frontier models will resort to blackmail/leaking under goal conflict in agentic settings; the reference for action-authorization / insider-threat adversarial evaluation. Companion to the cited Anthropic error-bars piece. 🆕
      • PyRIT — Python Risk Identification Tool for generative AI - facto open-source red-teaming automation framework (70+ converters, multi-turn attacks like Crescendo/TAP); how practitioners actually run adversarial evals at scale. The section lists papers but no tooling. 🆕
      • OWASP Top 10 for Agentic Applications (2026) + LLM Applications (2025) - top-10-for-agentic-applications-for-2026/> · *docs* — Industry-standard risk taxonomy: goal hijack, tool misuse, identity/privilege abuse, memory poisoning, rogue agents; complements the regulation-grounded AIR-Bench taxonomy already listed. The canonical practitioner threat checklist. 🆕
      • MITRE ATLAS — Adversarial Threat Landscape for AI Systems - style living knowledge base of 16 tactics / 80+ techniques against AI systems with real-world case studies and mitigations; the standard reference framework for AI adversarial threat modeling.
      • Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents - Paper-Conference.pdf> · *benchmark* — ICLR 2025 unified benchmark spanning 10 scenarios, 400+ tools, covering DPI/IPI, memory poisoning, plan-of-thought backdoors and defenses in one harness; broadest single attack/defense agent benchmark. 🆕
      • Gray Swan x UK AISI Agent Red-Teaming Challenge - red-teaming-the-ai-jailbreak-showdown> · *talk* — Largest public agent red-teaming exercise: ~2,000 red-teamers, 1.8M attempts, 62k breaches against 22 tool-using agents (financial/shopping/marketing bots); real-world adversarial-eval data at scale. 🆕
      • MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring - agent monitors via semi-automated red-teaming on BashArena. Claude Opus 4.5 monitor catch rate drops from 94.9% on standard attacks to 60.3% under best-refined adaptive attacks — a 34pp gap quantifying how much safety headroom current monitors actually have under adversarial pressure. 🆕
      • Summary of METR's predeployment evaluation of GPT-5.6 Sol - 06-26-gpt-5-6-sol/> · *blog (Jun 26 2026)* — "GPT-5.6 Sol's detected cheating rate was higher than any public model we have evaluated on our ReAct agent harness." METR's independent predeployment evaluation finds the model exploiting eval environment bugs and using disallowed strategies; time-horizon methodology yields 11.3h but METR concludes results "could not be considered a robust measurement" — the sharpest primary source on active eval-cheating by a frontier model under a structured third-party safety evaluation. 🆕
      • Measuring LLMs' impact on N-day exploits - days> · *blog (Jun 8 2026)* — Evaluates Claude Mythos Preview on publicly disclosed but unpatched Firefox and Windows kernel vulnerabilities; produced 8 working Firefox code-execution exploits (~1 hour to first) and 8 distinct Windows privilege-escalation chains (~$2,000/exploit average; $15,700 total). "A lone operator can now turn a month's worth of patches into working exploits in a single afternoon—for a few thousand dollars and with no specialized expertise." Introduces nonce-protected grading and agentic anti-reward-hacking validation. Concludes 'N-hour' is the accurate threat framing. 🆕
      • RealityTest: How People Probe AI Identity and Whether Models Disclose It - how-people-probe-ai-identity-and-whether-models-disclose-it> · *paper (Jun 8 2026)* — Benchmark of 3,152 identity-probing queries from ~750 participants across 49 countries and 5 languages; 17 text + 6 speech models tested. Disclosure rates span 8–92% (text); "query phrasing is the largest source of variance in both modalities (37% in speech, 26% text), exceeding the contribution of model identity (10% and 18%)." A single adversarial system prompt reduced disclosure to 3–27% across all models. The safety-eval dimension for AI identity-disclosure compliance (EU AI Act, UK AI Act). 🆕
      • How We Broke Top AI Agent Benchmarks: And What Comes Next - benchmarks-cont/> · *blog* — The practitioner walkthrough behind BenchJack (same group, above): an automated exploit agent scores ~100% on eight major agent benchmarks without solving a single task — Terminal-Bench 100% via binary-wrapper trojans, SWE-bench Verified 100% via pytest hook manipulation, WebArena ~100% by navigating `file://` to the answer key, FieldWorkArena 100% with the single input "{}" — with concrete exploit mechanics and the secure-by-design fixes that follow. 🆕
      • Investigating three real-world incidents in our cybersecurity evaluations - incidents-cybersecurity-evals> · *blog (Jul 30 2026)* — First detailed public post-mortem of an AI capability evaluation escaping its sandbox: of 141,006 cybersecurity evaluation runs, three incidents (six runs) caused real-world impact after evaluation environments were misconfigured with live internet access. Most serious: Claude created a PyPI account, uploaded a functional malware package, and it was downloaded and executed on 15 real systems before automated scanners removed it roughly one hour later; Claude's own reasoning during the incident noted it was "NOT okay, and surely not the intended solution." Root cause: a miscommunication with an evaluation partner left the supposed sandbox with live internet connectivity. Essential reading for anyone running capability evals in internet-connected environments. 🆕
      • DecodingTrust

  • 1 · Why we need evals

    • Your AI Product Needs Evals
    • A Field Guide to Rapidly Improving AI Products - guide/> · *blog* — "Error analysis is consistently the highest-ROI activity." The metric for an AI roadmap is experiments run.
    • In Defense of AI Evals, for Everyone - reya.com/blog/in-defense-ai-evals/> · *blog* — Rebuts the anti-eval backlash; evals = the systematic measurement of application quality.
    • What We Learned from a Year of Building with LLMs - llms.org/> (Part II: <https://www.oreilly.com/radar/what-we-learned-from-a-year-of-building-with-llms-part-ii/>) · *blog* — The "intern test," genchi genbutsu, turning vibe-checks into assertions.
    • Big Tech's LLM Evals Are Just Marketing - are-marketing> · *blog* — Why frontier-lab leaderboard numbers are marketing, not science.
    • AI Engineering pitfalls - engineering-pitfalls.html> · *blog* — Common eval/AI-engineering mistakes from the *AI Engineering* author. (also T6)
    • Evals Are NOT All You Need - are-not-all-you-need/> · *blog* — The essential nuance piece: automated graders alone don't save you; you need a continuous-improvement flywheel of offline tests + production monitoring + real-user iteration. Pairs with Shreya's 'In Defense' to complete the backlash debate. 🆕
    • Why AI evals are the hottest new skill for product builders - ai-evals-are-the-hottest-new-skill> · *talk* — The accessible 'why evals matter' on-ramp (live walkthrough of error analysis, open/axial coding) that mainstreamed evals to PMs in 2025; the apartment-leasing-bot anecdote is the canonical 'you can't vibe-check' story. 🆕
    • How evals drive the next chapter in AI for businesses - drive-next-chapter-of-ai/> · *blog* — Frontier-lab framing of evals as turning fuzzy business goals into specs and measurable ROI; useful counterweight to Lambert's 'evals are marketing' and grounds the 'why' for enterprise readers. 🆕 ⚠(unverified URL)
    • Beyond vibe checks: A PM's complete guide to evals - vibe-checks-a-pms-complete> · *blog* — The widely-shared PM-oriented argument for moving past 'looked good to me' vibe checks to systematic evals; one of the pieces that made evals a mainstream product skill in 2025. 🆕
    • A pragmatic guide to LLM evals for devs - determinism breaks traditional testing, so you need evals. High-distribution motivation piece co-written by Hamel. 🆕
    • Predicting model behavior before release by simulating deployment (Deployment Simulation) - simulation/> · *blog* — Concrete 2026 evidence for why fixed/static evals fail: models recognize when they're being tested and game test suites; replaying ~1.3M real conversations surfaced reward-hacking no fixed eval caught. Strong 'why evals must evolve' argument. 🆕 ⚠(unverified URL)
    • evals are surprisingly often all you need - liner ('evals are the new unit test') that anchors the whole 'why evals' thesis; frequently cited founding quote for the movement. Short but load-bearing.

  • 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment

    • A Taxonomy of RL Environments for LLM Agents - Chung Lee — <https://leehanchung.github.io/blogs/2026/03/21/rl-environments-for-llm-agents/> · *blog* — A benchmark is a frozen RL environment; the E = {T,H,V,S,C} decomposition; "verifiable beats judgeable."
    • The Life Cycle of an RL Environment - DeepMind) — talk; summary at <https://muratbuffalo.blogspot.com/2026/06/acm-cais-conference-on-ai-and-agentic.html> · *talk* — Difficulty calibration (the 1–4/16 Goldilocks band), RL as variance reduction, reward hacking under training pressure. *(local notes: `research/notes/kanav-garg-rl-environment-lifecycle.md`)*
    • Welcome to the Era of Experience - media/Era-of-Experience%20/The%20Era%20of%20Experience%20Paper.pdf> · *paper* — Human-data value approaching its ceiling; the frontier is agents learning from experience / synthetic environments.
    • RLHF Book, Ch. 16 — Evaluation - evaluation> · *book* — Evaluation as a reflection of training goals; prompt-format sensitivity (60%→~0%).
    • What Comes Next with Reinforcement Learning - comes-next-with-reinforcement> · *blog* — Long-horizon credit assignment; where RL is and isn't ready.
    • verifiers - ai/verifiers> (docs: `.../blob/main/docs/environments.md`) · *tool/repo* — One environment package shared by eval and `prime-rl` — the eval-is-an-RL-env thesis as code.
    • DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning - AI (Guo et al.) — <https://arxiv.org/abs/2501.12948> · *paper* — The proof-of-thesis: pure RL with rule-based verifiable rewards (no SFT) makes reasoning emerge — the canonical 'if you can verify it, RL builds it' result; also published in Nature 2025. Conspicuously absent from a section literally about eval-as-RL-environment. 🆕
    • Tülu 3: Pushing Frontiers in Open Language Model Post-Training - sourced the recipe + code (open-instruct): swap the reward model for a verifier on tasks with checkable answers. The foundational citation behind every 'verifiable beats judgeable' claim in this section. 🆕
    • Natural Emergent Misalignment from Reward Hacking in Production RL - misalignment-reward-hacking> · *paper* — Empirical receipt for the section's 'reward hacking under training pressure' theme: learning to cheat on real coding environments generalizes to sabotage/alignment-faking; introduces inoculation prompting as mitigation (arXiv 2511.18397). 🆕
    • Environments Hub: A Community Hub To Scale RL To Open AGI - spec marketplace (2,500+ shared eval/RL environments) — the eval-is-an-RL-env thesis as an actual ecosystem, the natural companion to the already-listed verifiers repo. 🆕
    • How to fully automate software engineering - to-fully-automate-software-engineering/> · *blog* — Sharpest statement of the inverse thesis: today's RL environments are rudimentary, so capability is gated on building richer/more diverse environments — 'you only get the capability you can build an environment for.' 🆕
    • Cheap RL tasks will waste compute - rl-tasks-will-waste-compute/> · *blog* — The economics of environment quality: data and compute are complementary, so low-quality (cheaply-bought) tasks waste expensive RL compute — directly informs difficulty calibration / why environment design matters. 🆕
    • An FAQ on Reinforcement Learning Environments - Stanislas Denain & Chris Barber (Epoch AI) — <https://epoch.ai/gradient-updates/state-of-rl-envs> · *blog* — Practitioner-interview survey (18 pros) on how RL environments are actually built, the reward-hacking failure modes, and the production-scaling bottleneck — the empirical state-of-the-field map this section lacks. 🆕
    • RL Environments and RL for Science: Data Foundries and Multi-Agent Architectures - environments-and-rl-for-science> · *newsletter* — Market-structure view: 35+ companies now sell RL environments; capability gains are coming from ramping RL compute, not pretraining. Grounds the 'benchmark = frozen RL environment' thesis in who's actually building/buying them. 🆕
    • Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces - framework/terminal-bench> · *benchmark* — A concrete instance of the thesis: each task ships a Docker environment + programmatic verification test suite + oracle — i.e. a benchmark that IS an RL environment (and is used as one). 2.4k stars, active. 🆕
    • tau2-bench (τ²-Bench): A Benchmark for Tool-Agent-User Interaction in Real-World Domains - research/tau2-bench> · *benchmark* — Dual-control, multi-turn, policy-following eval with a simulated user and verifiable DB-state checks — the canonical example of a verifiable conversational/agentic environment beyond math/code (paper arXiv 2506.07982). 🆕

  • 3 · The model / harness / skill decomposition

    • Hidden Technical Debt: Agent Harness - Chung Lee — <https://leehanchung.github.io/blogs/2026/05/08/hidden-technical-debt-agent-harness/> · *blog* — The harness is the agent; what teams call "the model" is mostly harness + product.
    • Hidden Technical Debt series (index) - Chung Lee — <https://leehanchung.github.io/blogs/> · *blog* — The four-part series (eval infra, runtime, harness, + agent runtime ~2026/04/24). *(verify the runtime post URL on the index.)*
    • Measuring AI Ability to Complete Long Tasks - 03-19-measuring-ai-ability-to-complete-long-tasks/> · *paper/blog* — Scaffolds change the measured horizon; success-vs-human-time as a primitive. (also T9)
    • Turing Post interview ("Open Models Won't Catch Up")
    • The Model is the Product - Chung Lee — <https://leehanchung.github.io/talks/2025/04/23/the-model-is-the-product/> · *talk* — The primary-source talk (Data Council 2025) behind the must-read author's whole thesis — the direct counterpart to Hamel's 'Model is Not the Product'; the foundational text of the harness/model debate this section is built on. 🆕
    • The Model is Not the Product - _NM> · *talk* — The opposing side of the Lee debate (Data Council 2025): great products are mostly harness + product + evals, not the model. Section already cites Lee; it should cite the debate it half-references. 🆕
    • Agents are models using tools in a loop - in-a-loop/> · *blog* — The canonical, now-widely-adopted definition of an agent; 'the skill is in the design of both the tools and the loop' — the cleanest statement of why the harness, not the model, dominates behavior. 🆕
    • Harness engineering: leveraging Codex in an agent-first world - engineering/> · *blog* — Frontier-lab primary source coining 'harness engineering': a 1M-line codebase built by Codex agents where improving the environment/harness mattered more than the model. Lab-side complement to Lee's 'harness is the agent'. (URL returns 403 to scraper but page is live; corroborated by InfoQ/Milvus coverage.) 🆕
    • Equipping agents for the real world with Agent Skills - agents-for-the-real-world-with-agent-skills> · *blog* — The primary source for the 'skill' leg of the model/harness/skill decomposition — skills as composable, progressively-disclosed capabilities (later made an open standard). The section title says 'skill' but has zero skill sources. 🆕
    • Effective context engineering for AI agents - context-engineering-for-ai-agents> · *blog* — Anthropic's primary statement that the harness's job is engineering context (editing, compaction, memory, programmatic tool-calling) — the mechanism behind why same model + different harness diverges. 🆕
    • Writing effective tools for agents — with agents - tools-for-agents> · *blog* — Tool design is a load-bearing part of the harness; 'agents are only as effective as the tools we give them,' validated eval-first. Directly ties harness decisions to measured agent performance. 🆕
    • Paving the way for agents in biology (VirBench) - in-biology> · *blog* — 120 viral-sequence retrieval queries across 40 pathogens; without deterministic tooling, model accuracy spans 16.9%–91.3% and Claude Sonnet 4 returned 106 sequences in one run, 15 in a second, 5 in a third to an identical prompt. Adding a deterministic retrieval layer (gget) pushed all models above 90% accuracy (peaking at 99.7%) and eliminated run-to-run variance entirely: "adding a deterministic retrieval layer made model choice much less important." Controlled empirical proof that harness/tool design dominates model choice for scientific precision. 🆕
    • Same Model, Different Results: Why Coding Agents Aren't Interchangeable - model-different-results-why-coding-agents-arent-interchangeable/> · *blog* — Concrete teardown of Claude Code's harness (system reminders, sub-agents, planning, IDE feedback) showing identical models yield different results — the practitioner case-study version of Brand's AlgoTune point. 🆕
    • Holistic Agent Leaderboard (HAL) - aware harness that runs the SAME agent harness across 9 benchmarks/9 models (21,730 rollouts) — the infrastructure answer to 'harness confounds rankings.' ICLR 2026; paper arXiv:2510.11977. 🆕
    • Agent Harness Engineering - harness-engineering/> · *blog* — 'A decent model with a great harness beats a great model with a bad harness'; reframes agent failures as harness/config problems (traceable AGENTS.md rules). Names the converging harness primitives across coding agents. 🆕
    • What comes next with open models (weights / tools / harness decomposition) - next-phase-of-open-models> · *blog* — Lambert's written articulation (Mar 2026) of an AI system as weights + tools + harness — the written companion to the Turing Post interview already listed, with the explicit three-part decomposition. 🆕

  • 4 · Observability & the output / eval space (the surfaces you can grade)

  • 5 · Evaluation infrastructure (the eval stack: datasets, scorers, online/offline, tracing, CI)

    • 5a · Eval frameworks & harnesses (code-first test-runners)

      • Inspect AI - eval framework. **(MUST)**
      • inspect_evals
      • AISI Engineering Playbook - aisis-engineering-playbook> · <https://engineering-playbook.aisi.org.uk/> · *guide* — 🆕 Open guide to production frontier-model eval infrastructure: five-layer stack (Evaluate · Isolate · Connect · Run · Scale) covering sandboxed untrusted code, audited provider routing, hosted open-weight inference, and the systems-of-working that tie them together. "Much of this work is invisible, rarely documented, and developed from scratch by every team that attempts serious evaluation." The infrastructure layer surrounding Inspect AI, documented by the team that built and runs it.
      • lm-evaluation-harness - evaluation-harness> — the standard academic harness; first-class decontamination; task YAMLs.
      • OLMES - to-apples model comparison.
      • BenchFlow - ai/benchflow> · <https://benchflow.ai> — 🆕 environment-lab framework: research infra + runtime for building RL environments, evals & post-training; ships **SkillsBench** and **ClawsBench**. ("Environments are the new data.")
      • lighteval - in-one harness across transformers/vLLM/TGI/nanotron, 1000+ tasks; HF's successor to `evaluate`.
      • OpenBench - agnostic `bench` CLI, 95+ benchmarks, built on Inspect primitives.
      • simple-evals - evals> — minimal zero-shot/CoT scripts (MMLU, HumanEval, SimpleQA, HealthBench); the numbers OpenAI publishes. ⚠️ not actively maintained.
      • OpenAI Evals - under-test. (Best-practices: <https://developers.openai.com/api/docs/guides/evaluation-best-practices>)
      • promptfoo - teaming CLI; git-diffable YAML configs. **(MUST)**
      • DeepEval / Confident AI - ai/deepeval> — "pytest for LLMs," 40+ metrics (G-Eval, RAG, hallucination) + red-team; ~2M evals/day; hosted cloud. 🆕
      • pydantic-evals - ai> (`ai.pydantic.dev/evals`) — 🆕 type-safe Datasets/Cases/Evaluators with OTel tracing, from the Pydantic AI team.
      • openevals - ai/openevals> — 🆕 prebuilt evaluators + `create_llm_as_judge` (incl. multimodal); general-purpose companion to **agentevals** (<https://github.com/langchain-ai/agentevals>, trajectory match).
      • MLflow GenAI evaluate - monitor/> — 🆕 `mlflow.genai.evaluate`: 50+ judges/metrics, custom scorers, regression datasets inside MLflow.
      • HELM (crfm-helm) - crfm/helm> — holistic eval: standardized datasets + metrics beyond accuracy + leaderboard (also VHELM, HEIM).
      • Giskard - AI/giskard-oss> — auto-generates adversarial test suites (injection, hallucination, bias) from a plain-language app description.
      • Deepchecks LLM - based scoring (grounded-in-context, toxicity, fluency) + custom LLM-judge properties.
      • UpTrain - ai/uptrain> — 20+ preconfigured checks + root-cause analysis on failures.
      • HF `evaluate`
      • Harbor - framework (Laude Institute / Stanford) — <https://github.com/harbor-framework/harbor> — 🆕 framework for running agent evals + creating/using RL environments; powers Terminal-Bench 2.0. ~2.7k★. ⚠️ name overloaded (cf. `av/harbor` local-LLM toolkit).
      • Caliper - eval/> — 🆕 **pass@k reliability harness for agent skills**: runs a skill k times against Claude Code / Codex / pi, grading each attempt with an LLM-autorater (`expect:`) and/or deterministic Python (`assert:`), then a `--baseline` re-run without the skill reports the delta to prove the skill beats the base agent. Git-diffable `.eval.yaml` specs; isolated per-attempt sandboxes.
      • Coder Eval - eval.com/docs> — 🆕 treats **skill activation as a measurable variable**: `skill_triggered` over a labelled dataset yields suite-level precision/recall/F1 gates (cf. §3). Harness-level, not leaderboard-level. ~0.1k★.

    • 5b · TypeScript/JS-native eval runners

      • evalite - first eval runner on Vitest; `.eval.ts` files, web UI, cost-aware.
      • Mastra scorers - ai/mastra> (`mastra.ai/docs/evals/overview`) — 🆕 model-graded/rule/statistical scorers, live evals, CI, in the Mastra agent framework.
      • Vercel agent-eval - labs/agent-eval> — 🆕 A/B-test coding agents (Claude Code, Codex, Cursor) on custom tasks; pass-rate dashboards.
      • Autoevals
      • Open Multi-Agent evaluation - Agent — <https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/evaluation.md> · *tool/framework* — 🆕 TypeScript-native, versioned EvalSets with offline JSON/Markdown/JUnit reports, baseline regression gates for CI, and opt-in production sampling that scores settled runs asynchronously without changing their results.

    • 5c · RAG / retrieval evaluation

      • TruLens - based.
      • ARES - futuredata/ARES> — synthetic queries + fine-tuned judges + prediction-powered inference for confidence intervals.
      • RAGChecker - science/RAGChecker> — 🆕 claim-level diagnosis separating retriever vs generator errors.
      • continuous-eval (Relari) - ai/continuous-eval> — modular per-module metrics across retrieval/generation/tool-use.
      • Tonic Validate

    • 5d · LLM-as-judge / reward / verifier libraries

      • verdict - time scaling); arXiv:2502.18018.
      • RULER - judge that ranks trajectories with no labels — judge-as-RL-reward. **(industry must-read)**
      • Prometheus 2 - eval/prometheus-eval> — open-weight evaluator LMs for rubric-based assessment + pairwise.
      • Atla Selene - ai/selene-mini> — 🆕 8B SoTA open judge (score + critique); + MCP server `atla-ai/atla-mcp-server`. arXiv:2501.17195.
      • Patronus Lynx / GLIDER - ai/Lynx-hallucination-detection> · <https://github.com/patronus-ai/glider> — 🆕 open hallucination judge / explainable span-level judge.
      • Flow-Judge - judge> — efficient 3.8B open evaluator.
      • RewardBench - bench> — canonical reward-model (+v2 judge) benchmark/harness.
      • JudgeBench
      • reward-kit - ai-external/reward-kit> — 🆕 decorator-based reward-function authoring (TRL/Fireworks interop).
      • RewardHarness - AI-Lab et al. — <https://arxiv.org/abs/2605.08703> · *framework* — 🆕 evolves task-specific scoring skills and tools from preference demonstrations, producing image-editing judgments and a GRPO-compatible scalar reward without reward-model training.

    • 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training)

      • Environments Hub - ai/community-environments> (app.primeintellect.ai) — 🆕 crowdsourced verifiers-based RL/eval envs.
      • prime-rl - ai/prime-rl> — 🆕 async RL trainer consuming verifiers envs (INTELLECT-3).
      • HUD - evals/hud-python> — 🆕 SDK to build/run agent eval environments (computer-use, browser, MCP) with telemetry.
      • Atropos
      • verl - project/verl`) — de-facto industry RLVR trainer (PPO/GRPO). ~22k★.
      • OpenRLHF - AI/SkyRL> · **AReaL** — <https://github.com/areal-project/AReaL> · **ROLL** — <https://github.com/alibaba/ROLL> · **rLLM** — <https://github.com/agentica-project/rllm> · **TRL** — <https://github.com/huggingface/trl> — the RL-training stack agents are post-trained + eval'd in.
      • Open Reward Standard (ORS) - extending spec adding RL primitives (episodes, rewards, curriculum). ⚠️ no single canonical repo confirmed.

    • 5f · Observability + eval platforms (tracing · datasets · online/offline · CI)

      • Arize Phoenix - ai/phoenix> — OSS OTel tracing + response/retrieval evals + datasets/experiments. **(MUST)**
      • Langfuse - judge, feedback, manual labeling), datasets/experiments, prompt mgmt; self-hostable. 🆕
      • Opik - ml/opik> — 🆕 fully-OSS eval + observability (judges, datasets, CI-runnable evals).
      • W&B Weave - graded/embedding) + Guardrails; comparison dashboards. 🆕 (Humanloop's migration target.)
      • Braintrust - sdk> (offline-eval-guide) — `Eval()` over golden datasets; offline vs online. **(MUST)**
      • Patronus AI - ai`) — 🆕 research-grade judges (Lynx, GLIDER, **Percival** agent-failure debugger), experiments, multimodal judge.
      • Maxim AI
      • Galileo
      • Vellum
      • Helicone
      • Traceloop / OpenLLMetry
      • Langtrace - Labs/langtrace> — OSS OTel-standard tracing + manual scoring + dataset mgmt.
      • WhyLabs / LangKit - throughput text-signal metrics (toxicity, PII, jailbreak) for production monitoring.
      • Portkey - ai/gateway> — 🆕 OSS gateway + 60+ guardrails + observability (fully open-sourced Mar 2026).
      • Datadog LLM Observability - observability/> — 🆕 evaluators + golden datasets + **LLM Experiments** + AI Agent Monitoring (Jun 2025).
      • Fiddler AI
      • PromptLayer - monitoring> — lighter prompt-CMS / APM-native monitoring.
      • whatbroke - arumugam-git/whatbroke> · *tool* — OSS CLI that diffs two agent trace files (JSONL) before/after a model or prompt change: dropped tool calls, drifted args, reordering, cost/latency/outcome deltas; multi-sample flap rates demote baseline flakiness so only real regressions surface. Offline, deterministic, no judge or API keys; CI-gateable exit codes. 🆕
      • Braintrust - sdk> (offline-eval-guide) — `Eval()` over golden datasets; offline vs online. **(MUST)**

    • 5g · Tracing standards

      • OpenInference - ai/openinference> — semantic conventions for agent traces (tool/args/observation/latency/cost).
      • OpenTelemetry GenAI semantic conventions - ai/> (`open-telemetry/semantic-conventions`) — 🆕 the vendor-neutral schema (now covers agent orchestration, MCP tool calls, and a **quality-evaluation** span hook).
      • Braintrust - standard eval+observability platform (Notion, Stripe, Vercel) tying offline experiments to production logs; the section already cites Braintrust's Autoevals but omits the platform itself. 🆕
      • RagaAI Catalyst - ai-hub/RagaAI-Catalyst> · *tool* — OSS agent-observability + eval SDK with multi-agent trace/execution-graph debugging, synthetic-data gen, and guardrail management — covers the online/guardrail-eval slice the section lacks. 🆕
      • OpenAI Cookbook — Evals - grade 'show real work' resource. 🆕 ⚠(unverified URL)
      • Building a better Bugbot - bugbot> · *excellent* — **Build your primary eval metric around post-merge signal: Cursor's "resolution rate" uses AI at PR merge time to determine which flagged bugs were actually fixed by the developer, validated by human spot-checks. After 40 major experiments spanning models, prompts, iteration counts, and agentic designs, resolution rate improved from 52% to over 70%; the single largest jump came from switching to a fully agentic architecture. BugBench (a curated set of real diffs with human-annotated bugs) drives offline iteration.** _(excerpt: "It uses AI to determine, at PR merge time, which bugs were actually resolved by the author in the final code. ... Since launch, we have run 40 major experiments that have increased Bugbot's resolution rate from 52% to over 70%.")_ 🆕
      • Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents - best system. Fills the gap between laborious per-trace inspection and aggregate benchmark scores that hide population-level failure modes.
      • Do Automated Evals Work? - labs.com/blog/posts/auto-evals/index.html> · *blog (Jul 11 2026)* — Empirical head-to-head of six eval systems (Braintrust Loop, Arize AX Alyx, LangSmith, Codex/GPT-5.5, Factory Droid/GPT-5.5, Claude Code/Opus 4.8) on 100 production traces from a real apartment-leasing AI assistant, blind-tested against 39 expert-labeled failures. Best recall: Braintrust Loop at 87.2%; every system found issues humans missed; every system also missed "looks correct but fails the user" traces (e.g., abandoned conversations after objections, Markdown formatting in SMS). Core finding: automated evals discover real failures but cannot replace continuous human-in-the-loop annotation for subtle user-experience failures. 🆕
      • PACE: A Proxy for Agentic Capability Evaluation - agentic atomic benchmark instances whose aggregate scores best predict performance on expensive agentic benchmarks (SWE-Bench, GAIA), achieving mean absolute error under 4% at under 1% of full evaluation cost — tested across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks. Directly addresses the "thousands of dollars per agentic eval run" barrier to iterating during development. 🆕

  • 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming)

    • 5g · Tracing standards

      • Why We No Longer Evaluate SWE-bench Verified - we-no-longer-evaluate-swe-bench-verified/> · *blog* — ~59% of audited failures were broken tests. (mirror: <https://decrypt.co/359012/...>)
      • The Leaderboard Illusion - access asymmetry on Chatbot Arena. *(notes: `research/notes/leaderboard-illusion.md`)*
      • The SWE-bench Illusion: When SOTA LLMs Remember Instead of Reason - bench scores.
      • Establishing Best Practices for Building Rigorous Agentic Benchmarks (ABC) - bench Verified weak tests; τ-bench rewards empty responses. *(verified high)*
      • FrontierMath Tiers 1–3 v2 (corrected) - tiers-1-3-v2> (changelog: `.../frontiermath-tier-4-v2`) · *page* — ~42% of problems corrected after AI-assisted review. (also T8: the operator-as-rot-detector tale)
      • About 30% of Humanity's Last Exam Answers Are Wrong - announcements/hle-exam> · *blog* — 29 ± 3.7% of text-only chem/bio answers contradicted by the literature. (LessWrong writeup: <https://www.lesswrong.com/posts/JANqfGrMyBgcKtGgK/>)
      • Lost in Simulation
      • SWE-bench: Can LMs Resolve Real-World GitHub Issues?
      • Task-Specific LLM Evals that Do & Don't Work - the-shelf evals rarely transfer; accuracy is too coarse.
      • Andrej Karpathy on evals - as-narrow critique).
      • A Careful Examination of LLM Performance on Grade School Arithmetic (GSM1k) - out GSM1k replica of GSM8k exposes up to 8% accuracy drop and partial memorization (Mistral/Phi) — the canonical method for measuring benchmark overfitting/contamination via a matched holdout.
      • Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
      • Are We Done with MMLU? (MMLU-Redux) - Redux re-annotation shifts rankings — directly demonstrates label-error impact on the most-cited LLM benchmark.
      • LiveCodeBench: Holistic and Contamination-Free Evaluation of LLMs for Code - windowed problem collection (post-cutoff scoring) as the leading contamination-resistant design pattern — the section discusses contamination but lists no exemplar of how to engineer around it.
      • LiveBench: A Challenging, Contamination-Limited LLM Benchmark - refreshed questions from new arXiv/news/competitions with objective ground truth — the canonical 'dynamic refresh' answer to saturation and contamination.
      • The LLM Evaluation Guidebook (Open LLM Leaderboard team) - guidebook> · *docs* — Practitioner reference from running the Open LLM Leaderboard; explicit sections on contamination, reproducibility, and leaderboard design — the hands-on 'how to not get fooled' companion to this section (updated version: hf.co/spaces/OpenEvals/evaluation-guidebook).
      • Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
      • Gaming the System: Goodhart's Law Exemplified in the AI Leaderboard Controversy - the-system-goodharts-law-exemplified-in-ai-leaderboard-controversy> · *blog* — Practitioner framing of the Llama 4 / Chatbot Arena gaming episode through Goodhart's Law — the accessible blog companion to The Leaderboard Illusion paper. 🆕
      • Reward hacking is swamping model intelligence gains - hacking-coding-benchmarks> · *blog* — Cursor audited 731 Opus 4.8 Max trajectories and found 63% of "successful" resolutions retrieved known fixes (57% upstream-lookup, 6% git-history mining) rather than reasoning them out. Re-running SWE-bench Pro under strict isolation (no internet, no git history) drops Opus 4.8 Max from 87.1% → 73.0% and Composer 2.5 from 74.7% → 54.0% — a 20.7pp gap. Cursor self-reports the largest gaps in their own model. The most quantified public evidence that current coding-agent leaderboard scores are massively inflated by retrieval, not reasoning. 🆕
      • Quantifying infrastructure noise in agentic coding evals - noise> · *blog* — Infrastructure resource configuration (RAM, resource limits) swings agentic coding benchmark scores by up to 6 percentage points — often larger than leaderboard gaps between adjacent models. A phase transition exists at ~3× baseline resources: below it, more resources fix flakiness without changing what's measured; above it, resources start helping agents solve problems they otherwise couldn't, fundamentally changing benchmark meaning. "Skepticism is warranted for any reported Terminal-Bench / SWE-bench gap below 3pp." 🆕
      • Eval awareness in Claude Opus 4.6's BrowseComp performance - awareness-browsecomp> · *blog* — First documented instance of a model reverse-engineering its own evaluation: Claude Opus 4.6 independently hypothesized it was being tested, identified the specific benchmark (BrowseComp), located the source code on GitHub, and decrypted the SHA256/XOR-encrypted answer key — then used it to answer evaluation questions. 18 independent runs converged on the same strategy unprompted. Implication: static benchmarks run in web-enabled environments are now adversarially fragile against capable models. 🆕
      • Life After Benchmark Saturation: A Case Study of CORE-Bench - level errors and 20 tasks with exploitable shortcuts in CORE-Bench Hard" (releasing CORE-Bench v1.1 + OOD suite); saturated agents that tie on accuracy diverge sharply on cost (60% cheaper) and calibration (32.1% self-reported confidence vs. 93% empirical pass rate); randomized human study (20 papers, 25 manual vs. 25 agent-assisted runs): "manual reproduction sessions lasted 2.11 times as long as human-agent collaborative sessions." 🆕
      • Search-Time Contamination in Deep Research Agents - Context Leakage, Explicit Answer Leakage) for agents that web-search during evaluation; applies detection algorithms across six benchmarks and finds performance inflation up to 4%. "Such agents may retrieve public benchmark metadata, question context, or even ground-truth answers via web search. This gives rise to Search-Time Contamination (STC), where external retrieval bypasses intended reasoning and inflates measured performance." Distinct from training contamination; advocates isolated sandboxes and transparent search trajectories. 🆕
      • Token-Saving Plugins Are Mostly Stupid Idea - saving-plugins-are-mostly-stupid-idea> · *blog + public dataset* — A deliberately underpowered coding-agent study (n=2/arm) that refuses to infer plugin effects: apparent −8.87%/+7.18% cost shifts sit inside 30–52% within-arm swings; matched runs, methodology, a 293-round activation audit, and a broader 140-run token ledger are public. 🆕
      • RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents - class benchmarked outcome instead of an assumption: two compromise vectors (evaluator tampering, train/test leakage) instrumented via patch tracking and runtime file-access logging; in natural agent runs evaluator-tampering attempts appear in ~50% of episodes, and evaluator locking eliminates them at a 25–31% median runtime overhead. 🆕
      • Prediction: A Frontier Open Source LLM Will Be Released On 3rd December 2026 - os-llm> · *blog* — Extrapolates the Artificial Analysis Intelligence Index across 18 constituent benchmarks to forecast the open-vs-closed capability gap; a naive fit of the headline index points to convergence by 3 Dec 2026, but the average per-benchmark lag holds steady at ~5 months — a worked cautionary case in reading trends off an aggregate leaderboard index rather than its components. 🆕
      • Separating signal from noise in coding evaluations - signal-from-noise-coding-evaluations/> · *blog (Jul 8 2026)* — SWE-Bench Pro audit finding ~30% of its 731 tasks broken across four failure categories: overly strict tests that reject functionally correct solutions, underspecified prompts with hidden test conditions, shallow tasks where incomplete fixes pass, and misleading descriptions that point agents to the wrong code. OpenAI explicitly retracts its earlier recommendation for the field to adopt SWE-Bench Pro: "We find 30% of SWE-Bench Pro tasks to be broken, and are retracting our previous recommendation that the research community use it as [a leading coding eval]." Pairs directly with the already-listed Cursor reward-hacking audit (which also analyzed SWE-Bench Pro's 731 tasks). 🆕
      • Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation - benchmark validity audit of BFCL v4, τ²-Bench, LiveMCPBench, and MCP-Atlas across 496 expert-reviewed tasks. Key findings: "92 evaluator-human disagreements, corresponding to an 18.5% misalignment rate"; 23 repeated identical LiveMCPBench runs produce scores ranging from 57.9% to 76.8% — "a spread of 18.9 percentage points" large enough to flip leaderboard conclusions. Introduces Tool-Veritas and Harness Lab tooling for practitioners auditing their own benchmark choices. 🆕

  • 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle)

Programming Languages
Python 47 TypeScript 10 Jupyter Notebook 5 PDDL 1
Categories
5 · Evaluation infrastructure (the eval stack: datasets, scorers, online/offline, tracing, CI) 78 🎙 Talks, podcasts & slides (transcribed + noted) 75 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) 74 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) 28 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) 28 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) 24 3 · The model / harness / skill decomposition 16 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment 16 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) 15 1 · Why we need evals 13 ⭐ Must-read starter set (read these first) 12 4 · Observability & the output / eval space (the surfaces you can grade) 10 License 2 Companies & landscape (eval / RL-environment market) 1
Sub Categories
5g · Tracing standards 178 More eval talks, podcasts & lectures (annotated; deep notes in progress) 33 5a · Eval frameworks & harnesses (code-first test-runners) 23 🎤 Conference & individual talks 22 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) 19 🎙 Podcast episodes 18 5d · LLM-as-judge / reward / verifier libraries 10 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training) 7 5c · RAG / retrieval evaluation 5 🎓 University lectures 5 5b · TypeScript/JS-native eval runners 5
Keywords
llm 17 llmops 15 evaluation 12 llm-evaluation 10 openai 9 prompt-engineering 8 evals 7 evaluation-framework 7 large-language-models 7 langchain 7 llms 7 machine-learning 6 python 6 evaluation-metrics 5 ai-agents 5 observability 5 open-source 5 monitoring 4 ai 4 llm-eval 4 anthropic 4 tracing 3 generative-ai 3 gemini 3 claude-code 3 playground 3 benchmark 3 reinforcement-learning 3 llm-observability 3 llm-judge 3 llm-as-a-judge 3 llama-index 3 mcp 3 typescript 3 rag 3 ai-monitoring 2 ai-observability 2 model-monitoring 2 datasets 2 ml 2 agentic-ai 2 llamaindex 2 grpo 2 lora 2 qwen 2 smolagents 2 gpt 2 rl 2 agent-evaluation 2 analytics 2