Awesome-AI-Evaluations-Tools
Collection of frameworks and tools for AI evalations, including tool-use, agentic AI, MCP, and multimodal
https://github.com/danielrosehill/Awesome-AI-Evaluations-Tools
Last synced: 29 days ago
JSON representation
Agentic & Tool Use Evals
Out Of Scope
- Gorilla - commit/ShishirPatil/gorilla) |
- AgentBench - commit/THUDM/AgentBench) |
- fast-agent - agent?style=social) | |
- AgentEvals - ai/agentevals?style=social) | |
- any-agent - ai/any-agent?style=social) | |
- Strands Agents Evals - agents/evals?style=social) | |
- AgentCPM - to-end infrastructure for training and evaluating various LLM agents. | | |
- MemoryAgentBench - turn interactions (ICLR 2026). | | |
- HaluMem - level hallucination evaluation benchmark tailored to agent memory systems. | | |
- HarnessLab - commit/polskiTran/HarnessLab) |
- ResearchHarness - using LLM agents with fair benchmark evaluation and personal assistant workflows. | | |
- iris-eval mcp-server - eval/mcp-server?style=social) | |
- AIOpsLab - commit/microsoft/AIOpsLab) |
Awesome Lists / Resources
Out Of Scope
- Every Eval Ever - commit/evaleval/every_eval_ever) |
- llm-benchmark - benchmark?style=social) | |
Benchmarks
Out Of Scope
- Bloom - research/bloom?style=social) | |
- Dangerous Capability Evaluations - deepmind/dangerous-capability-evaluations?style=social) | |
- RewardBench - bench?style=social) | |
- OpenBench - agnostic, open-source evaluation infrastructure for language models. | | |
- Claw-Eval - verified. | | |
- genai-bench - level performance evaluation of LLM serving systems. | | |
- Sparse Frontier - free sparse attention in LLMs. | | |
- med-lm-envs - AI/med-lm-envs?style=social) | |
- MedEvalKit - damo-academy/MedEvalKit?style=social) | |
- OpenHands Benchmarks - commit/OpenHands/benchmarks) |
- SkillsBench - ai/skillsbench?style=social) | |
- article-extraction-benchmark - extraction-benchmark?style=social) | |
- GameWorld - based benchmark environment for evaluating AI agents. | | |
- BIG-bench - bench?style=social) | |
Bias & Fairness (Cultural, Political, Social)
Out Of Scope
- BBQ - mll/BBQ?style=social) | |
- StereoSet - commit/moinnadeem/StereoSet) |
- CrowS-Pairs - mll/crows-pairs?style=social) | |
- CDEval - style axes. | | |
- OpinionQA - lab/opinions_qa?style=social) | |
- WorldValuesBench - commit/Demon702/WorldValuesBench) |
- BLEnD - centric bias. | | |
Browser & Web Agent Evals
Out Of Scope
- WebArena - arena-x/webarena?style=social) | |
- VisualWebArena - arena-x/visualwebarena?style=social) | |
- Mind2Web - NLP-Group/Mind2Web?style=social) | |
- BrowserGym - commit/ServiceNow/BrowserGym) |
- WebShop - commerce environment for evaluating language-grounded web-interaction agents. | | |
- MiniWoB++ - interaction tasks for evaluating web-using agents. | | |
Coding Evals
Out Of Scope
- SWE-bench - world GitHub issue resolution. | | |
Grounding & Hallucination
Out Of Scope
- TruthfulQA - commit/sylinrl/TruthfulQA) |
- HaluEval - scale hallucination evaluation benchmark for LLMs. | | |
- FActScore - grained atomic evaluation of factual precision in long-form text generation. | | |
- SelfCheckGPT - resource black-box hallucination detection for LLMs via self-consistency. | | |
- RAGTruth - level annotations on model outputs. | | |
List Author
Out Of Scope
Multimodal Evals
Out Of Scope
- lmms-eval - for-all multimodal evaluation toolkit across text, image, video, and audio tasks. | | |
- VLMEvalKit - source evaluation toolkit for large multi-modality models — supports 220+ LMMs and 80+ benchmarks. | | |
- FlagEvalMM - baai/FlagEvalMM?style=social) | |
- EASI - Lab/EASI?style=social) | |
- EmoBench-M - gml/EmoBench-M?style=social) | |
- multimodalhugs - commit/GerrySant/multimodalhugs) |
- OmniSafeBench-MM - MM?style=social) | |
- MMT-Bench - Bench?style=social) | |
- MME-Emotion - Emotion?style=social) | |
Platforms And Software
Out Of Scope
- DeepEval - ai/deepeval?style=social) | |
- Phoenix - ai/phoenix?style=social) | |
- Opik - ml/opik?style=social) | |
- Inspect AI - commit/UKGovernmentBEIS/inspect_ai) |
- LangWatch - commit/langwatch/langwatch) |
- lm-evaluation-harness - shot evaluation of language models. | | |
- Harbor - framework/harbor?style=social) | |
- Evaluate - commit/huggingface/evaluate) |
- OLMES - commit/allenai/olmes) |
- Giskard OSS - source evaluation and testing library for LLM agents. | | |
- OpenAI Evals - source registry of benchmarks. | | |
- EverOS - term memory for self-evolving agents. | | |
- OpenEvals - ai/openevals?style=social) | |
- Evalite - powered apps with TypeScript. | | |
- LightEval - in-one toolkit for evaluating LLMs across multiple backends. | | |
- EvalAI - of-the-art AI on community challenges. | | |
- PyKEEN - commit/pykeen/pykeen) |
- RouteLLM - sys/RouteLLM?style=social) | |
- Oumi - tune, evaluate, and deploy gpt-oss, Qwen3, DeepSeek-R1, or any open source LLM/VLM. | | |
- Ignite - level library to help train and evaluate neural networks in PyTorch flexibly and transparently. | | |
- Bench - ai/bench?style=social) | |
- OpenLIT - native observability, evaluations, prompt management, guardrails. | | |
- GuideLLM - world inference needs. | | |
- Vivaria - commit/METR/vivaria) |
- Helicone - commit/Helicone/helicone) |
- EvalScope - commit/modelscope/evalscope) |
- Eval (ai-twinkle) - performance LLM evaluation framework with parallel API calls — up to 17× faster than sequential tools. | | |
- simple-llm-eval - as-a-judge, from CyberArk. | | |
- Evidently - source ML and LLM observability framework — evaluate, test, and monitor any AI-powered system. | | |
- ai-eval - eval?style=social) | |
- neuro-judge - as-a-Judge evaluation framework — multi-model, multi-criteria, with cost tracking and HTML reports. | | |
- OpenJudge - ai/OpenJudge?style=social) | |
- One-Eval - Eval?style=social) | |
- GAGE - Research/GAGE?style=social) | |
- OpenCompass - compass/opencompass?style=social) | |
- Langfuse - commit/langfuse/langfuse) |
- MLflow - commit/mlflow/mlflow) |
- Agenta - source LLMOps platform — prompt playground, prompt management, LLM evaluation, and observability. | | |
- Ollama Grid Search - platform desktop application to evaluate and compare LLM models, written in Rust and React. | | |
- Speculators - project/speculators?style=social) | |
- AlpacaEval - following language models — LLM-based, fast, cheap, replicable. | | |
- continuous-eval - driven evaluation framework for LLM applications. | | |
- Eureka ML Insights - ml-insights?style=social) | |
- SumEval - tested multilingual evaluation framework for text summarization. | | |
- HELM - crfm/helm?style=social) | |
- Promptimize - io/promptimize?style=social) | |
- MiroEval - commit/MiroMindAI/MiroEval) |
- OmniEvalKit - extensions across modalities, languages, and tasks. | | |
- promptfoo - teaming LLM apps — test-driven prompt engineering. | | |
- FastChat - Bench. | | |
RAG & Retrieval
Out Of Scope
- Ragas - commit/vibrantlabsai/ragas) |
- TruLens - commit/truera/trulens) |
- AutoRAG - source framework for RAG evaluation and optimization with AutoML-style automation. | | |
- RAG Experiment Accelerator - experiment-accelerator?style=social) | |
- RAG-grounding-eval - grounding-eval?style=social) | |
Refusal & Safety
Out Of Scope
- HarmBench - commit/centerforaisafety/HarmBench) |
- Rogue - dev/rogue?style=social) | |
- Moonshot - team any LLM application. | | |
- Agent Security Sandbox - using LLM agents. | | |
- AgentDefense-Bench - layer defenses in MCP-based AI agent systems. | | |
- AgentDojo - spylab/agentdojo?style=social) | |
- XSTest - refusal) in LLMs. | | |
- SORRY-Bench - Bench/sorry-bench?style=social) | |
- StrongREJECT - commit/alexandrasouly/strongreject) |
- Do-Not-Answer - AI/do-not-answer?style=social) | |
- WildGuard - commit/allenai/wildguard) |
- CValues - PLUG/CValues?style=social) | |
- Flames - adversarial Chinese values alignment benchmark for evaluating refusal on sensitive topics. | | |
- SafetyBench - coai/SafetyBench?style=social) | |
Sycophancy & Dissent
Out Of Scope
- sycophancy-eval - tong/sycophancy-eval?style=social) | |
Vision Evals
Out Of Scope
- OmniDocBench - commit/opendatalab/OmniDocBench) |