llm-eval-tracker
Live index of LLM evaluation tools and benchmarks, refreshed every 15 minutes from GitHub
https://github.com/linny006/llm-eval-tracker
Last synced: about 1 month ago
JSON representation
📋 Current Items
- goldbarth/chartula-evals - 10-12 | How Chartula is measured: eval cases, run costs, and judgement of the generated changelogs. |
- promptfoo/promptfoo - 10-13 | Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, C |
- Kondwani10/Origin-Continuum - 10-13 | 🌐 Define and explore the Origin ↔ Continuum framework, ensuring proper attribution and continuity in dependency relation |
- verifywise-ai/verifywise - 10-13 | Complete AI governance and LLM Evals platform with support for EU AI Act, ISO 42001, NIST AI RMF and 20+ more AI framewo |
- Arize-ai/phoenix - 10-13 | AI Observability & Evaluation |
- decimal-labs/decimalai-python - 10-12 | Python SDK for agent evals and skill routing — measure a skill's real lift before you trust it |
- ChelseaKR/plumbline - 10-07 | v0.2.0. Fail-closed evaluation harness for government-facing chat systems: reproducible, provenance-stamped audit verdic |
- ahmedmoha9088/PhoenixFish - 10-12 | Elevate your Paper server with an immersive fishing overhaul featuring custom fish, rods, bait, and a dynamic minigame. |
- ChelseaKR/sprout - 10-09 | In-build reference implementation: an offline-first plant-care assistant and public evaluation harness with cited-corpus |
- saddled-panicattack529/idea-evaluation-pipeline - 10-12 | Streamline research idea evaluation for finance and economics to reach top journal quality using an iterative, AI-assist |
- truera/trulens - 10-12 | Evaluation and Tracking for LLM Experiments and AI Agents |
- Giskard-AI/giskard-oss - 10-12 | 🐢 Open-Source Evaluation & Testing library for LLM Agents |
- ChelseaKR/fare-policy-assistant - 10-12 | Beta. Reduced-fare policy assistant citing dated corpus passages in English and Spanish; the bilingual-parity gate is cu |
- Sans-cell-art/-Project-Phoenix-The-E-Waste-Supercomputer- - 10-11 | ♻️ Transform e-waste into a powerful, low-cost cloud operating system, unlocking computing potential and promoting resou |
- bhavya7995/AI_governance - 10-11 | 🤖 Streamline AI-assisted development with a governance kit for rules, enforcement, and decision-making, ensuring speed a |
- ChelseaKR/gauntlet - 10-03 | pip install gauntlet-evals · v0.1.0. Merge-blocking evaluation gates for generative AI features: YAML suites run against |
- HaileyStorm/Creative-Writing-Rubrics - 10-11 | HBQ-RS: composable binary-question rubrics for creative writing, draft judging, benchmarking, and synthetic data. |
- jeremylongshore/j-rig-skill-binary-eval - 10-12 | Binary-criteria evaluation harness for Claude skills with planned extension to plugins, agents, and MCP servers. Score e |
- homayoun-safarpour/homayoun-safarpour - 10-08 | judge-drift-sentinel · judge-reliability-kit · agent-loop-engine · trace-gate · ai-eng-skill-range |
- SFX-TECH/sfx-lead-intelligence - 10-05 | SFX Lead Intelligence Command Center: local-LLM hub plus lead dashboard, quality lifted 61 to 99 percent via a ground-tr |
- IonDen/mlx-quant-fidelity - 09-29 | Measure quantization quality loss on Apple Silicon MLX — KL divergence, top-token flip rate and perplexity delta for KV- |
- open-compass/opencompass - 10-12 | OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, |
- instax-dutta/sysone-bench - 10-10 | First independent head-to-head benchmark of System One decision models (Laya vs Jev) on byte-identical inputs |
- arunash/earshot - 10-07 | Blind, reproducible benchmark for production phone voice agents. Measures barge-in latency, false-stop rate and the late |
- AndyZhang666666/AndyZhang666666.github.io - 10-06 | Personal site with two interactive LLM eval harness demos |
- lilyco-42/lyco-router-eval - 10-11 | Independent-eval-Qwen3-0.6B-ondevice-router-40cases |
- IcodeNet/eval-dashboards - 10-12 | Beautiful, themeable HTML dashboards, quality gates, and publishing for AI agent eval runs. NYC/Istanbul-style — bring y |
- crashlabsai/trojan-harbor - 10-06 | An indirect prompt injection evaluation suite for Harbor, with eight matched attack and clean task families and determin |
- sx4im/skillcheck - 10-10 | Blind A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts. Detects placebo and harmful instruct |
- judgekeeper/judgekeeper - 10-13 | Check your LLM-as-a-judge: see how often it agrees with human labels, in one command. |
- dylanma8232-art/daming-skillgain - 10-12 | 你的 Skill 到底有没有用?跑一次就知道——用真实任务做对照测试,看它在每个模型上是帮忙还是拖后腿。 | Measure whether a skill actually helps, per model, via paired wit |
- amareshhebbar/gitgrounded - 10-12 | Catches AI regressions before your users do, diffs a prompt or model change, has an AI write targeted tests, judges old |
- mrodgersjs-web/rig-enhanced-evals - 10-10 | L10 self-evolving LLM evaluation harness that learns from failures — faithfulness, relevance, toxicity, format metrics w |
- quannie255-star/FinData-Agent - 10-10 | 金融长文本智能问答系统:检索·判定·记忆·归因四层全链(BM25F 逐选项检索 + 选项级证据隔离 + 分层记忆压缩 + 引用观测归因),自建五域评测集(法规/财报/保险/合同/研报,50 题含 193 页募集说明书),791 例测试。场景 |
- Icaro0310/devin-evals - 10-09 | Deterministic eval harness: replay recorded Devin sessions against rubric graders |
- gopals09920/llm-bias-toxicity-audit - 10-09 | Automated LLM bias and toxicity auditing pipeline with flexible scoring and Streamlit UI. |
- yukincom/llm-SugarScape - 10-08 | Multi-agent simulation using LLMs. Agents autonomously decide actions for survival, reproduction, and social behavior in |
- sthanika-ai/Indic-KCC-Agri-Advisory-Benchmark - 10-08 | Agricultural-advisory QA benchmark in 11 Indian languages — 500 questions × 11, built from real Kisan Call Centre farmer |
- jin2004-cmd/codex-practical-skills - 10-07 | 13 ready-to-use Codex/Claude Agent skills: job-search memory, offer decision, LLM output quality gate, portfolio publish |
- nxtaillc/disclosure-eval - 10-06 | Test AI customer service agents for confirmation-based PII disclosure. |
- Striveworks/valor - 10-06 | Valor is a lightweight, numpy-based library designed for fast and seamless evaluation of machine learning models. |
- nowackk-cp/karkontrol - 10-05 | AI yazdı, testler kanıtladı: pazaryeri kâr motoru · 669 test · %98,83 mutasyon skoru · LLM eval 52/52 |
- nikolas-sapa/sigeval - 10-05 | Statistically rigorous LLM evaluation for pytest — treats every eval as a proportion with a Wilson confidence interval, |
- jatinsihag2345/awesome-ai-evaluations - 10-04 | A curated list of frontier AI evaluation benchmarks, long-horizon agent test suites, terminal sandboxes, RLVR math verif |
- jatinsihag2345/llm-eval-toolkit - 10-04 | Modular Python SDK for deterministic grading, AST semantic diffing, sandboxed execution, and fuzzy scoring of LLM output |
- derrickqin/google-adk-laya-router - 10-02 | Route each ADK request to the right Gemini tier with Laya on a CPU-only container (Cloud Run), plus an eval scored by co |
- Deivid9920/PROMETHEUSV2-NEUROTOPOLOGY - 10-02 | Topological Data Analysis (TDA) layer for LLM self-improvement loops featuring persistent homology promotion gates, repr |
- therealfullmetal55555/llm-eval-harness - 09-30 | Automated LLM Evaluation Harness (Superseded by Passmark & Bench-Suite) |
- lehigh-university-libraries/htr - 09-29 | Handwritten Text Recognition llm eval tool |
- VioletScar-Hui/sparkjury - 09-29 | 给别人的 Agent 做体检的评测 Agent:本地三家族裁判打分、分歧交云端仲裁、badcase 聚类、证据卡片、pass^3 回归 | 第三届 NVIDIA DGX Spark 黑客松 · 能工智人5X |
🔗 Related live trackers
- trending-claude-skills - skills`)
- mcp-servers-live - server`)
- cursor-rules-live - rules`)
- claude-code-plugin-tracker - code`)
- llm-agents-radar - agent`)
- rag-radar
- agent-framework-radar - framework`)
- vector-db-live - database`)
- llmops-radar
-
prompt-tools-live
- engineering tools and prompt repos (
topic:prompt-engineering) - agent-eval-harness - eval`)
- skills-tracker - skills`)
-
awesome-agent-skills
- updated awesome-list of AI agent skills (
topic:agent-skills)