An open API service indexing awesome lists of open source software.

llm-eval-tracker

Live index of LLM evaluation tools and benchmarks, refreshed every 15 minutes from GitHub
https://github.com/linny006/llm-eval-tracker

Last synced: about 1 month ago
JSON representation

📋 Current Items

  • goldbarth/chartula-evals - 10-12 | How Chartula is measured: eval cases, run costs, and judgement of the generated changelogs. |
  • promptfoo/promptfoo - 10-13 | Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, C |
  • Kondwani10/Origin-Continuum - 10-13 | 🌐 Define and explore the Origin ↔ Continuum framework, ensuring proper attribution and continuity in dependency relation |
  • verifywise-ai/verifywise - 10-13 | Complete AI governance and LLM Evals platform with support for EU AI Act, ISO 42001, NIST AI RMF and 20+ more AI framewo |
  • Arize-ai/phoenix - 10-13 | AI Observability & Evaluation |
  • decimal-labs/decimalai-python - 10-12 | Python SDK for agent evals and skill routing — measure a skill's real lift before you trust it |
  • ChelseaKR/plumbline - 10-07 | v0.2.0. Fail-closed evaluation harness for government-facing chat systems: reproducible, provenance-stamped audit verdic |
  • ahmedmoha9088/PhoenixFish - 10-12 | Elevate your Paper server with an immersive fishing overhaul featuring custom fish, rods, bait, and a dynamic minigame. |
  • ChelseaKR/sprout - 10-09 | In-build reference implementation: an offline-first plant-care assistant and public evaluation harness with cited-corpus |
  • saddled-panicattack529/idea-evaluation-pipeline - 10-12 | Streamline research idea evaluation for finance and economics to reach top journal quality using an iterative, AI-assist |
  • truera/trulens - 10-12 | Evaluation and Tracking for LLM Experiments and AI Agents |
  • Giskard-AI/giskard-oss - 10-12 | 🐢 Open-Source Evaluation & Testing library for LLM Agents |
  • ChelseaKR/fare-policy-assistant - 10-12 | Beta. Reduced-fare policy assistant citing dated corpus passages in English and Spanish; the bilingual-parity gate is cu |
  • Sans-cell-art/-Project-Phoenix-The-E-Waste-Supercomputer- - 10-11 | ♻️ Transform e-waste into a powerful, low-cost cloud operating system, unlocking computing potential and promoting resou |
  • bhavya7995/AI_governance - 10-11 | 🤖 Streamline AI-assisted development with a governance kit for rules, enforcement, and decision-making, ensuring speed a |
  • ChelseaKR/gauntlet - 10-03 | pip install gauntlet-evals · v0.1.0. Merge-blocking evaluation gates for generative AI features: YAML suites run against |
  • HaileyStorm/Creative-Writing-Rubrics - 10-11 | HBQ-RS: composable binary-question rubrics for creative writing, draft judging, benchmarking, and synthetic data. |
  • jeremylongshore/j-rig-skill-binary-eval - 10-12 | Binary-criteria evaluation harness for Claude skills with planned extension to plugins, agents, and MCP servers. Score e |
  • homayoun-safarpour/homayoun-safarpour - 10-08 | judge-drift-sentinel · judge-reliability-kit · agent-loop-engine · trace-gate · ai-eng-skill-range |
  • SFX-TECH/sfx-lead-intelligence - 10-05 | SFX Lead Intelligence Command Center: local-LLM hub plus lead dashboard, quality lifted 61 to 99 percent via a ground-tr |
  • IonDen/mlx-quant-fidelity - 09-29 | Measure quantization quality loss on Apple Silicon MLX — KL divergence, top-token flip rate and perplexity delta for KV- |
  • open-compass/opencompass - 10-12 | OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, |
  • instax-dutta/sysone-bench - 10-10 | First independent head-to-head benchmark of System One decision models (Laya vs Jev) on byte-identical inputs |
  • arunash/earshot - 10-07 | Blind, reproducible benchmark for production phone voice agents. Measures barge-in latency, false-stop rate and the late |
  • AndyZhang666666/AndyZhang666666.github.io - 10-06 | Personal site with two interactive LLM eval harness demos |
  • lilyco-42/lyco-router-eval - 10-11 | Independent-eval-Qwen3-0.6B-ondevice-router-40cases |
  • IcodeNet/eval-dashboards - 10-12 | Beautiful, themeable HTML dashboards, quality gates, and publishing for AI agent eval runs. NYC/Istanbul-style — bring y |
  • crashlabsai/trojan-harbor - 10-06 | An indirect prompt injection evaluation suite for Harbor, with eight matched attack and clean task families and determin |
  • sx4im/skillcheck - 10-10 | Blind A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts. Detects placebo and harmful instruct |
  • judgekeeper/judgekeeper - 10-13 | Check your LLM-as-a-judge: see how often it agrees with human labels, in one command. |
  • dylanma8232-art/daming-skillgain - 10-12 | 你的 Skill 到底有没有用?跑一次就知道——用真实任务做对照测试,看它在每个模型上是帮忙还是拖后腿。 | Measure whether a skill actually helps, per model, via paired wit |
  • amareshhebbar/gitgrounded - 10-12 | Catches AI regressions before your users do, diffs a prompt or model change, has an AI write targeted tests, judges old |
  • mrodgersjs-web/rig-enhanced-evals - 10-10 | L10 self-evolving LLM evaluation harness that learns from failures — faithfulness, relevance, toxicity, format metrics w |
  • quannie255-star/FinData-Agent - 10-10 | 金融长文本智能问答系统:检索·判定·记忆·归因四层全链(BM25F 逐选项检索 + 选项级证据隔离 + 分层记忆压缩 + 引用观测归因),自建五域评测集(法规/财报/保险/合同/研报,50 题含 193 页募集说明书),791 例测试。场景 |
  • Icaro0310/devin-evals - 10-09 | Deterministic eval harness: replay recorded Devin sessions against rubric graders |
  • gopals09920/llm-bias-toxicity-audit - 10-09 | Automated LLM bias and toxicity auditing pipeline with flexible scoring and Streamlit UI. |
  • yukincom/llm-SugarScape - 10-08 | Multi-agent simulation using LLMs. Agents autonomously decide actions for survival, reproduction, and social behavior in |
  • sthanika-ai/Indic-KCC-Agri-Advisory-Benchmark - 10-08 | Agricultural-advisory QA benchmark in 11 Indian languages — 500 questions × 11, built from real Kisan Call Centre farmer |
  • jin2004-cmd/codex-practical-skills - 10-07 | 13 ready-to-use Codex/Claude Agent skills: job-search memory, offer decision, LLM output quality gate, portfolio publish |
  • nxtaillc/disclosure-eval - 10-06 | Test AI customer service agents for confirmation-based PII disclosure. |
  • Striveworks/valor - 10-06 | Valor is a lightweight, numpy-based library designed for fast and seamless evaluation of machine learning models. |
  • nowackk-cp/karkontrol - 10-05 | AI yazdı, testler kanıtladı: pazaryeri kâr motoru · 669 test · %98,83 mutasyon skoru · LLM eval 52/52 |
  • nikolas-sapa/sigeval - 10-05 | Statistically rigorous LLM evaluation for pytest — treats every eval as a proportion with a Wilson confidence interval, |
  • jatinsihag2345/awesome-ai-evaluations - 10-04 | A curated list of frontier AI evaluation benchmarks, long-horizon agent test suites, terminal sandboxes, RLVR math verif |
  • jatinsihag2345/llm-eval-toolkit - 10-04 | Modular Python SDK for deterministic grading, AST semantic diffing, sandboxed execution, and fuzzy scoring of LLM output |
  • derrickqin/google-adk-laya-router - 10-02 | Route each ADK request to the right Gemini tier with Laya on a CPU-only container (Cloud Run), plus an eval scored by co |
  • Deivid9920/PROMETHEUSV2-NEUROTOPOLOGY - 10-02 | Topological Data Analysis (TDA) layer for LLM self-improvement loops featuring persistent homology promotion gates, repr |
  • therealfullmetal55555/llm-eval-harness - 09-30 | Automated LLM Evaluation Harness (Superseded by Passmark & Bench-Suite) |
  • lehigh-university-libraries/htr - 09-29 | Handwritten Text Recognition llm eval tool |
  • VioletScar-Hui/sparkjury - 09-29 | 给别人的 Agent 做体检的评测 Agent:本地三家族裁判打分、分歧交云端仲裁、badcase 聚类、证据卡片、pass^3 回归 | 第三届 NVIDIA DGX Spark 黑客松 · 能工智人5X |

🔗 Related live trackers