{"id":51002555,"url":"https://github.com/chenhaodev/agentbench","last_synced_at":"2026-06-20T16:32:35.265Z","repository":{"id":365341127,"uuid":"1271677561","full_name":"chenhaodev/agentbench","owner":"chenhaodev","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-16T23:04:41.000Z","size":73,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-17T00:27:37.654Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chenhaodev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-16T22:55:44.000Z","updated_at":"2026-06-17T00:22:39.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/chenhaodev/agentbench","commit_stats":null,"previous_names":["chenhaodev/agentbench"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/chenhaodev/agentbench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenhaodev%2Fagentbench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenhaodev%2Fagentbench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenhaodev%2Fagentbench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenhaodev%2Fagentbench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chenhaodev","download_url":"https://codeload.github.com/chenhaodev/agentbench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenhaodev%2Fagentbench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34578089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-20T02:00:06.407Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-20T16:32:34.050Z","updated_at":"2026-06-20T16:32:35.258Z","avatar_url":"https://github.com/chenhaodev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AgentBench · 主观榜单评议 (subjective leaderboard review)\n\n\u003e 不是再做一个排行榜,而是给现有的 AI 榜单写**专家判语**:这个榜可不可信、高分到底买到了什么、要不要据此在 MoA(Mixture-of-Agents)里选模型。\n\n## 一分钟看懂(无需 AI 背景)\n\n市面上有几百个 AI\"排行榜\"(leaderboard)——谁的模型在医疗、情商、推理上更强。问题是:**榜单本身良莠不齐**。有的有几百位医生背书,有的只是网友点赞;有的题目早被模型背熟(饱和),有的还在区分高下。\n\nAgentBench 不重复排名。它对每个值得关注的榜单出一份**人签字的主观判语**:\n- 这个榜**谁在做**、**怎么测**、**现在谁领先**(客观事实,由 AI 整理);\n- 然后一位**专家**(本仓库:chenhao)写下**该不该信、信到什么程度**(主观判语,只能人来写)。\n\n产物是一个 GitHub 页面,给两类人看:① 非 AI 人士科普查阅;② 跨领域的 AI 工程师快速判断\"选哪个榜的冠军进我的系统\"。\n\n## 它和 HuggingFace 的\"榜中榜\"有什么不同\n\nHuggingFace 上**已经有**客观的榜单聚合器(leaderboard of leaderboards),能按 likes/trending 排序、按类别筛选,例如\n[OpenEvals/every-leaderboards](https://huggingface.co/spaces/OpenEvals/every-leaderboards)、\n[MAYA-AI/all-leaderboard](https://huggingface.co/spaces/MAYA-AI/all-leaderboard)。\n\n\u003e **聚合器回答\"谁在榜上\";AgentBench 回答\"该不该信这个榜\"。** 我们的差异化是**专家判语层**,不是再聚合一次排名——那会输给自动更新的 Space。\n\n## 一个榜单值不值得信:看什么 metric\n\n不同\"排序\"是**三条不同的轴**,本仓库的 schema 把它们显式分开,避免读者把人气当质量:\n\n| 轴 | 含义 | schema 字段 |\n|---|---|---|\n| **人气** HF likes / downloads / trending | 采用度,**会被刷,≠好** | `popularity` |\n| **学术权威** 引用数(Google Scholar / Semantic Scholar) | 被同行引用的程度 | `authority.citation_count` |\n| **能力** 榜内 rank / score | 模型在该榜的成绩 | `models_ranked` |\n\n**医疗榜单尤其要注意**:最有名的医疗榜(HealthBench、MedBench、MedHELM)**不是靠 HF likes 出名的**——它们多半连 HF Space 的点赞都没有。它们的权威货币是:\n\n1. **临床参与度** — HealthBench 有 262 位医生写评分标准;MedHELM 覆盖 15 个专科的临床医生;MedBench v4 由 500+ 机构的医生审核;\n2. **背书机构** — OpenAI / Stanford / 国家级医疗 AI 基地;\n3. **引用数**;\n4. **临床真实度** — 真实病历文本/多轮对话 \u003e 选择题知识背诵(MultiMedQA 类已**饱和**)。\n\n\u003e 结论:在医疗领域,`popularity` 是弱信号,真正决定\"值不值得信\"的是 `authority`(机构/医生/引用)+ **临床真实度**。\n\n## 已收录的榜单(草稿)\n\n| 榜单 | 领域 | 权威信号 | 状态 |\n|---|---|---|---|\n| [MedBench](entries/medbench.md) | 中文医疗(LLM/多模态/Agent) | 500+ 机构、动态防污染 | draft(待签) |\n| [HealthBench](entries/healthbench.md) | 医疗安全/健康对话 | 262 位医生评分标准(OpenAI) | draft(待签) |\n| [MedHELM](entries/medhelm.md) | 121 项临床任务(整体评估) | Stanford,临床验证的任务分类 | draft(待签) |\n| [BRIDGE](entries/bridge.md) | 多语种真实临床文本 | 87 任务/9 语种/百万样本 | draft(待签) |\n| [Open Medical-LLM Leaderboard](entries/open-medical-llm-leaderboard.md) | 开源医疗 QA | HF 社区榜,**已饱和** | draft(待签) |\n| [RCQ](entries/rcq.md) | 真实临床提问(n=100,盲评) | Nature Medicine,NYU+UT Austin | draft(待签) |\n| [MEDIQA-Chat](entries/mediqa-chat.md) | 医患对话→病历摘要(shared task) | ACL ClinicalNLP 2023,Microsoft | draft(待签) |\n| [MedQA](entries/medqa.md) | 医疗 QA 原始数据集(USMLE 式) | Jin et al. 2020,MIT,**已饱和** | draft(待签) |\n| [Leaderboard of Leaderboards](entries/leaderboard-of-leaderboards.md) | 榜中榜聚合器(meta) | HF MAYA-AI / OpenEvals | draft(待签) |\n| [LMArena](entries/lmarena.md) | 通用对话·人类盲投 Elo(**非医疗锚点**) | Berkeley/LMSYS,arXiv 2403.04132 | draft(待签) |\n| [MMMU](entries/mmmu.md) | 多模态推理(**非医疗·多模态锚点**) | Yue et al. CVPR 2024,11.5K 题 | draft(待签) |\n| [OmniBench](entries/omnibench.md) | 图+音+文 三模态(omni) | M-A-P,arXiv 2409.15272 | draft(待签) |\n| [Video-MME](entries/video-mme.md) | 视频理解(长达 1 小时) | MME-Benchmarks,CVPR 2025 | draft(待签) |\n| [VoiceBench](entries/voicebench.md) | 语音助手(口语指令+安全) | NUS,arXiv 2410.17196 | draft(待签) |\n\n\u003e 还可一键起草:EQ-Bench(情商)、MTEB(向量检索)、Open LLM Leaderboard;及 Qwen3-Omni 报告里的 MathVista / DocVQA / GPQA 等。\n\u003e\n\u003e 当前 14 条中 **6 条非医疗**,模态覆盖 文/图/音/视频。\"用 tech report 当发现源\"见 [docs/SOURCING.md](docs/SOURCING.md)。\n\n### 怎么找榜单:按\"体裁 × 来源\"普查,不靠名字搜\n\n每个条目带一个 `genre` 字段——`online-leaderboard` / `shared-task` / `paper-bound` / `aggregator` / `dataset`。这是为了**让整类缺口可见**:按名字搜只能找到你叫得出、且在网络搜索里靠前的榜,会**系统性漏掉** workshop shared task(如 MEDIQA-Chat)、论文内置榜(如 RCQ)、区域榜。检索方法与已知缺口见 **[docs/SOURCING.md](docs/SOURCING.md)**。\n\n## 人在环中:草稿 → 签字 → 发布\n\n整个流程把**人和 AI 的笔分开**(详见 [CLAUDE.md](CLAUDE.md) 的\"Integrity rules\"):\n\n1. **草稿**:AI 把 `## Agent summary`(纯事实)写好,`## Expert verdict` 留空。\n2. **签字**:**专家在自己的编辑器里**写 `## Expert verdict` 判语,并填 `expert_verdict` frontmatter(`signed_by` / `signed_date` / `confidence` / `one_liner`)。\n   - 这一步不是工具调用,所以 `hooks/guard_verdict.py` 不会拦你;但它会**拦住 AI** 写进判语区。\n3. **发布**:`python3 bin/check_publish.py` 只放行**已签字**的条目进 GitHub 页面(\"草稿可提交,发布需签字\")。\n\n```bash\npython3 bin/check_publish.py            # 发布门禁:未签字 = 拦截\n```\n\n## 数据契约与项目结构\n\n数据优先:**schema 就是契约**,渲染与工具都薄。\n\n```\nSKILL.md                   技能清单(本 repo 即 Skill;触发后教 agent 走整套工作流)\nschema/entry.schema.json   一个榜单条目的契约(popularity / authority / models_ranked / expert_verdict / moa)\nentries/\u003cid\u003e.md            一个榜单 = frontmatter + 「## Agent summary」+「## Expert verdict」\nentries/_TEMPLATE.md       起草模板\nbin/check_publish.py       发布门禁(--schema-only 为 CI 硬门禁)\nbin/render_site.py         静态站点渲染(只渲染已签字条目)\nhooks/guard_verdict.py     PreToolUse:禁止 AI 代写专家判语\nhooks/validate_entry.py    PostToolUse:写入即按 schema 校验(草稿放行)\n.github/workflows/pages.yml CI:schema 门禁 → 渲染 → 部署 GitHub Pages\ndocs/SOURCING.md           按\"体裁×来源\"普查的发现法\nCLAUDE.md                  决策、完整性规则、语言约定、hook 与门禁说明\n```\n\n依赖:`pip install pyyaml jsonschema`(仅用于校验/门禁;无运行时框架)。\n\n## 作为 Skill 使用\n\n本 repo 即一个 [Claude Code Skill](SKILL.md)(开放 `SKILL.md` 格式)。安装:`ln -s \"$PWD\" ~/.claude/skills/agentbench`。触发后,它教 agent 走完整工作流:**发现(按体裁×来源普查)→ 起草(先核实再写,判语留空)→ 校验 → 交专家签字 → 渲染/部署**,并强制\"AI 写事实、专家签判语\"的边界。已上线示例:\u003chttps://chenhaodev.github.io/agentbench/\u003e。\n\n## 设计取舍\n\n- **CLI/docs-as-data,不做 TUI**:Skill 的交互界面是 agent 对话,TUI agent 驱动不了。若日后需要高频比对式人工评审,再在同一份数据上加一个轻量 TUI 视图。\n- **无自建 CLI 编排**:本仓库只渲染**策展文本**(不像 `med-agent-verifier` 要跑评测),故首版用现成静态站点生成器 + CI 校验,不造 CLI。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenhaodev%2Fagentbench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchenhaodev%2Fagentbench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenhaodev%2Fagentbench/lists"}