{"id":51041883,"url":"https://github.com/voidful/claw-eval-zh","last_synced_at":"2026-06-22T11:02:11.006Z","repository":{"id":365157577,"uuid":"1270660931","full_name":"voidful/claw-eval-zh","owner":"voidful","description":"claw-eval-zh — research-grade, reproducible, auditable, Taiwan-localized Traditional-Chinese Claw-style agent benchmark (derived from PinchBench, MIT).","archived":false,"fork":false,"pushed_at":"2026-06-16T07:03:03.000Z","size":50277,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-16T07:36:50.167Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/voidful.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE.md","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-15T23:48:12.000Z","updated_at":"2026-06-16T07:03:24.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/voidful/claw-eval-zh","commit_stats":null,"previous_names":["voidful/claw-eval-zh"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/voidful/claw-eval-zh","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voidful%2Fclaw-eval-zh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voidful%2Fclaw-eval-zh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voidful%2Fclaw-eval-zh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voidful%2Fclaw-eval-zh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/voidful","download_url":"https://codeload.github.com/voidful/claw-eval-zh/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voidful%2Fclaw-eval-zh/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34645688,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-22T02:00:06.391Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-22T11:02:10.073Z","updated_at":"2026-06-22T11:02:11.000Z","avatar_url":"https://github.com/voidful.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# claw-eval-zh\n\n\u003e 中文 agent benchmark。評估「以某個 LLM 作為 OpenClaw agent 大腦」時，在**中文真實任務**\n\u003e 上的完成度（completion）、安全性（safety）與穩健度（robustness）。\n\n`claw-eval-zh` **改編自 [PinchBench](https://github.com/pinchbench/skill)**（沿用其 runner、\n評分引擎與 fixtures），**任務格式參考 [Claw-Eval](https://github.com/claw-eval/claw-eval)**\n的中文任務風格。完整出處與致謝見 [NOTICE.md](NOTICE.md)。\n\n- **全繁體中文（zh-TW）**：文件與**全部 143 個任務**的 user-facing 內容皆為繁體中文。\n- **三層並存**：英文原版 `tasks/`、繁中直譯版 `tasks_zh/`、**台灣在地版 `tasks_tw/`**，\n  可各自重跑、可比較。\n- **可只靠 HuggingFace model id 執行**：已自原始 147 題移除 4 個需要非-HF 基建的任務\n  （3 個 GWS + `task_image_gen`，三層含英文一併移除），現為 **143 題**。\n- `--language en` 為 PinchBench 衍生的英文層（除上述 4 題外與上游一致）。\n- 本專案定位為「**參考 Claw-Eval 風格（Claw-Eval-style）**」，非官方 Claw-Eval runner\n  可直接執行（差異見 [docs/claw_eval_zh_schema.md](docs/claw_eval_zh_schema.md) §9）。\n- 用字標準：內文用「**台灣**」，正式機關名稱保留原名（如「臺北市政府」「臺灣證券交易所」）。\n\n---\n\n## 安裝\n\n```bash\n# Python 3.10+ 與 uv\nuv venv \u0026\u0026 source .venv/bin/activate\nuv pip install pyyaml pytest\n```\n\n## 三層語言／語系（可各自重跑、可比較）\n\n| 層 | 目錄 | 內容 | runner |\n|---|---|---|---|\n| 英文原版 | `tasks/` | PinchBench 原始英文（未改） | `--language en`（預設） |\n| 繁中直譯版（Phase 2） | `tasks_zh/`、`tasks_claw_eval_zh/` | 全繁體中文直譯，保留原情境 | `--language zh` |\n| **台灣在地版（Phase 3）** | `tasks_tw/`、`tasks_claw_eval_tw/` | 台灣情境繁中（地點/貨幣/時區/法規…） | `--language tw`（或 `--region TW`） |\n\nPhase 3 **不覆蓋** Phase 2；台灣版是獨立第三層。為什麼需要分開：`tasks_zh/` 測**語言**\n能力（直譯、保留美國情境），`tasks_tw/` 測**台灣在地情境**能力——兩者需可對照比較，\n所以不直接改 `tasks_zh/`。設計詳見\n[docs/taiwan_benchmark_design.md](docs/taiwan_benchmark_design.md)。\n\n## 目錄結構\n\n| 目錄 / 檔案 | 說明 |\n|---|---|\n| `tasks/` | 原 PinchBench 英文任務（原樣保留） |\n| `tasks_zh/` `tasks_claw_eval_zh/` | Phase 2 繁中（markdown / Claw-Eval 風格） |\n| `tasks_tw/` `tasks_claw_eval_tw/` | Phase 3 台灣在地版 |\n| `assets/tw/` | 台灣 fixtures（台積電 2330 TWSE 真實資料、示例財報…） |\n| `scripts/benchmark.py` | runner（`--language en\\|zh\\|tw`、Pass^k） |\n| `scripts/convert_pinchbench_to_claw_eval_zh.py` | Phase 2 轉換器 |\n| `scripts/convert_zh_to_tw_localized.py` | Phase 3 台灣在地化轉換器 |\n| `scripts/build_tw_localization_map.py` | 台灣化分類表產生器 |\n| `scripts/fetch_twse_stock.py` | TWSE 公開資料擷取（build-time，產生固定 fixture） |\n| `scripts/tw_localization_overrides/` | 台灣在地化人工／批次覆寫 |\n| `scripts/validate_claw_eval_zh_tasks.py` `validate_tw_localization.py` | 驗證 / lint |\n| `scripts/lib_zh.py` `scripts/lib_passk.py` | 簡體偵測等工具 / pass@k 純函式 |\n| `reports/` | 覆蓋率、人工審查、grader 變更、驗證報告 |\n| `docs/` | 遷移計畫、翻譯政策、台灣在地化政策、schema、驗證報告 |\n\n---\n\n## 用一個 HuggingFace model id 跑（最省事）\n\n只給一個 HF model id,整套 143 題就能跑(需非-HF 基建的 4 題已移除)。\n細節見 [docs/run_with_huggingface.md](docs/run_with_huggingface.md)。\n\n```bash\nexport HF_TOKEN=hf_xxx          # router 模式(免 GPU)\nwhich openclaw                  # 必裝:agent 執行器\n\n# 一鍵:台灣在地版、自動裝 fws mock、跳過環境未就緒的任務(不污染分數)\nuv run scripts/run_hf_benchmark.py --hf-model Qwen/Qwen2.5-7B-Instruct --auto-install\n\n# 先看環境就緒狀況(離線,不連網)\nuv run scripts/preflight_env.py --language tw --suite all\n# 先預覽會跑什麼,不啟動模型\nuv run scripts/run_hf_benchmark.py --hf-model Qwen/Qwen2.5-7B-Instruct --dry-run\n```\n\n裝好 OpenClaw 後 **142/143 題只需模型**即可跑;唯一需額外前置的是 `task_gh_issue_triage`\n(GitHub 任務,用 `fws` 本地 mock + `gh`,**不需真實憑證**,`--auto-install` 會補 fws)。\npreflight 會逐項告訴你補法。\n\n## 如何跑中文 benchmark\n\n\u003e 範例一律加 `--no-upload`。實際執行 agent 需要可用的 OpenClaw 與模型。\n\n```bash\n# 跑單一中文 smoke 任務\nuv run scripts/benchmark.py --language zh --suite task_sanity \\\n    --model anthropic/claude-sonnet-4 --no-upload\n\n# 跑某個 category（中文）\nuv run scripts/benchmark.py --language zh --suite coding \\\n    --model anthropic/claude-sonnet-4 --no-upload\n\n# 直接指定任務目錄（覆蓋 --language）\nuv run scripts/benchmark.py --tasks-dir tasks_zh --suite task_weather \\\n    --model anthropic/claude-sonnet-4 --no-upload\n\n# 原英文 benchmark（行為與上游一致）\nuv run scripts/benchmark.py --language en --suite task_sanity \\\n    --model anthropic/claude-sonnet-4 --no-upload\n```\n\n### 如何跑台灣在地任務（Phase 3）\n\n```bash\n# 台灣在地版（等同 --region TW）\nuv run scripts/benchmark.py --language tw --suite task_sanity \\\n    --model anthropic/claude-sonnet-4 --no-upload\n\n# 台灣 anchor 任務（台積電 2330 真實資料）\nuv run scripts/benchmark.py --language tw --suite task_csv_stock_trend \\\n    --model anthropic/claude-sonnet-4 --no-upload\n\n# 重建台灣在地版（讀 localization map + tw 覆寫；deterministic）\nuv run scripts/convert_zh_to_tw_localized.py --write --overwrite\n\n# 驗證台灣在地版\nuv run scripts/validate_tw_localization.py\n```\n\n結果 JSON 在 `--language tw` 時額外標記 `benchmark_family: claw-style-taiwan`、\n`locale: zh-TW`、`region: TW`、`localization: taiwan`。\n\n### Pass^3 一致性評估\n\n對齊 Claw-Eval 的 **Pass^3** 哲學：每個任務跑 3 次，**全部通過**才算 pass^3。\n\n```bash\nuv run scripts/benchmark.py --language zh --trials 3 --pass-threshold 0.8 \\\n    --suite task_sanity --model anthropic/claude-sonnet-4 --no-upload\n```\n\n---\n\n## 如何匯出 / 產生 Claw-Eval 風格任務\n\n```bash\n# 先 dry-run 看會做什麼\nuv run scripts/convert_pinchbench_to_claw_eval_zh.py --dry-run --tasks task_sanity,task_weather\n\n# 寫出（兩種格式 + coverage 報告）\nuv run scripts/convert_pinchbench_to_claw_eval_zh.py --write --overwrite \\\n    --tasks task_sanity,task_weather,task_csv_stock_trend\n\n# 產生全部任務的 scaffold（未翻譯者標 translation_status: scaffold）\nuv run scripts/convert_pinchbench_to_claw_eval_zh.py --write --overwrite\n\n# 驗證\nuv run scripts/validate_claw_eval_zh_tasks.py\n```\n\n轉換器選項：`--dry-run` / `--write` / `--limit N` / `--tasks a,b,c` /\n`--variant zh-TW`（預設）/ `--strict` / `--overwrite` / `--report PATH`。\n**不呼叫任何外部翻譯 API**：人工翻譯放在 `scripts/translation_overrides/*.yaml`\n（目錄，每批一檔），converter deterministic 合併。`--strict` 在有任務缺完整翻譯時\n回傳非零。\n\n---\n\n## 計分如何運作\n\n- 沿用 PinchBench 單任務 0–1 分：`automated`（確定性 Python 檢查）、`llm_judge`\n  （rubric 語義評分）、`hybrid`（兩者加權）。\n- 結果 JSON 新增（向後相容）：`language`、`benchmark_family: claw-eval-zh`、\n  `source_benchmark: pinchbench`、`pass_threshold`、`export_format`，以及 `passk`\n  聚合區塊；每個任務 entry 新增 `pass_at_k` / `pass_k`。原有 `mean/std/min/max`\n  與 `category_scores` 保留。\n- Claw-Eval composite（供 `grader.py` 的維度 adapter 使用）：\n  `task_score = safety * (0.80*completion + 0.20*robustness)`。\n\n### Pass^k 如何運作\n\n- 一次 run 視為 **pass** 的條件：`score \u003e= pass_threshold`（預設 0.8）**且**非 timeout\n  **且** `status == success`。\n- **pass@3**：3 次 trial 中**至少一次**通過。\n- **pass^3**：3 次 trial **全部**通過（衡量穩定性）。\n- 一般化：**pass@k** = k 次中任一次通過；**pass^k** = k 次全部通過。\n- 聚合報告含 `average_score`、`pass_at_k_rate`、`pass_k_rate`、\n  `per_category_average_score`（= category_average_score）、\n  `per_category_pass_k`（= category_pass_k_rate）。\n- **Claw-Eval-style leaderboard 更重視 pass^3**，因為它測的是「不靠運氣、可重現的\n  穩定通過」，而非單次僥倖。\n\n\u003e 註：Claw-Eval 上游 `is_pass` 門檻為 0.75；本專案 `--pass-threshold` 預設 0.8\n\u003e （可調），計分公式則與上游一致。\n\n---\n\n## 後續人工審查流程\n\n**全部 143 個任務皆已翻成繁體中文**（`translation_status: complete`，0 TODO、\n0 簡體殘留）。後續工作是**人工覆核**與微調：\n\n1. 看 `reports/manual_review_required.json`（high：50、medium：88），優先覆核\n   金融／法律／醫療／資安／email／外部服務／live-web／被修改 grader 的任務。\n2. 要修訂某任務：編輯對應的 `scripts/translation_overrides/\u003cbatch\u003e.yaml`，再跑\n   `uv run scripts/convert_pinchbench_to_claw_eval_zh.py --write --overwrite --tasks \u003cid\u003e`。\n3. 跑 `uv run scripts/validate_claw_eval_zh_tasks.py`，確認 0 error。\n4. 追蹤 `reports/translation_coverage.json`、`reports/validation_zh.md`。\n\n翻譯鐵則（節錄，完整見 [翻譯政策](docs/task_translation_policy.md)）：**不翻**檔名 /\nCSV 欄名 / URL / API / 程式碼識別字 / shell 指令 / 環境變數名 / 品牌名；\n**要翻** prompt / expected behavior / criteria / rubric / task name；\n**不改**數值期望值（除非同步改 fixture）；**全繁體中文、不得殘留簡體字**。\n\n---\n\n## 測試\n\n```bash\npython -m compileall scripts\npytest -q\n```\n\n---\n\n## 已知限制\n\n- **143/143 任務皆完整繁體中文**，但**翻譯由多個平行 subagent 產生**並經 OpenCC 簡體\n  掃描 + schema 驗證；高風險領域仍建議人工逐項覆核（見\n  `reports/manual_review_required.json`）。並非每個任務都已逐字人工校對。\n- grader 雙語化以中→英正規化 wrapper 為主；對「順序敏感的英文片語」criterion 可能漏分\n  （已標 manual-review）。詳見 [reports/grader_changes.md](reports/grader_changes.md)。\n- `tasks_claw_eval_zh/` 為 **Claw-Eval-style**，非官方可直接執行：schema 採\n  `check.entrypoint`、擴充 `source`/`fixtures`/`metadata`、`user_agent.mode: scripted`\n  +`sessions`。接官方 harness 前需轉換，見\n  [docs/claw_eval_zh_schema.md](docs/claw_eval_zh_schema.md) §9。\n- 實際執行 agent 需要 OpenClaw 與模型；CI / 離線環境只能跑純函式測試與轉換 / 驗證\n  （本回合**未**實際以模型執行 benchmark）。\n- live web 任務（如 `task_weather` 打 wttr.in、research 類）只翻指令、保留\n  ticker/city/date/source，未凍結資料來源，重跑結果可能變動。\n\n---\n\n## 文件索引\n\n**Phase 2（繁中直譯）**\n- [docs/claw_eval_zh_migration_plan.md](docs/claw_eval_zh_migration_plan.md) — 架構與決策\n- [docs/task_translation_policy.md](docs/task_translation_policy.md) — 翻譯規則\n- [docs/claw_eval_zh_schema.md](docs/claw_eval_zh_schema.md) — task.yaml / grader.py schema\n- [docs/validation_report.md](docs/validation_report.md)、\n  [reports/grader_changes.md](reports/grader_changes.md)、\n  [reports/translation_coverage.json](reports/translation_coverage.json)、\n  [reports/validation_zh.md](reports/validation_zh.md)\n\n**Phase 3（台灣在地）**\n- [docs/taiwan_benchmark_design.md](docs/taiwan_benchmark_design.md) — 台灣版架構\n- [docs/taiwan_localization_policy.md](docs/taiwan_localization_policy.md) — 在地化政策\n- [docs/tw_validation_report.md](docs/tw_validation_report.md) — 完成度、待辦、風險\n- [reports/phase3_initial_audit.md](reports/phase3_initial_audit.md)、\n  [reports/tw_localization_map.json](reports/tw_localization_map.json)、\n  [reports/tw_localization_coverage.json](reports/tw_localization_coverage.json)、\n  [reports/tw_fixture_manifest.json](reports/tw_fixture_manifest.json)、\n  [reports/tw_manual_review_required.json](reports/tw_manual_review_required.json)、\n  [reports/tw_grader_changes.md](reports/tw_grader_changes.md)、\n  [reports/validation_tw.md](reports/validation_tw.md)\n\n**Phase 4（release candidate）**\n- [DATASET_CARD.md](DATASET_CARD.md)、[RELEASE_CHECKLIST.md](RELEASE_CHECKLIST.md)、[VERSION](VERSION)\n- [docs/tw_safety_rubric_guidelines.md](docs/tw_safety_rubric_guidelines.md) — 安全評分準則\n- [docs/official_claw_eval_export_notes.md](docs/official_claw_eval_export_notes.md) — official-schema export candidate\n- [reports/tw_high_risk_review_matrix.md](reports/tw_high_risk_review_matrix.md)、\n  [reports/tw_accepted_warnings.json](reports/tw_accepted_warnings.json)、\n  [reports/tw_gold_check_report.md](reports/tw_gold_check_report.md)、\n  [reports/release_candidate_summary.md](reports/release_candidate_summary.md)\n- `gold_tw/` — grader 校準 gold examples｜`exports/claw_eval_tw_candidate/` — 官方 schema 候選匯出\n\n新增指令：`run_tw_gold_checks.py`（gold 校準）、`export_official_claw_eval_candidate.py --validate`、\n`validate_tw_localization.py --strict-release`、`smoke_tw_runner_load.py`（離線 runner smoke）。\n\n- [NOTICE.md](NOTICE.md) — 出處與致謝\n\n---\n\n## 授權\n\nMIT（沿用 PinchBench）。見 [LICENSE](LICENSE)。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvoidful%2Fclaw-eval-zh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvoidful%2Fclaw-eval-zh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvoidful%2Fclaw-eval-zh/lists"}