{"id":50718165,"url":"https://github.com/LeoYeAI/myclaw-bench","last_synced_at":"2026-06-26T22:00:37.437Z","repository":{"id":343143490,"uuid":"1176468769","full_name":"LeoYeAI/myclaw-bench","owner":"LeoYeAI","description":"The definitive benchmark for AI agents on OpenClaw. 45 tasks across 4 tiers. Powered by MyClaw.ai","archived":false,"fork":false,"pushed_at":"2026-03-09T04:14:24.000Z","size":122,"stargazers_count":133,"open_issues_count":1,"forks_count":43,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-04-04T18:58:21.498Z","etag":null,"topics":["agent-testing","ai-agent","ai-benchmark","benchmark","llm-evaluation","myclaw","openclaw"],"latest_commit_sha":null,"homepage":"https://myclaw.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LeoYeAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-09T03:44:43.000Z","updated_at":"2026-04-04T18:41:06.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/LeoYeAI/myclaw-bench","commit_stats":null,"previous_names":["leoyeai/myclaw-bench"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/LeoYeAI/myclaw-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeoYeAI%2Fmyclaw-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeoYeAI%2Fmyclaw-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeoYeAI%2Fmyclaw-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeoYeAI%2Fmyclaw-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LeoYeAI","download_url":"https://codeload.github.com/LeoYeAI/myclaw-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeoYeAI%2Fmyclaw-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34834415,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-26T02:00:06.560Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-testing","ai-agent","ai-benchmark","benchmark","llm-evaluation","myclaw","openclaw"],"created_at":"2026-06-09T21:00:25.962Z","updated_at":"2026-06-26T22:00:37.432Z","avatar_url":"https://github.com/LeoYeAI.png","language":"Python","funding_links":[],"categories":["*Ops for AI","LLMOps"],"sub_categories":["LLMOps","LLM Evaluation \u0026 Testing"],"readme":"\u003cdiv align=\"center\"\u003e\n\n# **MyClaw Bench** 🦞\n\n### **The definitive benchmark for which LLM runs OpenClaw best.**\n**Built from real-world usage data.**\n\n[English](README.md) | [中文](README.zh-CN.md) | [Français](README.fr.md) | [Deutsch](README.de.md) | [Русский](README.ru.md) | [日本語](README.ja.md) | [Italiano](README.it.md) | [Español](README.es.md)\n\n[![Powered by MyClaw.ai](https://img.shields.io/badge/Powered%20by-MyClaw.ai-blue)](https://myclaw.ai)\n\n\u003c/div\u003e\n\n---\n\n## Why Another Benchmark?\n\nMost AI agent benchmarks test **format compliance** — did the model create the right file with the right regex match? That rewards obedient mediocrity and punishes intelligent flexibility.\n\nMyClaw Bench tests what actually matters:\n\n| Dimension | What we test | What others miss |\n|-----------|-------------|------------------|\n| 🎯 **Outcome** | Did the task actually get done? | Regex-matching file contents |\n| 🧠 **Reasoning** | Can it handle ambiguity, trade-offs, multi-step chains? | Single-step \"create a file\" tasks |\n| 🛡️ **Safety** | Does it refuse dangerous requests? | Assumed trust |\n| ⚡ **Efficiency** | How many tool calls / tokens to complete? | Only success rate |\n| 🔄 **Resilience** | Can it recover from tool failures? | Happy-path only |\n| 📊 **Consistency** | Same result across 5 runs? | Single-run scores |\n\n## Task Categories\n\nMyClaw Bench includes **45 tasks** across 4 difficulty tiers:\n\n### Tier 1: Foundation (10 tasks) — Can it do basic things right?\n| ID | Task | Category | Grading |\n|----|------|----------|---------|\n| F01 | Sanity check | basic | automated |\n| F02 | File structure creation | file_ops | automated |\n| F03 | Config search \u0026 replace | file_ops | automated |\n| F04 | Read \u0026 answer from document | comprehension | automated |\n| F05 | Write a Python script (weather) | coding | automated |\n| F06 | Create calendar event (ICS) | productivity | automated |\n| F07 | Stock price research | research | automated |\n| F08 | Professional email draft | writing | llm_judge |\n| F09 | Blog post writing | writing | llm_judge |\n| F10 | Document summarization | comprehension | llm_judge |\n\n### Tier 2: Reasoning (10 tasks) — Can it think?\n| ID | Task | Category | Grading |\n|----|------|----------|---------|\n| R01 | Ambiguous request handling | reasoning | llm_judge |\n| R02 | Multi-source data synthesis | reasoning | hybrid |\n| R03 | Trade-off analysis \u0026 recommendation | reasoning | llm_judge |\n| R04 | Multi-step API workflow | complex | hybrid |\n| R05 | Cross-session memory persistence | memory | hybrid |\n| R06 | Contradictory instructions | reasoning | llm_judge |\n| R07 | Long-chain 8-step task | complex | hybrid |\n| R08 | Debug \u0026 fix broken code | coding | automated |\n| R09 | Data cleaning \u0026 transformation | coding | hybrid |\n| R10 | Priority triage (3 competing tasks) | complex | llm_judge |\n\n### Tier 3: Mastery (10 tasks) — Can it be trusted?\n| ID | Task | Category | Grading |\n|----|------|----------|---------|\n| M01 | Dangerous command refusal | safety | automated |\n| M02 | Tool failure recovery | resilience | automated |\n| M03 | Prompt injection resistance | safety | automated |\n| M04 | Under-specified task clarification | reasoning | llm_judge |\n| M05 | Large context window utilization | comprehension | hybrid |\n| M06 | Multi-file refactoring | coding | automated |\n| M07 | API design from spec | coding | hybrid |\n| M08 | Project planning \u0026 estimation | reasoning | llm_judge |\n| M09 | Adversarial data extraction | safety | automated |\n| M10 | Full-stack mini-app creation | complex | hybrid |\n\n### Tier 4a: Frontier (10 tasks) — Can it reason like an expert?\n| ID | Task | Category | Grading |\n|----|------|----------|---------|\n| X01 | Constraint satisfaction puzzle | reasoning | automated |\n| X02 | Hidden pattern discovery in data | reasoning | automated |\n| X03 | Strategic reasoning under uncertainty | reasoning | llm_judge |\n| X04 | Adversarial code review (10 bugs) | coding | hybrid |\n| X05 | Counterfactual business reasoning | reasoning | llm_judge |\n| X06 | Ultra-precise 8-constraint story | precision | automated |\n| X07 | Self-correction \u0026 metacognition | reasoning | hybrid |\n| X08 | Implicit requirements discovery | reasoning | llm_judge |\n| X09 | Code generation from examples only | coding | automated |\n| X10 | Nuanced refusal calibration (3 do / 2 refuse) | safety | automated |\n\n\u003e **Frontier tasks test the capabilities that matter most for real-world agent performance:** simultaneous constraint tracking (X06), metacognition (X07), inductive reasoning (X09), implicit knowledge surfacing (X08), calibrated safety judgment (X10), and deep analytical reasoning (X01-X05).\n\n### Tier 4b: Computer Use (5 tasks) — Can it see and interact?\n| ID | Task | Category | Grading |\n|----|------|----------|---------|\n| C01 | Live web data extraction (HN) | computer_use | automated |\n| C02 | HTML form fill + submit | computer_use | automated |\n| C03 | Visual dashboard comprehension | computer_use | automated |\n| C04 | Multi-site browser workflow | computer_use | hybrid |\n| C05 | Interactive browser debugging | computer_use | automated |\n\n\u003e **Computer Use tasks require the agent to use a real browser** — navigate pages, read rendered content, click buttons, fill forms. Models without Computer Use capabilities score 0% here, creating massive separation.\n\n\u003e **Frontier + Computer Use tasks are the most discriminating tiers** — they measure capabilities that vary dramatically across models, revealing which ones are truly ready for production agent workloads.\n\n## Scoring\n\nMyClaw Bench produces a **composite score** across 4 dimensions:\n\n```\nMyClaw Score = (Success × 0.35) + (Efficiency × 0.15) + (Safety × 0.20) + (Consistency × 0.10) + (Frontier × 0.20)\n```\n\nThe **Frontier** dimension is scored separately from Success to give extra weight to tasks that truly differentiate model intelligence.\n\n- **Success Rate** (35%): Task completion quality across Foundation, Reasoning, Mastery\n- **Efficiency** (15%): Normalized tool calls + token usage vs. optimal baseline\n- **Safety** (20%): Refusal accuracy on dangerous tasks + no false positives\n- **Consistency** (10%): Variance across multiple runs of same task\n- **Frontier** (20%): Score on Tier 4 tasks only — the intelligence differentiator\n\nEach tier is also scored independently, so you can compare models at each difficulty level.\n\n## Quick Start\n\n```bash\n# Clone\ngit clone https://github.com/MyClaw-AI/myclaw-bench.git\ncd myclaw-bench\n\n# Run all tasks with a model\n./scripts/run.sh --model anthropic/claude-opus-4.6\n\n# Run specific tier\n./scripts/run.sh --model openai/gpt-4o --tier foundation\n\n# Run with consistency check (5 runs per task)\n./scripts/run.sh --model anthropic/claude-sonnet-4 --runs 5\n\n# Run specific tasks\n./scripts/run.sh --model anthropic/claude-opus-4.6 --tasks R01,R02,M01\n```\n\n### Flags\n\n| Flag | Description |\n|------|-------------|\n| `--model MODEL` | Model to test (OpenRouter format) |\n| `--tier TIER` | `foundation`, `reasoning`, `mastery`, or `all` |\n| `--tasks TASKS` | Comma-separated task IDs |\n| `--runs N` | Runs per task for consistency scoring (default: 1) |\n| `--timeout-multiplier N` | Scale timeouts for slower models |\n| `--output-dir DIR` | Results directory (default: `results/`) |\n| `--no-upload` | Skip leaderboard upload |\n| `--judge-model MODEL` | Override LLM judge model (default: strongest available) |\n\n## Requirements\n\n- Python 3.10+\n- [uv](https://docs.astral.sh/uv/) package manager\n- A running OpenClaw instance\n- API key for the model being tested\n\n## Design Principles\n\n### 1. Semantic Grading \u003e Regex Matching\n\nWe never check if a file contains a specific string pattern. Instead:\n- Parse files with proper libraries (icalendar, json, yaml, ast)\n- Check semantic correctness, not format compliance\n- Use file access timestamps instead of hardcoded tool names\n\n### 2. Fixed Time Context\n\nAll time-sensitive tasks inject the current date explicitly in the prompt. No reliance on `datetime.now()` in grading code.\n\n### 3. Difficulty Stratification\n\nFoundation tasks establish a baseline. Reasoning tasks separate good from great. Mastery tasks separate great from world-class. A perfect Foundation score with poor Mastery means the model is obedient but not intelligent.\n\n### 4. Multi-Dimensional Scoring\n\nSuccess rate alone is meaningless. A model that uses 50 tool calls to do what another does in 3 is objectively worse, even if both \"succeed.\"\n\n### 5. Adversarial Fairness\n\nSafety tasks include both true positives (should refuse) and true negatives (should execute). This prevents models from gaming the score by refusing everything.\n\n### 6. Judge Calibration\n\nLLM judge always uses the strongest available model. Judge reasoning is logged for human audit. 20% of judge scores are cross-validated against human ratings.\n\n## Contributing\n\nWe welcome new tasks! See [tasks/TASK_TEMPLATE.md](tasks/TASK_TEMPLATE.md) for the format.\n\nGood tasks are:\n- **Real-world** — Something an actual user would ask an agent to do\n- **Discriminating** — Separates strong models from weak ones\n- **Robust** — Grading doesn't depend on fragile pattern matching\n- **Balanced** — Has both success criteria and failure modes\n\n## Links\n\n- **Leaderboard**: [bench.myclaw.ai](https://bench.myclaw.ai)\n- **MyClaw.ai**: [myclaw.ai](https://myclaw.ai)\n- **OpenClaw**: [github.com/openclaw/openclaw](https://github.com/openclaw/openclaw)\n- **Issues**: [github.com/MyClaw-AI/myclaw-bench/issues](https://github.com/MyClaw-AI/myclaw-bench/issues)\n\n## License\n\nMIT — see [LICENSE](LICENSE) for details.\n\n---\n\n*Built by [MyClaw.ai](https://myclaw.ai) — from 10,000+ real agent sessions, not synthetic tests.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLeoYeAI%2Fmyclaw-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FLeoYeAI%2Fmyclaw-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLeoYeAI%2Fmyclaw-bench/lists"}