{"id":51475270,"url":"https://github.com/devswha/claw-bench","last_synced_at":"2026-07-06T20:03:17.856Z","repository":{"id":348429612,"uuid":"1198048563","full_name":"devswha/claw-bench","owner":"devswha","description":"Benchmark suite: Claw Code (Rust) vs Claude Code (Node.js)","archived":false,"fork":false,"pushed_at":"2026-04-13T07:12:26.000Z","size":155,"stargazers_count":20,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-13T09:52:53.119Z","etag":null,"topics":["ai-tools","benchmark","cli","developer-tools","nodejs","performance","rust"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devswha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-01T04:42:28.000Z","updated_at":"2026-04-13T07:12:29.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/devswha/claw-bench","commit_stats":null,"previous_names":["devswha/claw-bench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/devswha/claw-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devswha%2Fclaw-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devswha%2Fclaw-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devswha%2Fclaw-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devswha%2Fclaw-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devswha","download_url":"https://codeload.github.com/devswha/claw-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devswha%2Fclaw-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35204422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-06T02:00:07.184Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-tools","benchmark","cli","developer-tools","nodejs","performance","rust"],"created_at":"2026-07-06T20:03:16.461Z","updated_at":"2026-07-06T20:03:17.822Z","avatar_url":"https://github.com/devswha.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# claw-bench\n\n**Runtime overhead benchmarks for AI coding CLIs — Rust vs Node.js**\n\n\u003e **Note:** Task-effectiveness benchmarking (SWE-bench, practical coding tasks, token tracking) has moved to **[OpenBench](https://github.com/devswha/OpenBench)** — an open benchmarking framework that compares coding-agent CLIs (Claude Code, Codex, and their plugin wrappers) with same tasks, same environment, and comparable results. claw-bench remains available for runtime overhead measurements (startup, memory, binary size).\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![CI](https://github.com/devswha/claw-bench/actions/workflows/benchmark.yml/badge.svg)](https://github.com/devswha/claw-bench/actions/workflows/benchmark.yml)\n[![Benchmark](https://img.shields.io/badge/benchmark-stable_suite-E8590C)](bench-all.sh)\n\n\u003cbr/\u003e\n\n\u003cimg src=\"assets/benchmark-chart.svg\" alt=\"Claw Code Performance Benchmarks\" width=\"820\"/\u003e\n\n\u003cbr/\u003e\n\n\u003e Measured on Ubuntu 24.04 · [Run your own benchmarks](#quick-start) · Inspired by [claw-code](https://github.com/devswha/claw-code)\n\n\u003c/div\u003e\n\n---\n\nMinimal benchmark suite comparing **Claw Code** (Rust single-binary CLI) vs **Claude Code** (Node.js CLI), with **optional Codex runtime comparison** for the stable local-only benchmarks.\n\n## Stable Default Suite\n\n`./bench-all.sh` intentionally runs only the **stable local runtime core**:\n- startup time\n- install footprint\n- idle memory\n\nEverything else in this repo is **manual / experimental** and should be treated as exploratory rather than headline-quality benchmarking.\n\nDefault stable-suite behavior remains **human-readable terminal output**. If you want machine-readable artifacts, the three stable benchmark scripts support an optional `--json` mode that writes one JSON file under `results/tier0/`, and `./bench-all.sh --json` writes a timestamped Tier 0 directory containing one per-benchmark JSON file plus `manifest.json`. The three per-benchmark Tier 0 JSON files share one envelope: `schema_version`, `suite`, `tier`, `benchmark`, `script`, `run_id`, `generated_at`, `environment`, `tools`, and `results`; `manifest.json` is the companion index for the suite run.\n\n## Sample Results\n\nMeasured on Ubuntu 24.04 (Linux 6.8), same machine, same API endpoint.\n\n| Benchmark | Claw (Rust) | Claude (Node.js) | Ratio |\n|-----------|-------------|-------------------|-------|\n| Startup time | **1.2 ms** | 86.4 ms | 73.2x faster |\n| Binary size | **13 MB** | 218 MB | 17x smaller |\n| Memory (idle) | **4.1 MB** | 191.5 MB | 46.7x less |\n\n\u003e Results vary by machine, network, and API provider. Run your own benchmarks.\n\n### Latest Local Validation Snapshot — 2026-04-04\n\nThe following values come from the most recent local verification run on this machine. Treat them as a **manual snapshot**, not a headline-quality published benchmark:\n\n| Benchmark | Claw | Claude | Codex |\n|-----------|------|--------|-------|\n| Startup time (`--version`) | **1.2 ms** | 445.7 ms | 37.7 ms |\n| Total install footprint | **13 MB** | 13 MB* | 163 MB |\n| Idle memory (`--version`) | **4.2 MB** | 166.3 MB | 46.0 MB |\n\n\\* Claude's install footprint resolved as **binary-only** on this machine because `node_modules` was not found adjacent to the CLI binary during the check.\n\n### Latest Tier 1 Practical Preview — 2026-04-04\n\nThis is the current one-task bootstrap slice from `experimental/bench-practical.sh` (`01-add-function`):\n\n| Tool | Status | Duration | Diff lines |\n|------|--------|----------|------------|\n| Claw | failed | 46 ms | 0 |\n| Claude | failed (timeout) | 180510 ms | 0 |\n| Codex | **passed** | 22491 ms | 3 |\n\nThis Tier 1 result is still **manual / experimental** and should be read as a harness-validation snapshot, not a broad capability claim.\n\n### Optional Codex Runtime Comparison\n\nIf `CODEX_BIN` is configured and `codex` is already authenticated locally, the stable core scripts (`startup`, `size`, `memory`) will include **Codex CLI** automatically.\n\nCodex is currently treated as a **runtime-only comparison target** in this repo:\n- included automatically in the lightweight runtime benchmarks\n- **not** yet wired into the long-running task-effectiveness harnesses\n- uses its own local Codex auth/session, not `ANTHROPIC_API_KEY`\n\n| Benchmark | Claw (Rust) | Codex CLI | Ratio |\n|-----------|-------------|-----------|-------|\n| Startup time | **1.2 ms** | 34.5 ms | 29.2x faster |\n| Memory (idle) | **4.1 MB** | 46.0 MB | 11.2x less |\n\n## Experimental Scripts\n\nThese scripts are still in the repo, but are **not** part of the default suite because they are much more environment-sensitive:\n\n- `experimental/bench-ttft.sh`\n- `experimental/bench-session.sh`\n- `experimental/bench-syscall.sh`\n- `experimental/bench-cpu.sh`\n- `experimental/bench-io.sh`\n- `experimental/bench-threads.sh`\n- `experimental/bench-gc.sh`\n- `experimental/bench-practical.sh`\n- `experimental/bench-swebench.sh`\n- `experimental/bench-terminal.sh`\n- `experimental/bench-polyglot.sh`\n\nUse them manually only if you are comfortable with:\n- API/provider variability\n- timeout-capped measurements\n- Docker / Python harness setup\n- Claw-first task harnesses rather than symmetric tool-vs-tool evaluation\n- manual/early Tier 1 practical-task previews rather than a full task suite\n\n## Benchmarks\n\n| Script | Measures | Tool |\n|--------|----------|------|\n| `bench-startup.sh` | Cold start time | [hyperfine](https://github.com/sharkdp/hyperfine) |\n| `bench-memory.sh` | Idle RSS memory | `/usr/bin/time -v` |\n| `bench-size.sh` | Binary and install size | `du` |\n| `bench-all.sh` | Stable runtime suite | — |\n| `experimental/bench-ttft.sh` | Time to first response | `date +%s%N` |\n| `experimental/bench-session.sh` | Memory over long session | `ps` polling |\n| `experimental/bench-syscall.sh` | Syscall count and breakdown | `strace -c` |\n| `experimental/bench-cpu.sh` | CPU cycles, IPC, cache misses | `perf stat` |\n| `experimental/bench-io.sh` | File open/read/write counts | `strace -e trace=` |\n| `experimental/bench-threads.sh` | Thread/process footprint | `/proc/pid/task` |\n| `experimental/bench-gc.sh` | Page faults, RSS growth | `perf stat` + `/proc` |\n| `experimental/bench-practical.sh` | Tier 1 practical coding task preview | temp workspace + verifier |\n| `experimental/bench-swebench.sh` | SWE-bench Verified score | Docker + Python |\n| `experimental/bench-terminal.sh` | Terminal-Bench 2.0 score | Docker + Harbor |\n| `experimental/bench-polyglot.sh` | Aider Polyglot score | Python + git |\n\n## Prerequisites\n\n```bash\n# Stable default suite\nsudo apt install hyperfine bc\n\n# Experimental profiling scripts\nsudo apt install strace linux-tools-common linux-tools-$(uname -r)\n\n# perf may require relaxing paranoid mode:\necho 0 | sudo tee /proc/sys/kernel/perf_event_paranoid\n```\n\n### Experimental task-harness prerequisites\n\n```bash\n# Docker (required for SWE-bench and Terminal-Bench)\n# See: https://docs.docker.com/get-docker/\n\n# Python 3.10+ with venv\nsudo apt install python3-venv\n\n# Harnesses are auto-installed on first run\n# Disk space: ~50GB for SWE-bench, ~20GB for Terminal-Bench\n```\n\n## Quick Start\n\n```bash\ngit clone https://github.com/devswha/claw-bench.git\ncd claw-bench\n\n# Configure paths and API key\ncp env.example.sh env.sh\nvi env.sh\n\n# Run the stable suite\n./bench-all.sh\n\n# Or run individual stable scripts\n./bench-startup.sh\n./bench-size.sh\n./bench-memory.sh\n\n# Optional: persist one machine-readable Tier 0 result file\n./bench-startup.sh --json\n\n# Optional: persist machine-readable Tier 0 results for the full stable suite\n./bench-all.sh --json\n\n# Manual / experimental: run the Tier 1 practical bootstrap slice\n./experimental/bench-practical.sh --json\n```\n\n## Configuration\n\nEdit `env.sh` (never committed — gitignored):\n\n```bash\nCLAW_BIN=\"$HOME/workspace/claw-code/rust/target/release/claw\"\nCLAUDE_BIN=\"$(which claude)\"\nCODEX_BIN=\"$(which codex)\"\nAPI_KEY=\"your-anthropic-api-key\"\n```\n\nAn **Anthropic API key** is only needed for the experimental API-path / task harness scripts. Get one at [console.anthropic.com](https://console.anthropic.com/).\n\nCodex does **not** use `ANTHROPIC_API_KEY` in this repo; it relies on your existing `codex login` session.\n\n## Security\n\n- `env.sh` is gitignored and never committed\n- API keys are exported in subshells only — automatically cleaned up on exit\n- Binary paths are validated before execution\n- Stable runtime benchmarks stay terminal-only by default, but `--json` writes timestamped Tier 0 artifacts under `results/tier0/`\n- `bench-all.sh --json` emits `startup-time.json`, `install-size.json`, `idle-memory.json`, and `manifest.json`\n- Tier 1 practical-task previews are manual/experimental and write JSON under `results/tier1/`\n- Task-effectiveness harnesses clone repos, create virtualenvs, and write timestamped results under `swebench/results/`, `terminal-bench/results/`, and `polyglot/results/`\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevswha%2Fclaw-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevswha%2Fclaw-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevswha%2Fclaw-bench/lists"}