{"id":50953561,"url":"https://github.com/ethswarm-tools/bee-bench","last_synced_at":"2026-06-18T04:01:47.269Z","repository":{"id":355259056,"uuid":"1227386261","full_name":"ethswarm-tools/bee-bench","owner":"ethswarm-tools","description":"Cross-client benchmark suite for the three Swarm Bee API clients","archived":false,"fork":false,"pushed_at":"2026-05-02T17:55:37.000Z","size":278,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-02T18:28:03.910Z","etag":null,"topics":["bee","bee-go","bee-js","bee-rs","ethereum","swarm"],"latest_commit_sha":null,"homepage":"https://ethswarm-tools.github.io/bee-bench/results/report.html","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ethswarm-tools.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-02T16:00:18.000Z","updated_at":"2026-05-02T17:55:41.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ethswarm-tools/bee-bench","commit_stats":null,"previous_names":["ethswarm-tools/bee-bench"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ethswarm-tools/bee-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethswarm-tools%2Fbee-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethswarm-tools%2Fbee-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethswarm-tools%2Fbee-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethswarm-tools%2Fbee-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ethswarm-tools","download_url":"https://codeload.github.com/ethswarm-tools/bee-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethswarm-tools%2Fbee-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34475375,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-18T02:00:06.871Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bee","bee-go","bee-js","bee-rs","ethereum","swarm"],"created_at":"2026-06-18T04:01:46.380Z","updated_at":"2026-06-18T04:01:47.262Z","avatar_url":"https://github.com/ethswarm-tools.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bee-bench\n\n[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)\n\nCross-client benchmark suite for the three Swarm Bee API clients living as siblings:\n\n- `bee-js/` — `@ethersphere/bee-js` v12.1.0\n- `bee-go/` — `github.com/ethswarm-tools/bee-go` v1.1.0\n- `bee-rs/` — `bee-rs` v1.1.0\n\nAll three runners hit the same local Bee node, run the same case set defined in `bench-spec.json`, and emit JSON results that the aggregator turns into a markdown report and an interactive HTML chart page.\n\n**Where to read the results:**\n- **[Live HTML report](https://ethswarm-tools.github.io/bee-bench/results/report.html)** — interactive Chart.js view, served from GitHub Pages (no clone needed)\n- [`results/INDEX.md`](results/INDEX.md) — landing page, links into everything else\n- [`results/report.md`](results/report.md) — auto-generated numerical report\n- [`results/report.html`](results/report.html) — same data, interactive (raw source)\n- [`results/report.csv`](results/report.csv) — flat per-row CSV (one row per case × param × runner) for spreadsheet / pandas analysis\n- [`FINDINGS.md`](FINDINGS.md) — qualitative observations (F1–F21)\n\n\u003e ## ⚠ MB/s numbers are NOT Swarm-network throughput\n\u003e\n\u003e Every byte/sec figure produced by this bench is **client ↔ local Bee node** over loopback HTTP, NOT real Swarm-network throughput. Uploads are buffered to local Bee under deferred-upload mode (chunks pushed to peers asynchronously after the call returns); downloads of just-uploaded references hit the local Bee cache. See [FINDINGS § measurement scope](FINDINGS.md#-measurement-scope-caveat-read-first) for how to re-run with `deferred: false` for real-network numbers.\n\n## Prereqs\n\n- Bee node reachable at `BEE_URL` (default `http://localhost:1633`).\n- A usable postage batch — find one with `curl -s $BEE_URL/stamps | jq '.stamps[] | select(.usable == true) | .batchID'`.\n- Go 1.25+, Rust (stable), Node 18+.\n\n## Quick start\n\n```bash\n# 1. fixtures (1MB / 10MB / 100MB; add 1GB with BENCH_LARGE=1)\n./scripts/gen-fixtures.sh\n\n# 2. point at a usable batch\nexport BEE_BATCH_ID=\u003chex\u003e\nexport BEE_URL=http://localhost:1633\n\n# 3. run all three (sequential; rotating order)\n./scripts/run-all.sh\n\n# Optional: snapshot before another run\n./scripts/preserve-run.sh \u003clabel\u003e          # → results/_\u003clabel\u003e/\n\n# 4. aggregate (run-all.sh calls this at the end automatically)\nnode scripts/aggregate.mjs\nopen results/report.html                    # or read results/report.md\n```\n\n### Docker\n\nSelf-contained image that builds all three runners against the upstream client repos — no host toolchain required.\n\n```bash\n# CPU-only (no Bee node needed; net.* cases skip cleanly)\ndocker build -t bee-bench .\ndocker run --rm -v \"$PWD/out:/workspace/bee-bench/results\" bee-bench\n\n# Full suite, against a Bee node on the host\ndocker run --rm --network=host \\\n  -e BEE_URL=http://localhost:1633 \\\n  -e BEE_BATCH_ID=\u003chex\u003e \\\n  -v \"$PWD/out:/workspace/bee-bench/results\" \\\n  bee-bench\n```\n\nPin upstream versions with `--build-arg`:\n\n```bash\ndocker build \\\n  --build-arg BEE_GO_REF=v1.1.0 \\\n  --build-arg BEE_RS_REF=v1.1.0 \\\n  --build-arg BEE_JS_REF=v12.1.0 \\\n  -t bee-bench .\n```\n\nA `.devcontainer/devcontainer.json` is also included, so VS Code Remote / GitHub Codespaces will spin up a ready-to-go environment with the toolchain and editor extensions pre-configured.\n\n### CI\n\nTwo workflows in `.github/workflows/`:\n\n- **`validate.yml`** runs on every PR + main: JSON / script-syntax / shellcheck / hadolint, plus an end-to-end aggregate-pipeline test against a synthetic 3-runner fixture. Fast (~1 min).\n- **`bench-cpu.yml`** runs on every PR + main + manual dispatch: checks out `bee-go` / `bee-rs` / `bee-js` as siblings, builds all three runners, runs the CPU subset (all `cpu.*` cases; `net.*` cases skip without `BEE_BATCH_ID`), uploads `report.md` / `report.html` / `report.csv` / `aggregate.json` plus the per-runner JSONs as a 14-day artifact. Slower (~5–10 min cold; ~3 min warm with cargo / npm / go-mod cache).\n\n## Env vars\n\n| Var | Default | Meaning |\n|---|---|---|\n| `BEE_URL` | `http://localhost:1633` | Bee node base URL |\n| `BEE_BATCH_ID` | (required) | hex of usable postage batch |\n| `BENCH_LARGE` | `0` | Set to `1` to enable 1GB cases |\n| `BENCH_ITERS` | `5` | Override iteration count |\n| `BENCH_RUNNER_ORDER` | rotates daily | Override, e.g. `go,rs,js` |\n\n## Layout\n\n```\nbench-spec.json          cases, sizes, iters, runner_subset\nfixtures/                generated random bins (gitignored)\nresults/                 per-runner JSON output (gitignored)\n  _\u003clabel\u003e/              snapshots from preserve-run.sh\n  _baseline_*/           preserved canonical runs\n  _archive/              superseded partial runs\nrunner-go/               cmd-style Go runner; replace ../bee-go\nrunner-rs/               cargo bin; path = \"../bee-rs\"\nrunner-js/               Node runner; file:../bee-js\n  keccak-worker.mjs      Worker thread for cpu.keccak.parallel\nscripts/\n  gen-fixtures.sh        random binaries for sizes_mb + 1GB\n  run-all.sh             rotates runner order, runs all, aggregates\n  aggregate.mjs          results/*.json → report.md + report.html\n  export-csv.mjs         aggregate.json → flat report.csv\n  validate-spec.mjs      check each runner's latest JSON covers the spec\n  compare.mjs            diff two aggregate.json files\n  preserve-run.sh        snapshot results/ to results/_\u003clabel\u003e/\nFINDINGS.md              qualitative observations\n```\n\n## Cases\n\nSee `bench-spec.json` for the authoritative list. Each runner reads it at startup and dispatches by `id`. Cases are grouped by domain in the report:\n\n- **CPU** — `cpu.keccak.*`, `cpu.bmt.*`, `cpu.ecdsa.sign-1000`, `cpu.identity.create`, `cpu.manifest.hash-50files`. Pure client work, no Bee involvement.\n- **Calibration** — `net.stamps.list`, `net.stamps.concurrent`. Control + HTTP-stack overhead.\n- **Feeds** — `net.feed.write-read.fresh`, `.warm`. Bee `/feeds` endpoint cost (Sepolia-bound, slow).\n- **Network upload** — `net.bzz.upload`, `.upload.encrypted`, `net.bytes.upload`. POST paths.\n- **Network upload (streaming from disk)** — `net.bzz.upload-from-disk`. **bee-rs N/A** (no AsyncRead path; documented in FINDINGS).\n- **Network download** — `net.bzz.download`, `net.bytes.head`, `net.bytes.download.range`. ⚠ Local-cache hit when the bench just uploaded; not a real-network metric.\n- **Bee chunk-pipeline** — `net.chunks.upload`, `net.stream-dir.upload`, `net.soc.upload`. ⚠ Sepolia-bottlenecked, not a client comparison.\n\n### Adding a new case\n\nA case is the same code shape across three runners + one entry in `bench-spec.json`. Concretely:\n\n1. **`bench-spec.json`** — append an object to `cases`:\n   ```json\n   {\n     \"id\": \"net.bytes.upload\",\n     \"kind\": \"net\",\n     \"params_from\": \"sizes_mb\",\n     \"doc\": \"POST /bytes for each size_mb. Server-side default deferred upload.\"\n   }\n   ```\n   `kind` is `cpu` or `net`. Use `params: [...]` for an explicit list, or `params_from: \"sizes_mb\"` / `\"sizes_mb_plus_large\"` to inherit the global size sweep. The `doc` string is rendered under the case heading in the report — keep it one line, name the endpoint or the operation.\n\n2. **`runner-go/cases.go`** — add a function `func runMyCase(ctx ..., param Param) (CaseResult, error)`, register it in `cases.go:Dispatch` by `id`. Use the existing helpers (`measureIters`, `withRSS`, `randomBytes`).\n\n3. **`runner-rs/src/cases.rs`** — same shape: add an async `pub async fn run_my_case(...) -\u003e Result\u003cCaseResult\u003e` and dispatch by id.\n\n4. **`runner-js/runner.mjs`** — add a case in the dispatch switch; the helpers are inline at the top of the file.\n\n5. **Run all three.** Each runner reads `bench-spec.json`, hashes it, and embeds the hash in the result JSON so downstream consumers can detect spec drift between runners.\n\nIf a runner can't implement the case (e.g. `net.bzz.upload-from-disk` on bee-rs which lacks an AsyncRead path), return `CaseResult{ skipped: true, skip_reason: \"...\" }`. The aggregator renders skipped cells as `*skip:* \u003creason\u003e` and excludes them from the runner's geomean.\n\nThe compare script (`scripts/compare.mjs`) cross-checks two `aggregate.json` files for missing cases — useful when adding a case to confirm all runners actually emit it.\n\n### Default-mode caveat\n\nAll cases use Bee's server-side default `Swarm-Deferred-Upload: true`. Upload returns when chunks land in **local Bee**, not when network-replicated. Downloads of just-uploaded refs hit local cache. To re-run with `deferred: false` on each client and compare, see `compare.mjs` below.\n\n## Reading the report\n\n`results/report.md` (or `report.html`) is structured:\n\n1. **Runners** table — versions and host info per runner.\n2. **Scoreboard** — geometric mean of `median_ms / fastest_in_row` per runner. `1.00x` = fastest, higher = slower. Wins column = rows where the runner had the lowest median. Per-group columns (CPU, Network upload, etc.) help separate where each client wins/loses.\n3. **Per-domain sections** — each case gets its own table. Each cell shows:\n   - Line 1: median (or **best**) + ratio to fastest, e.g. `**516.6ms** (best)` or `590.7ms (1.14x)`\n   - Line 2: throughput · per-unit metric, e.g. `66.1 MB/s · 59µs/call`\n   - Line 3: variance (`±X%`, `cv X%`, `p95` when n ≥ 10), peak RSS, and CPU/wall ratio (`cpu Xms (Y.YYx)`). The CPU/wall ratio distinguishes:\n     - `~ 1.00x` → single-thread CPU-bound (e.g. keccak hashing)\n     - `\u003c 1.00x` → wait-bound, mostly blocked on I/O (e.g. `/chunks` upload waiting on Bee sync)\n     - `\u003e 1.00x` → multi-threaded; ~N for N cores active (e.g. `cpu.keccak.parallel`)\n     `⚠` flag prepended on `±X%` when variance \u003e 50% (flaky measurement).\n   - Line 4: per-iter sparkline — reveals JIT warmup, GC pauses, network jitter\n4. **Latency-vs-size linear fit** — for cases with multiple sizes (`cpu.bmt.file-root`, `net.bzz.upload`, etc.), a regression `time ≈ fixed_overhead + bytes / throughput` showing per-runner per-call overhead and peak throughput.\n5. **Inline SVG bars** in each row — visual ranking, fastest is green.\n\n## Comparing runs\n\n```bash\n# Snapshot a run\n./scripts/preserve-run.sh deferred_true\n\n# ...do another run with different parameters...\n./scripts/preserve-run.sh deferred_false\n\n# Diff\nnode scripts/compare.mjs \\\n  results/_deferred_true/aggregate.json \\\n  results/_deferred_false/aggregate.json \\\n  --out results/compare.md\n```\n\nThe compare report shows a per-runner geomean shift (e.g. `↓ 12.3% faster`), and per-row delta columns flagging anything `\u003e ±20%` with `⚠`.\n\n## Discipline\n\n- Sequential everywhere. One runner at a time, one case at a time, one iteration at a time. Concurrent cases (`net.stamps.concurrent`, `cpu.keccak.parallel`) deliberately fan out to test concurrency, but only one such case is in flight at a time.\n- Fixtures pre-loaded into RAM before timing; salt prefix per iter so each upload produces a unique reference (no Bee dedup warm-cache effect).\n- Body drains for downloads — never buffer the full response.\n- Peak RSS sampled in-process at 100ms intervals.\n\n## Findings\n\nQualitative observations live in `FINDINGS.md`. Highlights:\n\n- bee-js ECDSA is **221x slower** than bee-go on `cpu.ecdsa.sign-1000` (16ms vs 73µs per sign). bee-rs is 1.6x slower than bee-go because k256 ships no asm.\n- bee-js BMT chunker plateaus at ~5.9 MB/s regardless of size — pure-JS keccak floor. bee-go ~60 MB/s, bee-rs ~77 MB/s.\n- bee-js holds ~14x its input as RSS during chunking (1.4GB at 100MB BMT) — V8 + MerkleTree heap behavior.\n- bee-rs has no streaming raw-bytes upload (`upload_file` / `upload_data` buffer fully). `net.bzz.upload-from-disk` is the data point.\n- Sepolia `/chunks` and `/feeds` endpoints are **dominated by network sync** (~600ms/chunk, 30-60s for fresh feed lookup). Those rows are flagged as Bee-bottlenecked, not client comparisons.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fethswarm-tools%2Fbee-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fethswarm-tools%2Fbee-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fethswarm-tools%2Fbee-bench/lists"}