{"id":50998078,"url":"https://github.com/onsi/biloba-comparison","last_synced_at":"2026-06-20T11:30:47.047Z","repository":{"id":365113435,"uuid":"1270610303","full_name":"onsi/biloba-comparison","owner":"onsi","description":"Performance Comparisons between Biloba and Playwright","archived":false,"fork":false,"pushed_at":"2026-06-15T22:48:38.000Z","size":70,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-16T00:19:39.712Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/onsi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-15T22:10:41.000Z","updated_at":"2026-06-15T22:48:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/onsi/biloba-comparison","commit_stats":null,"previous_names":["onsi/biloba-comparison"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/onsi/biloba-comparison","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onsi%2Fbiloba-comparison","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onsi%2Fbiloba-comparison/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onsi%2Fbiloba-comparison/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onsi%2Fbiloba-comparison/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/onsi","download_url":"https://codeload.github.com/onsi/biloba-comparison/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onsi%2Fbiloba-comparison/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34568738,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-20T02:00:06.407Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-20T11:30:45.916Z","updated_at":"2026-06-20T11:30:47.039Z","avatar_url":"https://github.com/onsi.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Biloba vs Playwright — a performance comparison\n\n\u003e 🤖 **This whole comparison — harness, scenarios, charts, and write-up — was generated\n\u003e by [Claude](https://www.anthropic.com/claude) (Opus 4.8), working from a brief.\n\u003e Feedback is welcome, just [open an issue](https://github.com/onsi/biloba-comparison/issues)!**\n\nA fair, reproducible **speed** comparison between\n[Biloba](https://github.com/onsi/biloba) (Go, on chromedp) and\n[Playwright Test](https://playwright.dev) (TypeScript/Node). One shared server, one\ncanonical scenario list implemented identically on every side, a framework-neutral\nexternal stopwatch. The aim is to measure framework + runtime overhead for an\nidentical browser-test workload — not to manufacture a win. Read\n[`METHODOLOGY.md`](./METHODOLOGY.md) for the fairness controls and threats-to-validity,\nand [`SCENARIOS.md`](./SCENARIOS.md) for the exact workload.\n\n\u003e **Numbers below are from one machine** (Apple M1 Max, macOS, 8 performance cores),\n\u003e recorded 2026-06-15 — indicative, not universal. Run it yourself.\n\n## The three configurations\n\nBiloba 0.3.0 ships two interaction tracks, so this is a **three-way** comparison, and\nthe **identical 32-scenario suite runs under all three**:\n\n| config | how interactions run | speed |\n|---|---|---|\n| **biloba-fast** | the default `b` — fast, atomic JavaScript simulations (`el.click()`, value-set, synthetic events). No scroll-into-view, occlusion check, or real pointer. | fastest |\n| **biloba-realistic** | `b.Realistic()` — the *same tab*, with interactions routed through real Chrome DevTools Protocol input (scroll-into-view, real pointer moves, occlusion-aware clicks, real keystrokes). | middle |\n| **Playwright** | its only model — always realistic (real CDP input, actionability waits). | baseline |\n\nbiloba-fast and biloba-realistic are the *same* Ginkgo suite: every interaction goes\nthrough one handle that is either `b` or `b.Realistic()` (selected by an env var), so\n**only the interaction engine differs** between the two runs.\n\n**The fast ↔ realistic tradeoff is a speed-for-fidelity dial.** Fast skips the\nper-interaction CDP round-trips, so it's ~2× faster than realistic (serial) — use it\nfor the bulk of a suite. Realistic matches Playwright's input fidelity (real pointer,\nreal keys, scroll, occlusion) for the handful of tests that need it — and, as the\nnumbers show, still runs ~2× faster than Playwright whole-suite.\n\n## Topline — whole-suite wall clock (8 workers, n=15)\n\nMedian wall clock ± stddev over 15 repeats:\n\n| config | parallel(8) | serial(1) | vs Playwright (parallel / serial) |\n|---|---:|---:|---|\n| **biloba-fast** | **2.57 ± 0.06s** | **9.55 ± 0.05s** | **3.2× / 4.0×** |\n| **biloba-realistic** | **3.26 ± 0.03s** | **18.60 ± 0.07s** | **2.5× / 2.1×** |\n| **playwright** | 8.23 ± 0.24s | 38.37 ± 0.56s | — |\n\nEach bar below splits the total into **fixed startup** (Biloba shares one Chrome; Playwright\nlaunches one browser per worker) and **spec runtime**, with ±stddev error bars:\n\n![whole-suite wall clock — startup vs spec runtime](charts/config.svg)\n\n- biloba-fast is **3.2× faster parallel / 4.0× serial** than Playwright.\n- biloba-realistic — doing the *same* real-CDP-input work Playwright does — is still\n  **~2.5× faster parallel / ~2.1× serial**.\n- Realism costs **1.27× parallel / 1.95× serial** over fast: the price of real input.\n\n## By category — the same three configs, per bucket\n\nSerial **mean per-test duration ± SEM** (~25 samples/scenario — the same numbers the chart\nshows). This shows how the gap behaves as the work gets heavier. C/D have wide error bars\nbecause those buckets mix fast and slow scenarios (latency / async); ratios use the means.\n\n| bucket | biloba-fast | biloba-realistic | playwright | pw/fast | pw/real |\n|---|---:|---:|---:|---:|---:|\n| A/B static (reads) | 11.2 ± 0.2 ms | 25.7 ± 1.8 ms | 73.1 ± 1.7 ms | **6.5×** | 2.8× |\n| C network | 83.6 ± 10.6 ms | 102.6 ± 10.5 ms | 249.9 ± 27.2 ms | 3.0× | 2.4× |\n| D scale | 120.9 ± 17.2 ms | 175.5 ± 17.1 ms | 295.1 ± 32.8 ms | **2.4×** | 1.7× |\n| F semantic locators | 15.7 ± 0.2 ms | 29.7 ± 1.2 ms | 69.8 ± 0.7 ms | 4.4× | 2.4× |\n| G interaction vocabulary | 11.2 ± 0.1 ms | 47.0 ± 1.9 ms | 83.1 ± 1.1 ms | 7.4× | 1.8× |\n| H pointer options | 11.2 ± 0.4 ms | 41.9 ± 1.2 ms | 71.9 ± 0.6 ms | 6.4× | 1.7× |\n| E realism (occlusion/scroll) | 13.2 ± 0.3 ms | 163.2 ± 18.4 ms | 211.7 ± 19.1 ms | 16.0× | 1.3× |\n\n![per-bucket per-test duration](charts/buckets.svg)\n\n- biloba-fast's lead is **widest (~7×) on trivial DOM** — almost pure framework overhead —\n  and **compresses to ~2.4×** once real browser work dominates (D scale), where both run on\n  the same Chromium engine.\n- biloba-realistic tracks **~2× faster than Playwright** wherever it does real input, and\n  **converges toward it on the heaviest work** (D 1.7×) where both are engine-bound.\n- **E realism** is the outlier: biloba-fast is near-instant there because its atomic click\n  *skips* the actionability wait (occlusion) and scroll that a real user needs; realistic\n  does that work, landing next to Playwright. (Use realistic — or `BeClickable()` — when that\n  actionability matters.)\n\n## Per-test — every scenario, all three configs\n\nEach scenario plotted as Biloba per-test (y) vs its Playwright per-test (x), log–log, with the\n`y=x` diagonal and 2×/4× reference lines; below the diagonal = Biloba faster. ~25 serial samples\nper point, P25–P75 error whiskers (small — per-test timing is stable).\n\n![per-test scatter — Biloba vs Playwright](charts/scatter.svg)\n\nbiloba-fast (green) sits near/below the **4× line** and is roughly flat — its cheap atomic path\nbarely moves with how much work Playwright is doing. biloba-realistic (blue) hugs the **2× line** —\nit scales *with* Playwright (both CDP-input-bound) but stays about twice as fast.\n\n## The workload\n\n32 scenarios across 7 buckets, replicated `REPS` times (default 8 → 256 specs/config). The same\nassertions run on every config. Buckets get heavier left-to-right:\n\n| bucket | tests | what they exercise | rough DOM |\n|---|---:|---|---|\n| **A/B static** | 10 | read-only DOM (count/visibility/text/attr/class/prop) + basic interactions (click counter, form fill, real-key typing) | trivial (~15 nodes) |\n| **C network** | 5 | observe a real fetch, stub it, wait through 300 ms latency, abort it, modify the real response (CDP `Fetch`/`Network`) | trivial (~6 nodes) |\n| **D scale** | 4 | render a **1000-row** table, filter a large list by real keys, drive a gated 4-step wizard, await staggered async appends | **heavy (~2000 nodes)** |\n| **E realism** | 2 | an occluding overlay (cleared at 250 ms) and a button ~4000 px below the fold | trivial (~5 nodes) |\n| **F semantic locators** | 3 | select by role+name / visible text / form label against **~200 distractor** roled elements (exercises the accessible-name engine) | medium (~600 nodes) |\n| **G interaction vocabulary** | 6 | double / right / middle-click, drag-and-drop, wheel scroll, tap | trivial (~12 nodes) |\n| **H pointer options** | 2 | click at an offset (`At`) and modifier-click (`Shift`) | trivial (~4 nodes) |\n\nThe Biloba suite also carries 3 fast-only CSS-hook variants of Bucket F (to measure the\nCSS-vs-locator cost inside Biloba — the locator engine adds ~1.1×); they're excluded from the\nheadline by label.\n\n## Running it\n\nPrereqs: Go, Node ≥18, a one-time browser install on each side (`biloba/README.md`,\n`playwright/README.md`). Then:\n\n```bash\n./run.sh                 # topline table: all three configs, whole-suite wall clock\n./scaling.sh             # startup vs runtime fit (behind the topline chart)\n./pertest.sh             # per-test timing → the per-bucket table + scatter/bucket charts\nREPS=16 ./buckets.sh     # cross-check: per-bucket marginal per-spec via wall clock (startup-subtracted)\n```\n\nThe per-bucket table \u0026 charts come from `pertest.sh` (mean per-test); `buckets.sh` is an\nindependent wall-clock cross-check that lands in the same ballpark. The figures are SVGs under\n[`charts/`](./charts), generated by the stdlib-only Go tool there.\nLayout: `server/` (shared Go target + fixtures), `biloba/` (Ginkgo suite), `playwright/`\n(`@playwright/test` suite), `charts/` (SVG generator), and the run scripts.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonsi%2Fbiloba-comparison","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fonsi%2Fbiloba-comparison","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonsi%2Fbiloba-comparison/lists"}