{"id":51143710,"url":"https://github.com/paradedb/benchmarker","last_synced_at":"2026-06-26T01:02:10.829Z","repository":{"id":354268220,"uuid":"1148624831","full_name":"paradedb/benchmarker","owner":"paradedb","description":"Database benchmarking suite based on Grafana K6. Benchmark ParadeDB, Elasticsearch, OpenSearch, ClickHouse, and others!","archived":false,"fork":false,"pushed_at":"2026-06-14T21:32:44.000Z","size":14616,"stargazers_count":24,"open_issues_count":10,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-14T23:16:54.786Z","etag":null,"topics":["benchmarks","clickhouse","elasticsearch","mongodb","opensearch","paradedb","postgres"],"latest_commit_sha":null,"homepage":"https://paradedb.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paradedb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["paradedb"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2026-02-03T07:12:06.000Z","updated_at":"2026-06-14T21:32:46.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/paradedb/benchmarker","commit_stats":null,"previous_names":["paradedb/benchmarker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/paradedb/benchmarker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradedb%2Fbenchmarker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradedb%2Fbenchmarker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradedb%2Fbenchmarker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradedb%2Fbenchmarker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paradedb","download_url":"https://codeload.github.com/paradedb/benchmarker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradedb%2Fbenchmarker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34798183,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-25T02:00:05.521Z","response_time":101,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarks","clickhouse","elasticsearch","mongodb","opensearch","paradedb","postgres"],"created_at":"2026-06-26T01:02:09.576Z","updated_at":"2026-06-26T01:02:10.793Z","avatar_url":"https://github.com/paradedb.png","language":"Go","funding_links":["https://github.com/sponsors/paradedb"],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  \u003cbr\u003e\n  ParadeDB Benchmarker\n  \u003cbr\u003e\n\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eBenchmark any database with k6. Real-time dashboard, consistent metrics, reproducible results.\u003c/b\u003e\n\u003c/p\u003e\n\n\u003ch3 align=\"center\"\u003e\n  \u003ca href=\"docs/scripting.md\"\u003eScripting Guide\u003c/a\u003e \u0026bull;\n  \u003ca href=\"docs/dashboard.md\"\u003eDashboard\u003c/a\u003e \u0026bull;\n  \u003ca href=\"docs/datasets.md\"\u003eDatasets\u003c/a\u003e \u0026bull;\n  \u003ca href=\"docs/loader.md\"\u003eData Loader\u003c/a\u003e \u0026bull;\n  \u003ca href=\"docs/docker.md\"\u003eDocker\u003c/a\u003e \u0026bull;\n  \u003ca href=\"CONTRIBUTING.md\"\u003eContributing\u003c/a\u003e\n\u003c/h3\u003e\n\n---\n\nA [k6](https://grafana.com/docs/k6/latest/) extension for benchmarking databases with a unified API, real-time dashboard, and comprehensive data loading tools. While the included datasets focus on full-text search, the framework works for any query workload. You write the SQL or API calls, it handles timing, metrics, and visualization.\n\nCompare performance across **ParadeDB**, **PostgreSQL FTS**, **Elasticsearch**, **OpenSearch**, **ClickHouse**, and **MongoDB Atlas Search** with consistent metrics and visualization. Docker Compose profiles are included for single-node benchmarking, but you can point at any database: local installs, remote servers, or cloud services.\n\nWe built this at [ParadeDB](https://paradedb.com) to drive our iterative performance improvement process and power our benchmarks. It's very early, but we hope it can help others get their TTFB (time to first benchmark) down. We'd ❤️ your help improving it, PRs welcome!\n\n![benchmarker](images/benchmarker.png)\n\n## How It Works\n\nThe benchmarker is built on [Grafana k6](https://grafana.com/docs/k6/latest/), an amazing load testing tool written in Go. You write a k6 JavaScript script that defines **scenarios** and **backends**. Each scenario specifies an executor (how load is generated), a duration, a number of virtual users (VUs), and which functions to run, using which backends. k6 spins up VUs as concurrent goroutines, each looping over your test function for the duration of the scenario.\n\nThis project extends k6 with the `k6/x/database` module, adding backend drivers, automatic metrics, a real-time dashboard, and an export format on top.\n\n### Composing tests\n\nA single script can compose multiple scenarios across multiple backends. You might run queries against ParadeDB for 30 seconds, then against Elasticsearch for 30 seconds, with a built-in phase timer staggering them so they don't compete for system resources. Within each phase you can layer different workloads: a full-text search at 200 QPS, an aggregation query at 100 QPS, and a 1,000 row/s ingest stream, all running concurrently. The framework times every operation, tags it with the backend name, and pushes metrics to the dashboard automatically.\n\nWhile the framework exposes many backends, it's up to the user to write the queries to test (in JSON or SQL). A user can expect that the _way_ the queries are run is optimal, but must still make sure the content of the queries is sane.\n\n### Ingest \u0026 update workloads\n\nIn addition to queries, you can run concurrent ingest (insert) and update workloads. The primary purpose is to put write pressure on the database while queries are running, measuring how query latency degrades under a realistic mixed workload. This is more meaningful than comparing raw ingest or update throughput across backends, since each database handles write consistency, indexing, and flush semantics differently.\n\n### Load strategies\n\nk6 gives you several strategies for generating load:\n\n- **`constant-vus`**: fixed number of users hammering queries in a loop. The simplest way to measure throughput and latency.\n- **`ramping-vus`**: gradually increases concurrency over stages, letting you find the point where latencies spike or errors appear.\n- **`constant-arrival-rate`**: fixed number of requests per second regardless of response time. Useful for SLA testing or measuring ingest at a target throughput.\n\nYou can mix these freely across scenarios in the same script.\n\n### Dashboard\n\nResults stream to a browser in real-time: latency percentiles (P50/P90/P95/P99), queries per second, ingest rate, and Docker container CPU/memory per backend. The dashboard also captures backend configs, setup scripts, and query patterns so results can be understood and reproduced later. Export as standalone HTML to share.\n\n## Quick Start\n\n### 1. Build\n\n```bash\nmake\n```\n\n### 2. Start backends\n\nEach dataset under `datasets/` ships with its own `docker-compose.yml` that pins\nthe exact images and tuning used for that benchmark. Run compose from the\ndataset directory so the captured `Container` tab in the dashboard reflects the\nreal configuration:\n\n```bash\ndocker compose -f datasets/sample/docker-compose.yml up -d\n```\n\nThe repo-root `docker-compose.yml` is a kitchen-sink template containing every\nsupported backend with profiles; it's intended as a starting point when\nauthoring a new dataset, not for running an existing one.\n\nFor backends running off-host (e.g. AWS RDS, managed Elasticsearch), pass\n`{ type: \"paradedb\", container: \"\" }` in the backend config to skip docker\nmetrics for that backend.\n\nPlease note the 'sample' dataset which is included does not provide a meaningful benchmark, it's designed to show how to use the system.\n\nSee [Docker Setup](docs/docker.md) for all available profiles and services.\n\n### 3. Load data\n\n```bash\n./bin/loader load --backend paradedb ./datasets/sample\n```\n\nSee [Data Loader](docs/loader.md) for advanced options like custom connection strings, parallel workers and S3 pulls.\n\n### 4. Run a benchmark\n\n```bash\n./k6 run --out dashboard datasets/sample/k6/simple.js\n```\n\nOpen http://localhost:5665/static/ to see real-time results. See [Dashboard](docs/dashboard.md) for export and replay options.\n\n## Writing Scripts\n\nScripts are standard k6 JavaScript with the `k6/x/database` extension. Here's a minimal example that benchmarks ParadeDB with 5 concurrent users for 30 seconds:\n\n```javascript\nimport db from \"k6/x/database\";\n\n// Connect to the paradedb container using the standard settings from docker-compose\nconst backends = db.backends({ backends: [\"paradedb\"] });\n\n// Open a file which has a termset we can use to customise queries\nconst terms = db.terms(open(\"./search_terms.json\"));\n\n// Define the scenarios to run\nconst scenarios = {\n  paradedb: {\n    executor: \"constant-vus\",\n    vus: 5,\n    duration: \"30s\",\n    exec: \"paradedbQuery\",\n  },\n};\n\n// Add docker based metric collection\nexport const collectMetrics = backends.addDockerMetricsCollector(scenarios, \"35s\");\n\nexport const options = { scenarios };\n\n// Create the function which k6 will call on each iteration\nexport function paradedbQuery() {\n  // Activate the paradedb backend and run the query, cycling through the items in terms\n  backends\n    .get(\"paradedb\")\n    .query(\n      `SELECT id, title FROM documents WHERE content ||| $1 LIMIT 10`,\n      terms.next(),\n    );\n}\n```\n\nEach VU runs the `paradedbQuery` function in a loop for 30 seconds. The framework times every call, records the hit count, and pushes metrics to the dashboard. Swap `\"constant-vus\"` for `\"ramping-vus\"` to ramp load up and down, or `\"constant-arrival-rate\"` to send a fixed number of requests per second.\n\nThe [Scripting Guide](docs/scripting.md) covers the full module API, backend configuration, multi-backend comparisons with phase timing, ingest/update benchmarks, and query examples for every supported backend.\n\n## Documentation\n\n| Guide                                | Description                                                     |\n| ------------------------------------ | --------------------------------------------------------------- |\n| [Scripting Guide](docs/scripting.md) | Module API, backend config, benchmark patterns, query reference |\n| [Dashboard](docs/dashboard.md)       | Real-time UI, metrics reference, export \u0026 replay                |\n| [Datasets](docs/datasets.md)         | Directory structure, schema format, pre/post scripts            |\n| [Data Loader](docs/loader.md)        | CLI usage, connection strings, S3 pulls                         |\n| [Docker Setup](docs/docker.md)       | Compose profiles, service ports, TLS                            |\n| [Contributing](CONTRIBUTING.md)      | Adding backends, development setup, PR workflow                 |\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparadedb%2Fbenchmarker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparadedb%2Fbenchmarker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparadedb%2Fbenchmarker/lists"}