{"id":51683171,"url":"https://github.com/rctruta/sql-benchmarks-dagster","last_synced_at":"2026-07-15T16:02:34.837Z","repository":{"id":366659988,"uuid":"1103644298","full_name":"rctruta/sql-benchmarks-dagster","owner":"rctruta","description":"SQL Benchmarking Laboratory using Dagster for Orchestration","archived":false,"fork":false,"pushed_at":"2026-07-14T01:21:19.000Z","size":4337,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-14T03:13:52.168Z","etag":null,"topics":["bash-script","benchmarking","benchmarks","dagster","docker-compose","duckdb","ec2","graph-database","knowledge-graph","polars","postgresql","sql","terraform","typedb","typeql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rctruta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-11-25T06:27:09.000Z","updated_at":"2026-07-14T01:21:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/rctruta/sql-benchmarks-dagster","commit_stats":null,"previous_names":["rctruta/sql-benchmarks-dagster"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/rctruta/sql-benchmarks-dagster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rctruta%2Fsql-benchmarks-dagster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rctruta%2Fsql-benchmarks-dagster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rctruta%2Fsql-benchmarks-dagster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rctruta%2Fsql-benchmarks-dagster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rctruta","download_url":"https://codeload.github.com/rctruta/sql-benchmarks-dagster/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rctruta%2Fsql-benchmarks-dagster/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35511400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-15T02:00:06.706Z","response_time":131,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash-script","benchmarking","benchmarks","dagster","docker-compose","duckdb","ec2","graph-database","knowledge-graph","polars","postgresql","sql","terraform","typedb","typeql"],"created_at":"2026-07-15T16:02:33.761Z","updated_at":"2026-07-15T16:02:34.824Z","avatar_url":"https://github.com/rctruta.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SQL Benchmarking Laboratory\n\n\u003e **A deterministic, orchestrated harness for verifying database performance at scale.**\n\u003e Developed by Ramona C. Truta\n\n---\n\n## The Mission: \"Ground Truth\" as Code\n\nThis platform is a specialized laboratory for testing SQL performance hypotheses. It transforms query tuning from intuition into a reproducible science. \n\nThe core of the system is a **Deterministic Orchestration Harness** that guarantees that if the logic or the environment changes, the benchmark result changes. If they do not, the result is addressable and cached.\n\n**Scope:** the focus to date is **synthetic and canonical (TPC-H) data**, which\nis the right instrument for *mechanism* experiments — where controlled,\nreproducible data isolates the variable under test. Real-data support exists but\nis experimental; see the [FAQ](FAQ.md) for the synthetic-vs-real rationale, the\ncontainer model, and the roadmap (AI-security testbed, real-data trust chain).\n\n---\n\n## Key Features \u0026 Innovations\n\n### 1. Context-Aware Semantic Hashing (The Experiment ID)\nThe Heart of the system is the **Experiment ID**, an 8-character hash that governs the entire lifecycle. This hash is a SHA-256 fingerprint generated from:\n*   **The Config**: Every dimension in your YAML (rows, skew, parameters).\n*   **The SQL Logic**: The actual content of the benchmarked scripts.\n*   **The Code**: All measurement-relevant Python — orchestration (`assets/`), engine clients (`resources/`), and data generators (`plugins/`).\n\n**Semantic Normalization**: The hashing engine distinguishes between a logic change and a formatting change. \n*   **SQL**: Comments, whitespace, and case are normalized before hashing.\n*   **Python**: Orchestration scripts are parsed into an **Abstract Syntax Tree (AST)** to strip docstrings and formatting variations, ensuring the Experiment ID only changes when execution logic changes.\n\n### 2. Multi-Layer Cold-Cache Isolation\nTo ensure IO-bound performance is not masked by memory buffers, we implement a dual-layer cold start mechanism:\n*   **Out-of-Process (Postgres)**: Mandatory **Docker Container Restarts** before every query to clear engine-level shared buffers.\n*   **Global OS Flush (mmap)**: A specialized `thrash_os_cache` primitive that maps and dirties a file larger than physical RAM. This forces the OS to evict Page Cache entries, ensuring cold read performance for both containerized and in-process (DuckDB) engines.\n\n### 3. Agentic AI Integration\nThe platform is built for the future of **Autonomous Engineering**. The Experiment ID allows AI agents to treat the laboratory as a **Deterministic Performance API**.\n*   See [AGENTS.md](AGENTS.md) for the full Agentic Benchmarking Protocol.\n\n### 4. Declarative Matrix Orchestration\nBenchmarks are defined as N-dimensional matrices in YAML. The platform expands these into a Cartesian product of **Independent Dagster Partitions**. This allows for parallel dispatch and granular retries.\n\n---\n\n## Usage \u0026 Technical Setup\n\n### Prerequisites\n*   **[uv](https://astral.sh/uv)**: the project's Python environment \u0026 dependency manager (fast, modern). `setup.sh` uses it, and it can install Python 3.11 for you.\n*   **Python 3.11+**: core runtime (uv provisions it if missing).\n*   **Docker**: for the containerized engines (Postgres, TypeDB), which the harness manages itself — *not* required for the DuckDB-only quickstart.\n\n### Installation \u0026 Setup\n\n**From PyPI** — the lab is installable, which gives you the tooling and the `sqlbench` CLI:\n```bash\npip install sqlbenchdag\n```\n\n**From source** — clone the repo to also get the published capsules (`sql_benchmarks/experiments/results/`) and the full harness. The laboratory includes a comprehensive setup script that manages virtual environments, dependencies, and directory initialization.\n\n```bash\n# 1. Automate Setup\nchmod +x setup.sh \u0026\u0026 ./setup.sh\n\n# 2. Activate Laboratory\nsource venv/bin/activate\n```\n\n### The Execution Workflow (CLI)\nWhile the system is powered by Dagster, the primary interface is the CLI for automated workflows.\n\n```bash\n# QUICKSTART: DuckDB only, no Docker required — runs in seconds\n./run.sh sql_benchmarks/experiments/queue/quickstart.yaml --auto\n\n# FULL BENCHMARK: Run all queued experiments (requires Docker for Postgres)\n./run.sh queue --auto\n\n# SINGLE EXPERIMENT: Pass a specific config path\n./run.sh sql_benchmarks/experiments/queue/quack_execution_modes.yaml --auto\n```\n\nTo write your own experiment, start from the annotated template and see the\ncatalog of published runs in **[docs/experiments.md](docs/experiments.md)**.\n\n---\n\n## Project Structure\nManaged with a strict separation between Harness and Scenario:\n\n```text\n/sql_benchmarks\n├── /assets                 # THE HARNESS: Dagster factories \u0026 pipeline logic\n├── /resources              # THE INFRASTRUCTURE: DB drivers \u0026 Docker management\n├── /scripts/sql            # THE SCENARIOS: Raw SQL partitioned by scenario\n├── /plugins                # THE DATA: Declarative generators \u0026 scenario providers\n├── /utils                  # THE BRAIN: AST-hashing, common logic, \u0026 system primitives\n└── /experiments            # THE LABORATORY\n    ├── active.yaml         # LOCAL runtime staging — the coordinator overwrites\n    │                       #   this on every run (gitignored; not shared state)\n    ├── /queue              # Staging area for new experiment configs\n    ├── /archive            # Library of previously defined experiment templates\n    ├── /configs            # Registry of immutable, hash-addressed experiment capsules\n    └── /results            # Data capsules: fragments, CSVs, and Dashboards\n```\n\n---\n\n## Published Experiments — the Quack investigation\n\nThe first published study with this lab: an independent measurement of DuckDB's\n**Quack** client-server protocol (beta, v1.5.3). Each row links the runnable\nconfig to its committed, verifiable capsule. Full numbers, scaling exponents,\nand verification steps in [docs/published_capsules.md](docs/published_capsules.md).\n\n| Act | Question | Config | Capsule | Finding |\n| :--- | :--- | :--- | :--- | :--- |\n| **I** | What does the protocol cost? | [quack_execution_modes.yaml](sql_benchmarks/experiments/queue/quack_execution_modes.yaml) | `b8e2bfaf` | Attach-mode overhead grows with scan size (2.6× → **9.5×** at 10M rows); pushdown stays flat at ~2×. |\n| **II** | Why is pushdown ~2× and not 1×? | [quack_residual_threads.yaml](sql_benchmarks/experiments/queue/quack_residual_threads.yaml) | `25b0e134` | The residual tracks reduced server-side parallelism (~2–4 of 8 threads), not transport. |\n| **III** | Does it generalize to joins? | [tpch_quack_validation.yaml](sql_benchmarks/experiments/queue/tpch_quack_validation.yaml) | `b198363e` | On canonical TPC-H Q3, pushdown holds (~1.9×); attach mode **cannot run the join at all** (DNF). |\n| **IV** | Does it beat the incumbent? | [quack_vs_postgres.yaml](sql_benchmarks/experiments/queue/quack_vs_postgres.yaml) | `902d1277` | DuckDB-over-Quack (pushdown) beats PostgreSQL by up to **13.2×**, and the gap widens with scale. |\n\n*Act 0 (`b82b4eae`) is the exploratory scout that started it — see [docs/published_capsules.md](docs/published_capsules.md).*\n\nThe lab also ships scenario suites for other studies — `selectivity/`,\n`null_logic/`, `null_sentinel/`, `recursion/`, `sort_spill/`, `tpch/`, and more\nunder `sql_benchmarks/scripts/sql/` — each runnable across the engine matrix.\n\n---\n\n## Naming \u0026 Provenance\n\nThe lab works at two layers, named two different ways. Keeping them distinct is\nthe difference between a number you can trace and a number you have to trust.\n\n| Layer | What it is | Named by | Example |\n| :--- | :--- | :--- | :--- |\n| **Capsule** | one experiment's complete results | its **Experiment ID** — an 8-char SHA-256 of the *question* (config + SQL + measurement code), assigned by the machine | `48c92f31` |\n| **Release** | a signed set of capsules backing a publication | a human name: **`sqlbenchdag-\u003ctopic\u003e-v\u003cN\u003e-\u003cYYYYMMDD\u003e`** | `sqlbenchdag-quack-v1-20260614` |\n\nA release *contains* capsules; a capsule is one experiment. The machine names\ncapsules (hashes you can't forge); the author names releases (words that mean\nsomething). `sqlbenchdag` is the lab's maker's mark — `sql` + `bench` +\n`dag` (Dagster, the orchestration that distinguishes this lab from a bare query\ntimer). It is carried on every release this lab produces.\n\n**Two hashes, two jobs** (a capsule contains both — they are not the same thing):\n- the **Experiment ID** (8 chars) is an *address* — it *names* the capsule, and\n  the same question always produces the same address;\n- the **integrity seal** (64 chars, in `integrity.seal`) is a *checksum* — it\n  *verifies* the capsule's contents are untouched. You file by the address; the\n  seal is the tamper-evident lid.\n\n**Reproducibility is pinned to the release.** Because the Experiment ID hashes\nthe measurement code, it is stable only at a fixed code revision — which is\nexactly what the signed release tag freezes. To reproduce a published capsule,\ncheck out its release tag and re-run its config; the same question yields the\nsame ID, or the comparison is refused by construction.\n\n\u003e Provenance roadmap: a per-capsule `generator` stamp (`sqlbenchdag@\u003ccommit\u003e`)\n\u003e will record the exact build that produced each capsule. It writes through the\n\u003e finalization code, which lives inside the Experiment-ID hash, so it ships with\n\u003e a future release rather than re-numbering capsules already published.\n\nSee [docs/published_capsules.md](docs/published_capsules.md) for the full trust\nchain (reproducibility, integrity, timestamp, authorship) and how to verify each.\nReproducibility and integrity are **automatic** on every run; timestamp and\nsignature are **optional** steps you add only when publishing — the producer\nworkflow is [docs/PUBLISHING.md](docs/PUBLISHING.md).\n\n---\n\n## License\n\nCopyright 2025-2026 Ramona C. Truta. Licensed under the [Apache License 2.0](LICENSE).\n\nBuilt with **Dagster**, **Polars**, **Docker**, **DuckDB**, and **Postgres**.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frctruta%2Fsql-benchmarks-dagster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frctruta%2Fsql-benchmarks-dagster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frctruta%2Fsql-benchmarks-dagster/lists"}