{"id":51486971,"url":"https://github.com/fluree/benchmark-db","last_synced_at":"2026-07-07T07:31:48.241Z","repository":{"id":369354334,"uuid":"1257456009","full_name":"fluree/benchmark-db","owner":"fluree","description":null,"archived":false,"fork":false,"pushed_at":"2026-07-04T20:37:45.000Z","size":675,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-04T22:12:09.581Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fluree.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-02T17:39:09.000Z","updated_at":"2026-07-04T20:37:34.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/fluree/benchmark-db","commit_stats":null,"previous_names":["fluree/benchmark-db"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/fluree/benchmark-db","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluree%2Fbenchmark-db","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluree%2Fbenchmark-db/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluree%2Fbenchmark-db/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluree%2Fbenchmark-db/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fluree","download_url":"https://codeload.github.com/fluree/benchmark-db/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluree%2Fbenchmark-db/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35219594,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-07T07:31:47.686Z","updated_at":"2026-07-07T07:31:48.232Z","avatar_url":"https://github.com/fluree.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# benchmark-db\n\nReproducible RDF / SPARQL benchmarks for [Fluree](https://labs.flur.ee), run head-to-head\nagainst other engines on **identical data and hardware**. Every benchmark is\nself-contained under [`benchmarks/`](benchmarks/); they share one query runner and one\nreport generator under [`common/`](common/). All engines run **natively** (no Docker,\nmatching the SPARQLoscope paper's recommendation), with each engine's result cache\ndisabled or cleared per query so every run actually re-executes.\n\nThe current suite is **[SPARQLoscope](https://github.com/ad-freiburg/sparqloscope)** —\n105 SPARQL 1.1 queries probing joins, aggregates, property paths, filters, string\nfunctions, and large result sets — run at two dataset scales (561 M → 8.19 B triples),\nplus the **[Wikidata Graph Pattern Benchmark](benchmarks/wgpb/)** (WGPB, 850 basic\ngraph pattern queries) on the full 21.5 B-triple Wikidata all-dump.\n\n---\n\n## Headline — DBLP-core: 7 engines, one box\n\nThe full SPARQLoscope suite over **DBLP-core** (~561 M triples) with\n**all seven engines on the same machine** (AWS `m7a.4xlarge`, 16 c / 64 GB) so the\ncomparison is purely engine-vs-engine. **Fluree leads every aggregate** and is one of\nonly two engines (with QLever) to answer all 105 queries.\n\n![DBLP-core geometric-mean query time, all 7 engines on one box](assets/dblp-core-geomean.svg)\n\n| metric (lower = faster) | **Fluree** | QLever | Virtuoso | MillenniumDB | Jena | Oxigraph | Blazegraph |\n|---|---|---|---|---|---|---|---|\n| **queries passed** | **105/105** | 105/105 | 103/105 | 103/105 | 34/105 | 39/105 | 3/105 |\n| **geo mean (P=2)** | **19.4 ms (1.0×)** | 202 ms (10.4×) | 300 ms (15.4×) | 1,664 ms (86×) | 67.7 s (3487×) | 87.0 s (4486×) | 333 s (17158×) |\n| **geo mean (P=10)** | **19.4 ms (1.0×)** | 202 ms (10.4×) | 309 ms (15.9×) | 1,716 ms (88×) | 200.9 s (10355×) | 239.4 s (12338×) | 1,590 s (81934×) |\n| **median (passed only)** | **41 ms (1.0×)** | 310 ms (7.6×) | 326 ms (7.9×) | 3,894 ms (95×) | 6,033 ms (147×) | 5,090 ms (124×) | 23.0 s (562×) |\n\n_The geo means follow the [SPARQLoscope paper](https://ad-publications.cs.uni-freiburg.de/ISWC_sparqloscope_BKTU_2025.pdf)'s\nofficial aggregate: a failed or timed-out query counts as 2× (P=2) or 10× (P=10) the\n180 s timeout, so every engine is scored on the same 105 queries._\n\n→ **[Full DBLP-core report](benchmarks/sparqloscope/reports/dblp-core/REPORT.md)** ·\n[per-engine raw TSVs](benchmarks/sparqloscope/reports/dblp-core/engines/) ·\n[run metadata \u0026 setup facts](benchmarks/sparqloscope/reports/dblp-core/meta.json)\n\n\u003e Fluree is **v4.0.6** (native source build). The other six engines were\n\u003e measured on the same box; the small box-to-box variance does not change the ranking —\n\u003e see the report caveats.\n\n---\n\n## Fluree scales down 4× — performance virtually unchanged\n\nWe then re-ran Fluree alone (same **v4.0.6** build) on progressively smaller boxes,\nand the headline is how little the numbers move: **geo mean 19 → 20 → 25 ms and median\n41 → 44 → 49 ms from the full 16 c / 64 GB box down to one-quarter the cores and RAM\n(4 c / 16 GB), with all 105 queries passing at every size.** And that ¼-box result is\nstill **8.1× faster on geo mean** than the next fastest engine (QLever) running on the\nfull box — 5.6× arith, 6.3× median.\n\n![Fluree scaling ramp vs QLever's full-box result](assets/dblp-core-scaling.svg)\n\n| Fluree config | cores | RAM | passed | arith | median | geo |\n|---|---|---|---|---|---|---|\n| 16c / 64 GB (full) | 16 | 64 GB | 105/105 | 251 ms | 41 ms | 19 ms |\n| 8c / 32 GB (½ box) | 8 | 32 GB | 105/105 | 265 ms | 44 ms | 20 ms |\n| **4c / 16 GB (¼ box)** | **4** | **16 GB** | **105/105** | **338 ms** | **49 ms** | **25 ms** |\n| _QLever, full 16c/64 GB (for reference)_ | 16 | 64 GB | 105/105 | _1,904 ms_ | _310 ms_ | _202 ms_ |\n\n→ **[Resource-scaling bench](benchmarks/sparqloscope/reports/dblp-core/fluree-scaling/)**\n(per-config raw TSVs + findings)\n\n---\n\n## All runs at a glance\n\nFluree leads every aggregate at both scales. On the SPARQLoscope penalized geo mean\n(P=2), the v4.0.6 build is **10.4× faster than the next fastest engine (QLever) on\nDBLP-core (561 M) and 10.5× on Wikidata-Truthy (8.19 B)**.\n\n| benchmark | triples | engines | box | Fluree passed | Fluree geo P=2 (vs next fastest) | report |\n|---|---|---|---|---|---|---|\n| **DBLP-core** | 561 M | 7 | `m7a.4xlarge` 16c/64 GB | **105/105** | **19.4 ms** (QLever 10.4×) | [report](benchmarks/sparqloscope/reports/dblp-core/REPORT.md) |\n| **Wikidata-truthy** | 8.19 B | 5 | `r7a.16xlarge` 64c/512 GB | **105/105** | **363 ms** (QLever 10.5×) | [report](benchmarks/sparqloscope/reports/wikidata-truthy/REPORT.md) |\n| **WGPB** (Wikidata all-dump) | 21.5 B | 1 (Fluree only) | `r7a.8xlarge` 32c/256 GB | **850/850** | **43 ms** | [report](benchmarks/wgpb/reports/wikidata-all/REPORT.md) |\n\n_Wikidata-truthy is the hardest SPARQLoscope scale (8.19 B triples); passed-counts fall\nfor every other engine there — Fluree is the only engine to answer all 105 queries, at\nboth scales. The WGPB row is the separate 850-query graph-pattern benchmark on the full\n21.5 B-triple Wikidata all-dump (794 GB index, ~3× RAM): 100% completion, 0 timeouts._\n\nAt the 8.19 B scale the same ordering holds — Fluree fastest on geo mean, QLever next:\n\n![Wikidata-truthy geometric-mean query time, 5 engines on one box](assets/wikidata-truthy-geomean.svg)\n\n---\n\n## Reproduce it\n\nDatasets are pinned and published to **`s3://fluree-benchmark-data/`**\n(`dblp-core/`, `wikidata-truthy/`, `wikidata-all/`) so you don't have to re-derive them;\nthe per-dataset notes under [`benchmarks/sparqloscope/datasets/`](benchmarks/sparqloscope/datasets/)\nrecord exact sources, versions, and checksums.\n\n```bash\n# 1. install Fluree (official v4.0.6 release — native binary, no source build).\n\ncurl --proto '=https' --tlsv1.2 -LsSf \\\n  https://github.com/fluree/db/releases/latest/download/fluree-db-cli-installer.sh | sh\n\n# 2. load a dataset, start the server, then run the suite\ncommon/run_benchmark.sh --endpoint http://localhost:8090/v1/fluree/query/dblp:main \\\n  -r 3 -w 1 -t 180 -o benchmarks/sparqloscope/reports/dblp-core/engines/fluree.tsv\n\n# 3. (re)generate a report and the headline charts\npython3 common/generate_report.py benchmarks/sparqloscope/reports/dblp-core/\npython3 common/make_charts.py\n```\n\n- **Native setup for every engine:** [`common/engine-setup/`](common/engine-setup/)\n  ([Fluree](common/engine-setup/fluree.md) ·\n  [QLever](common/engine-setup/qlever.md) ·\n  [Virtuoso](common/engine-setup/virtuoso.md) ·\n  [MillenniumDB](common/engine-setup/millenniumdb.md) ·\n  [Jena](common/engine-setup/jena.md) ·\n  [Oxigraph](common/engine-setup/oxigraph.md) ·\n  [Blazegraph](common/engine-setup/blazegraph.md))\n- **Query runner:** [`common/run_benchmark.sh`](common/run_benchmark.sh) —\n  warmup + median-of-N, per-query timeout/budget, body or form POST.\n- **Report + chart generators:** [`common/generate_report.py`](common/generate_report.py),\n  [`common/summarize.py`](common/summarize.py), [`common/make_charts.py`](common/make_charts.py).\n\n## Methodology notes\n\n- **Native, not Docker** — containerization distorts results (per the SPARQLoscope paper).\n- **No warm result cache** — each engine's result cache is disabled or cleared per query,\n  so every timed run re-executes (stricter than the paper's warm-cache protocol).\n- **1 warmup + median of 3 runs**, per-query timeout (180 s for DBLP-core, 300 s for the\n  billion-scale SPARQLoscope runs, 120 s for WGPB).\n- **Engine-vs-engine on one box per dataset** — absolute times are box-specific and not\n  bit-comparable to the published SPARQLoscope table (different dumps/dates). See each\n  report's caveats for the precise dataset version, deviations, and per-engine notes.\n\n## Repo layout\n\n```\nbenchmarks/\n  sparqloscope/\n    queries/            105 SPARQL 1.1 query files\n    datasets/           per-dataset source/version/checksum notes\n    reports/\n      dblp-core/        7-engine same-box run (REPORT.md, meta.json, engines/*.tsv,\n                        fluree-scaling/)\n      wikidata-truthy/  8.19 B-triple 5-engine run (Blazegraph excluded)\n  wgpb/\n    queries/            850 WGPB basic-graph-pattern queries (17 shapes x 50)\n    reports/\n      wikidata-all/     21.5 B-triple full all-dump run (Fluree)\ncommon/\n  run_benchmark.sh      generic SPARQL benchmark runner\n  generate_report.py    meta.json + engines/*.tsv -\u003e REPORT.md\n  summarize.py          raw TSV -\u003e per-query summary\n  make_charts.py        headline SVG charts (this README)\n  engine-setup/         native install/load/serve notes per engine\nassets/                 generated charts\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluree%2Fbenchmark-db","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffluree%2Fbenchmark-db","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluree%2Fbenchmark-db/lists"}