{"id":24311688,"url":"https://github.com/Query-farm/datasketches","last_synced_at":"2025-09-26T17:31:22.940Z","repository":{"id":270528575,"uuid":"906717928","full_name":"rustyconover/duckdb-datasketches","owner":"rustyconover","description":"A DuckDB extension for Apache DataSketches","archived":false,"fork":false,"pushed_at":"2025-01-02T17:29:44.000Z","size":383,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-08T12:57:23.868Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rustyconover.png","metadata":{"files":{"readme":"docs/README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-21T17:43:13.000Z","updated_at":"2025-01-06T15:22:58.000Z","dependencies_parsed_at":"2025-01-01T01:46:32.148Z","dependency_job_id":null,"html_url":"https://github.com/rustyconover/duckdb-datasketches","commit_stats":null,"previous_names":["rustyconover/duckdb-datasketches"],"tags_count":0,"template":false,"template_full_name":"duckdb/extension-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyconover%2Fduckdb-datasketches","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyconover%2Fduckdb-datasketches/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyconover%2Fduckdb-datasketches/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyconover%2Fduckdb-datasketches/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rustyconover","download_url":"https://codeload.github.com/rustyconover/duckdb-datasketches/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234325223,"owners_count":18814395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-17T07:13:55.163Z","updated_at":"2025-09-26T17:31:22.591Z","avatar_url":"https://github.com/rustyconover.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache DataSketches for DuckDB\n\n![Ducks creating sketches](./duck-sketches-2.jpg)\n\nThe DuckDB DataSketches Extension is an extension for [DuckDB](https://duckdb.org) that provides an interface to the [Apache DataSketches](https://datasketches.apache.org/) library. This extension enables users to efficiently compute approximate results for large datasets directly within DuckDB, using state-of-the-art streaming algorithms for distinct counting, quantile estimation, and more.\n\n## Why use this extension?\n\nDuckDB already has great implementations of [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog)  via `approx_count_distinct(x)`  and [TDigest](https://arxiv.org/abs/1902.04023) via `approx_quantile(x, pos)`, but it doesn't expose the internal state of the aggregates nor allow the the user to tune all of the parameters of the sketches.  This extension allows data sketches to be serialized as `BLOB`s which can be stored and shared across different systems, processes, and environments without loss of fidelity. This makes data sketches highly useful in distributed data processing pipelines.\n\nThis extension has implemented these sketches from Apache DataSketches.\n\n- Quantile Estimation\n  - [TDigest](https://datasketches.apache.org/docs/tdigest/tdigest.html)\n  - [Classic Quantile](https://datasketches.apache.org/docs/Quantiles/ClassicQuantilesSketch.html)\n  - [Relative Error Quantile (REQ)](https://datasketches.apache.org/docs/REQ/ReqSketch.html)\n  - [KLL](https://datasketches.apache.org/docs/KLL/KLLSketch.html)\n- Approximate Distinct Count\n  - [Compressed Probability Counting (CPC)](https://datasketches.apache.org/docs/CPC/CpcSketches.html)\n  - [HyperLogLog (HLL)](https://datasketches.apache.org/docs/HLL/HllSketches.html)\n\nAdditional sketch types can be implemented as needed.\n\n## Installation\n\nYou can install this extension by executing this SQL:\n\n```sql\ninstall datasketches from community;\nload datasketches;\n```\n\n## Features\n\n### Quantile Estimation\n\nThese sketches estimatate a specific quantile (or percentile) and ranks of a distribution or dataset while being memory-efficient.\n\n#### TDigest - \"`tdigest`\"\n\nThis implementation is based on the following paper:\nTed Dunning, Otmar Ertl. Extremely Accurate Quantiles Using t-Digests and the following implementation in Java\n[https://github.com/tdunning/t-digest](https://github.com/tdunning/t-digest). This implementation is similar to MergingDigest in the above Java implementation\n\nThe values that can be aggregated by this sketch are: `DOUBLE` and `FLOAT`.\n\nThe TDigest sketch is returned as a type `sketch_tdigest_[type]` which is equal to a BLOB.\n\n```sql\n-- Lets simulate a temperature sensor\nCREATE TABLE readings(temp integer);\n\nINSERT INTO readings(temp) select unnest(generate_series(1, 10));\n\n-- Create a sketch by aggregating id over the readings table.\nSELECT datasketch_tdigest_rank(datasketch_tdigest(10, temp), 5) from readings;\n┌────────────────────────────────────────────────────────────┐\n│ datasketch_tdigest_rank(datasketch_tdigest(10, \"temp\"), 5) │\n│                           double                           │\n├────────────────────────────────────────────────────────────┤\n│                                                       0.45 │\n└────────────────────────────────────────────────────────────┘\n\n-- Put some more readings in at the high end.\nINSERT INTO readings(temp) values (10), (10), (10), (10);\n\n-- Now the rank of 5 is moved down.\nSELECT datasketch_tdigest_rank(datasketch_tdigest(10, temp), 5) from readings;\n┌────────────────────────────────────────────────────────────┐\n│ datasketch_tdigest_rank(datasketch_tdigest(10, \"temp\"), 5) │\n│                           double                           │\n├────────────────────────────────────────────────────────────┤\n│                                        0.32142857142857145 │\n└────────────────────────────────────────────────────────────┘\n\n-- Lets get the cumulative distribution function from the sketch.\nSELECT datasketch_tdigest_cdf(datasketch_tdigest(10, temp), [1,5,9]) from readings;\n┌──────────────────────────────────────────────────────────────────────────────────┐\n│ datasketch_tdigest_cdf(datasketch_tdigest(10, \"temp\"), main.list_value(1, 5, 9)) │\n│                                     double[]                                     │\n├──────────────────────────────────────────────────────────────────────────────────┤\n│ [0.03571428571428571, 0.32142857142857145, 0.6071428571428571, 1.0]              │\n└──────────────────────────────────────────────────────────────────────────────────┘\n\n\n-- The sketch can be persisted and updated later when more data\n-- arrives without having to rescan the previously aggregated data.\nSELECT datasketch_tdigest(10, temp) from readings;\ndatasketch_tdigest(10, \"temp\") = \\x02\\x01\\x14\\x0A\\x00\\x04\\x00...\n```\n\n##### Aggregate Functions\n\n**`datasketch_tdigest(INTEGER, DOUBLE | FLOAT | sketch_tdigest) -\u003e sketch_tdigest_float | sketch_tdigest_double`**\n\nThe first argument is the base two logarithm of the number of bins in the sketch, which affects memory used. The second parameter is the value to aggregate into the sketch.\n\nThis same aggregate function can perform a union of multiple sketches.\n\n##### Scalar Functions\n\n**`datasketch_tdigest_rank(sketch_tdigest, value) -\u003e DOUBLE`**\n\nCompute approximate normalized rank of the given value.\n\n-----\n\n**`datasketch_tdigest_quantile(sketch_tdigest, DOUBLE) -\u003e DOUBLE`**\n\nCompute approximate quantile value corresponding to the given normalized rank\n\n-----\n\n\n**`datasketch_tdigest_pmf(sketch_tdigest, value[]) -\u003e double[]`**\n\nReturns an approximation to the Probability Mass Function (PMF) of the input stream given a set of split points.\n\nThe returned value is a list of \u003ci\u003em\u003c/i\u003e+1 doubles each of which is an approximation to the fraction of the input stream values (the mass) that fall into one of those intervals.\n\n-----\n\n**`datasketch_tdigest_cdf(sketch_tdigest, value[]) -\u003e double[]`**\n\nReturns an approximation to the Cumulative Distribution Function (CDF), which is the cumulative analog of the PMF, of the input stream given a set of split points.\n\nThe second argument is a list of of \u003ci\u003em\u003c/i\u003e unique, monotonically increasing values that divide the input domain into \u003ci\u003em+1\u003c/i\u003e consecutive disjoint intervals.\n\nThe returned value is a list of \u003ci\u003em\u003c/i\u003e+1 doubles, which are a consecutive approximation to the CDF of the input stream given the split_points. The value at array position j of the returned CDF array is the sum of the returned values in positions 0 through j of the returned PMF array. This can be viewed as array of ranks of the given split points plus\none more value that is always 1.\n\n-----\n\n**`datasketch_tdigest_k(sketch_tdigest) -\u003e USMALLINT`**\n\nReturn the value of K for the passed sketch.\n\n-----\n\n**`datasketch_tdigest_is_empty(sketch_tdigest) -\u003e BOOLEAN`**\n\nReturns if the sketch is empty.\n\n\n#### Quantile - \"`quantile`\"\n\nThis is an implementation of the Low Discrepancy Mergeable Quantiles Sketch\ndescribed in section 3.2 of the journal version of the paper [\"Mergeable Summaries\"\nby Agarwal, Cormode, Huang, Phillips, Wei, and Yi](http://dblp.org/rec/html/journals/tods/AgarwalCHPWY13).\n\nThe values that can be aggregated by this sketch are:\n\n* `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT`, `FLOAT`, `DOUBLE`, `UTINYINT`, `USMALLINT`, `UINTEGER`, `UBIGINT`\n\nThe Quantile sketch is returned as a type `sketch_quantiles_[type]` which is equal to a BLOB.\n\nThis algorithm is independent of the distribution of items and requires only that the items be comparable.\n\nThis algorithm intentionally inserts randomness into the sampling process for items that\nultimately get retained in the sketch. The results produced by this algorithm are not\ndeterministic. For example, if the same stream is inserted into two different instances\nof this sketch, the answers obtained from the two sketches may not be identical.\n\nSimilarly, there may be directional inconsistencies. For example, the result\nobtained from `datasketches_quantile_quantile` input into the reverse directional query\n`datasketches_quantile_rank` may not result in the original item.\n\nThe accuracy of this sketch is a function of the configured value \u003ci\u003ek\u003c/i\u003e, which also affects\nthe overall size of the sketch. Accuracy of this quantile sketch is always with respect to\nthe normalized rank. A \u003ci\u003ek\u003c/i\u003e of 128 produces a normalized, rank error of about 1.7%.\nFor example, the median item returned from `datasketches_quantile_quantile(0.5)` will be between the actual items\nfrom the hypothetically sorted array of input items at normalized ranks of 0.483 and 0.517, with\na confidence of about 99%.\n\n\u003cpre\u003e\nTable Guide for DoublesSketch Size in Bytes and Approximate Error:\n          K =\u0026gt; |      16      32      64     128     256     512   1,024\n    ~ Error =\u0026gt; | 12.145%  6.359%  3.317%  1.725%  0.894%  0.463%  0.239%\n             N | Size in Bytes -\u0026gt;\n------------------------------------------------------------------------\n             0 |       8       8       8       8       8       8       8\n             1 |      72      72      72      72      72      72      72\n             3 |      72      72      72      72      72      72      72\n             7 |     104     104     104     104     104     104     104\n            15 |     168     168     168     168     168     168     168\n            31 |     296     296     296     296     296     296     296\n            63 |     424     552     552     552     552     552     552\n           127 |     552     808   1,064   1,064   1,064   1,064   1,064\n           255 |     680   1,064   1,576   2,088   2,088   2,088   2,088\n           511 |     808   1,320   2,088   3,112   4,136   4,136   4,136\n         1,023 |     936   1,576   2,600   4,136   6,184   8,232   8,232\n         2,047 |   1,064   1,832   3,112   5,160   8,232  12,328  16,424\n         4,095 |   1,192   2,088   3,624   6,184  10,280  16,424  24,616\n         8,191 |   1,320   2,344   4,136   7,208  12,328  20,520  32,808\n        16,383 |   1,448   2,600   4,648   8,232  14,376  24,616  41,000\n        32,767 |   1,576   2,856   5,160   9,256  16,424  28,712  49,192\n        65,535 |   1,704   3,112   5,672  10,280  18,472  32,808  57,384\n       131,071 |   1,832   3,368   6,184  11,304  20,520  36,904  65,576\n       262,143 |   1,960   3,624   6,696  12,328  22,568  41,000  73,768\n       524,287 |   2,088   3,880   7,208  13,352  24,616  45,096  81,960\n     1,048,575 |   2,216   4,136   7,720  14,376  26,664  49,192  90,152\n     2,097,151 |   2,344   4,392   8,232  15,400  28,712  53,288  98,344\n     4,194,303 |   2,472   4,648   8,744  16,424  30,760  57,384 106,536\n     8,388,607 |   2,600   4,904   9,256  17,448  32,808  61,480 114,728\n    16,777,215 |   2,728   5,160   9,768  18,472  34,856  65,576 122,920\n    33,554,431 |   2,856   5,416  10,280  19,496  36,904  69,672 131,112\n    67,108,863 |   2,984   5,672  10,792  20,520  38,952  73,768 139,304\n   134,217,727 |   3,112   5,928  11,304  21,544  41,000  77,864 147,496\n   268,435,455 |   3,240   6,184  11,816  22,568  43,048  81,960 155,688\n   536,870,911 |   3,368   6,440  12,328  23,592  45,096  86,056 163,880\n 1,073,741,823 |   3,496   6,696  12,840  24,616  47,144  90,152 172,072\n 2,147,483,647 |   3,624   6,952  13,352  25,640  49,192  94,248 180,264\n 4,294,967,295 |   3,752   7,208  13,864  26,664  51,240  98,344 188,456\n\u003c/pre\u003e\n\n```sql\n-- Lets simulate a temperature sensor\nCREATE TABLE readings(temp integer);\n\nINSERT INTO readings(temp) select unnest(generate_series(1, 10));\n\n-- Create a sketch by aggregating id over the readings table.\nSELECT datasketch_quantiles_rank(datasketch_quantiles(16, temp), 5, true) from readings;\n┌──────────────────────────────────────────────────────────────────────────────────────┐\n│ datasketch_quantiles_rank(datasketch_quantiles(16, \"temp\"), 5, CAST('t' AS BOOLEAN)) │\n│                                        double                                        │\n├──────────────────────────────────────────────────────────────────────────────────────┤\n│                                                                                  0.5 │\n└──────────────────────────────────────────────────────────────────────────────────────┘\n\n-- Put some more readings in at the high end.\nINSERT INTO readings(temp) values (10), (10), (10), (10);\n\n-- Now the rank of 5 is moved down.\nSELECT datasketch_quantiles_rank(datasketch_quantiles(16, temp), 5, true) from readings;\n┌──────────────────────────────────────────────────────────────────────────────────────┐\n│ datasketch_quantiles_rank(datasketch_quantiles(16, \"temp\"), 5, CAST('t' AS BOOLEAN)) │\n│                                        double                                        │\n├──────────────────────────────────────────────────────────────────────────────────────┤\n│                                                                  0.35714285714285715 │\n└──────────────────────────────────────────────────────────────────────────────────────┘\n\n-- Lets get the cumulative distribution function from the sketch.\nSELECT datasketch_quantiles_cdf(datasketch_quantiles(16, temp), [1,5,9], true) from readings;\n┌────────────────────────────────────────────────────────────────────────────────────────────────────────────┐\n│ datasketch_quantiles_cdf(datasketch_quantiles(16, \"temp\"), main.list_value(1, 5, 9), CAST('t' AS BOOLEAN)) │\n│                                                  int32[]                                                   │\n├────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n│ [0, 0, 0, 1]                                                                                               │\n└────────────────────────────────────────────────────────────────────────────────────────────────────────────┘\n\n-- The sketch can be persisted and updated later when more data\n-- arrives without having to rescan the previously aggregated data.\nSELECT datasketch_quantiles(16, temp) from readings;\ndatasketch_quantiles(16, \"temp\") = \\x02\\x03\\x08\\x18\\x10\\x00\\x00...\n```\n\n##### Aggregate Functions\n\n**`datasketch_quantiles(INTEGER, DOUBLE | FLOAT | sketch_quantiles) -\u003e sketch_quantiles_[type]`**\n\nThe first argument is the the value of K for the sketch.\n\nThis same aggregate function can perform a union of multiple sketches.\n\n##### Scalar Functions\n\n**`datasketch_quantiles_rank(sketch_quantiles, value, BOOLEAN) -\u003e DOUBLE`**\n\nReturns an approximation to the normalized rank of the given item from 0 to 1, inclusive.\n\nThe third argument if true means the weight of the given item is included into the rank.\n\n-----\n\n**`datasketch_quantiles_quantile(sketch_quantiles, DOUBLE) -\u003e DOUBLE`**\n\nReturns an approximation to the data item associated with the given rank\nof a hypothetical sorted version of the input stream so far.\n\nThe third argument if true means the weight of the given item is included into the rank.\n\n-----\n\n\n**`datasketch_quantiles_pmf(sketch_quantiles, value[], BOOLEAN) -\u003e double[]`**\n\nReturns an approximation to the Probability Mass Function (PMF) of the input stream\ngiven a set of split points (items).\n\nThe resulting approximations have a probabilistic guarantee that can be obtained from the\n`datasketch_quantiles_normalized_rank_error(sketch, true)` function.\n\nThe second argument is a list of \u003ci\u003em\u003c/i\u003e unique, monotonically increasing items\nthat divide the input domain into \u003ci\u003em+1\u003c/i\u003e consecutive disjoint intervals (bins).\n\nThe third argument if true the rank of an item includes its own weight, and therefore\nif the sketch contains items equal to a split point, then in PMF such items are\nincluded into the interval to the left of split point. Otherwise they are included into the interval\nto the right of split point.\n\n-----\n\n**`datasketch_quantiles_cdf(sketch_quantiles, value[], BOOLEAN) -\u003e double[]`**\n\nReturns an approximation to the Cumulative Distribution Function (CDF), which is the\ncumulative analog of the PMF, of the input stream given a set of split points (items).\n\nThe second argument is a list of \u003ci\u003em\u003c/i\u003e unique, monotonically increasing items\nthat divide the input domain into \u003ci\u003em+1\u003c/i\u003e consecutive disjoint intervals.\n\nThe third argument if true the rank of an item includes its own weight, and therefore\nif the sketch contains items equal to a split point, then in PMF such items are\nincluded into the interval to the left of split point. Otherwise they are included into the interval\nto the right of split point.\n\nThe reesult is an array of m+1 double values, which are a consecutive approximation to the CDF\nof the input stream given the split_points. The value at array position j of the returned\nCDF array is the sum of the returned values in positions 0 through j of the returned PMF\narray. This can be viewed as array of ranks of the given split points plus one more value\nthat is always 1.\n\n-----\n\n**`datasketch_quantiles_k(sketch_quantiles) -\u003e USMALLINT`**\n\nReturn the value of K for the passed sketch.\n\n-----\n\n**`datasketch_quantiles_is_empty(sketch_quantiles) -\u003e BOOLEAN`**\n\nReturns if the sketch is empty.\n\n-----\n\n**`datasketch_quantiles_n(sketch_quantiles) -\u003e UBIGINT`**\n\nReturn the length of the input stream\n\n-----\n\n**`datasketch_quantiles_is_estimation_mode(sketch_quantiles) -\u003e BOOLEAN`**\n\nReturn if the sketch is in estimation mode\n\n-----\n\n**`datasketch_quantiles_num_retained(sketch_quantiles) -\u003e BOOLEAN`**\n\nReturn the number of items in the sketch\n\n-----\n\n**`datasketch_quantiles_min_item(sketch_quantiles) -\u003e value`**\n\nReturn the smallest item in the sketch.\n\n-----\n\n**`datasketch_quantiles_max_item(sketch_quantiles) -\u003e value`**\n\nReturn the largest item in the sketch.\n\n-----\n\n**`datasketch_quantiles_normalized_rank_error(sketch_quantiles, BOOLEAN) -\u003e DOUBLE`**\n\nGets the normalized rank error for this sketch. Constants were derived as the best fit to 99 percentile\nempirically measured max error in thousands of trials.\n\nThe second argument if true returns the \"double-sided\" normalized rank error.\nOtherwise, it is the \"single-sided\" normalized rank error for all the other queries.\n\n\n#### KLL - \"`kll`\"\n\nImplementation of a very compact quantiles sketch with lazy compaction scheme\nand nearly optimal accuracy per retained item.\nSee [Optimal Quantile Approximation in Streams](https://arxiv.org/abs/1603.05346v2).\n\nThis is a stochastic streaming sketch that enables near real-time analysis of the\napproximate distribution of items from a very large stream in a single pass, requiring only\nthat the items are comparable.\n\nThe values that can be aggregated by this sketch are:\n\n* `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT`, `FLOAT`, `DOUBLE`, `UTINYINT`, `USMALLINT`, `UINTEGER`, `UBIGINT`\n\nThe KLL sketch is returned as a type `sketch_kll_[type]` which is equal to a BLOB.\n\nThis sketch is configured with a parameter \u003ci\u003ek\u003c/i\u003e, which affects the size of the sketch and\nits estimation error.\n\nThe estimation error is commonly called \u003ci\u003eepsilon\u003c/i\u003e (or \u003ci\u003eeps\u003c/i\u003e) and is a fraction\nbetween zero and one. Larger values of \u003ci\u003ek\u003c/i\u003e result in smaller values of epsilon.\nEpsilon is always with respect to the rank and cannot be applied to the\ncorresponding items.\n\nThe \u003ci\u003ek\u003c/i\u003e of 200 yields a \"single-sided\" epsilon of about 1.33% and a\n\"double-sided\" (PMF) epsilon of about 1.65%.\n\n```sql\n-- Lets simulate a temperature sensor\nCREATE TABLE readings(temp integer);\n\nINSERT INTO readings(temp) select unnest(generate_series(1, 10));\n\n-- Create a sketch by aggregating id over the readings table.\nSELECT datasketch_kll_rank(datasketch_kll(16, temp), 5, true) from readings;\n┌──────────────────────────────────────────────────────────────────────────┐\n│ datasketch_kll_rank(datasketch_kll(16, \"temp\"), 5, CAST('t' AS BOOLEAN)) │\n│                                  double                                  │\n├──────────────────────────────────────────────────────────────────────────┤\n│                                                                      0.5 │\n└──────────────────────────────────────────────────────────────────────────┘\n\n-- Put some more readings in at the high end.\nINSERT INTO readings(temp) values (10), (10), (10), (10);\n\n-- Now the rank of 5 is moved down.\nSELECT datasketch_kll_rank(datasketch_kll(16, temp), 5, true) from readings;\n┌──────────────────────────────────────────────────────────────────────────────────────┐\n│ datasketch_kll_rank(datasketch_kll(16, \"temp\"), 5, CAST('t' AS BOOLEAN)) │\n│                                        double                                        │\n├──────────────────────────────────────────────────────────────────────────────────────┤\n│                                                                  0.35714285714285715 │\n└──────────────────────────────────────────────────────────────────────────────────────┘\n\n-- Lets get the cumulative distribution function from the sketch.\nSELECT datasketch_kll_cdf(datasketch_kll(16, temp), [1,5,9], true) from readings;\n┌────────────────────────────────────────────────────────────────────────────────────────────────┐\n│ datasketch_kll_cdf(datasketch_kll(16, \"temp\"), main.list_value(1, 5, 9), CAST('t' AS BOOLEAN)) │\n│                                            int32[]                                             │\n├────────────────────────────────────────────────────────────────────────────────────────────────┤\n│ [0, 0, 0, 1]                                                                                   │\n└────────────────────────────────────────────────────────────────────────────────────────────────┘\n\n-- The sketch can be persisted and updated later when more data\n-- arrives without having to rescan the previously aggregated data.\nSELECT datasketch_kll(16, temp) from readings;\ndatasketch_kll(16, \"temp\") = \\x05\\x01\\x0F\\x00\\x10\\x00\\x08\\x00\\x0E\\x00\\x00\\x0\n```\n\n##### Aggregate Functions\n\n**`datasketch_kll(INTEGER, DOUBLE | FLOAT | sketch_kll) -\u003e sketch_kll_[type]`**\n\nThe first argument is the the value of K for the sketch.\n\nThis same aggregate function can perform a union of multiple sketches.\n\n##### Scalar Functions\n\n**`datasketch_kll_rank(sketch_kll, value, BOOLEAN) -\u003e DOUBLE`**\n\nReturns an approximation to the normalized rank of the given item from 0 to 1, inclusive.\n\nThe third argument if true means the weight of the given item is included into the rank.\n\n-----\n\n**`datasketch_kll_kll(sketch_kll, DOUBLE) -\u003e DOUBLE`**\n\nReturns an approximation to the data item associated with the given rank\nof a hypothetical sorted version of the input stream so far.\n\nThe third argument if true means the weight of the given item is included into the rank.\n\n-----\n\n\n**`datasketch_kll_pmf(sketch_kll, value[], BOOLEAN) -\u003e double[]`**\n\nReturns an approximation to the Probability Mass Function (PMF) of the input stream\ngiven a set of split points (items).\n\nThe resulting approximations have a probabilistic guarantee that can be obtained from the\n`datasketch_kll_normalized_rank_error(sketch, true)` function.\n\nThe second argument is a list of \u003ci\u003em\u003c/i\u003e unique, monotonically increasing items\nthat divide the input domain into \u003ci\u003em+1\u003c/i\u003e consecutive disjoint intervals (bins).\n\nThe third argument if true the rank of an item includes its own weight, and therefore\nif the sketch contains items equal to a split point, then in PMF such items are\nincluded into the interval to the left of split point. Otherwise they are included into the interval\nto the right of split point.\n\n-----\n\n**`datasketch_kll_cdf(sketch_kll, value[], BOOLEAN) -\u003e double[]`**\n\nReturns an approximation to the Cumulative Distribution Function (CDF), which is the\ncumulative analog of the PMF, of the input stream given a set of split points (items).\n\nThe second argument is a list of \u003ci\u003em\u003c/i\u003e unique, monotonically increasing items\nthat divide the input domain into \u003ci\u003em+1\u003c/i\u003e consecutive disjoint intervals.\n\nThe third argument if true the rank of an item includes its own weight, and therefore\nif the sketch contains items equal to a split point, then in PMF such items are\nincluded into the interval to the left of split point. Otherwise they are included into the interval\nto the right of split point.\n\nThe reesult is an array of m+1 double values, which are a consecutive approximation to the CDF\nof the input stream given the split_points. The value at array position j of the returned\nCDF array is the sum of the returned values in positions 0 through j of the returned PMF\narray. This can be viewed as array of ranks of the given split points plus one more value\nthat is always 1.\n\n-----\n\n**`datasketch_kll_k(sketch_kll) -\u003e USMALLINT`**\n\nReturn the value of K for the passed sketch.\n\n-----\n\n**`datasketch_kll_is_empty(sketch_kll) -\u003e BOOLEAN`**\n\nReturns if the sketch is empty.\n\n-----\n\n**`datasketch_kll_n(sketch_kll) -\u003e UBIGINT`**\n\nReturn the length of the input stream\n\n-----\n\n**`datasketch_kll_is_estimation_mode(sketch_kll) -\u003e BOOLEAN`**\n\nReturn if the sketch is in estimation mode\n\n-----\n\n**`datasketch_kll_num_retained(sketch_kll) -\u003e BOOLEAN`**\n\nReturn the number of items in the sketch\n\n-----\n\n**`datasketch_kll_min_item(sketch_kll) -\u003e value`**\n\nReturn the smallest item in the sketch.\n\n-----\n\n**`datasketch_kll_max_item(sketch_kll) -\u003e value`**\n\nReturn the largest item in the sketch.\n\n-----\n\n**`datasketch_kll_normalized_rank_error(sketch_kll, BOOLEAN) -\u003e DOUBLE`**\n\nGets the normalized rank error for this sketch. Constants were derived as the best fit to 99 percentile\nempirically measured max error in thousands of trials.\n\nThe second argument if true returns the \"double-sided\" normalized rank error.\nOtherwise, it is the \"single-sided\" normalized rank error for all the other queries.\n\n\n\n#### Relative Error Quantile - \"`req`\"\n\nThis is an implementation based on the paper\n[\"Relative Error Streaming Quantiles\" by Graham Cormode, Zohar Karnin, Edo Liberty,\nJustin Thaler, Pavel Veselý](https://arxiv.org/abs/2004.01668), and loosely derived from a Python prototype written by Pavel Veselý.\n\nThis implementation differs from the algorithm described in the paper in the following:\n\n\u003cul\u003e\n \u003cli\u003eThe algorithm requires no upper bound on the stream length.\n Instead, each relative-compactor counts the number of compaction operations performed\n so far (via variable state). Initially, the relative-compactor starts with INIT_NUMBER_OF_SECTIONS.\n Each time the number of compactions (variable state) exceeds 2^{numSections - 1}, we double\n numSections. Note that after merging the sketch with another one variable state may not correspond\n to the number of compactions performed at a particular level, however, since the state variable\n never exceeds the number of compactions, the guarantees of the sketch remain valid.\u003c/li\u003e\n\n \u003cli\u003eThe size of each section (variable k and section_size in the code and parameter k in\n the paper) is initialized with a number set by the user via variable k.\n When the number of sections doubles, we decrease section_size by a factor of sqrt(2).\n This is applied at each level separately. Thus, when we double the number of sections, the\n nominal compactor size increases by a factor of approx. sqrt(2) (+/- rounding).\u003c/li\u003e\n\n \u003cli\u003eThe merge operation here does not perform \"special compactions\", which are used in the paper\n to allow for a tight mathematical analysis of the sketch.\u003c/li\u003e\n \u003c/ul\u003e\n\nThe values that can be aggregated by this sketch are:\n\n* `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT`, `FLOAT`, `DOUBLE`, `UTINYINT`, `USMALLINT`, `UINTEGER`, `UBIGINT`\n\nThe REQ sketch is returned as a type `sketch_req_[type]` which is equal to a BLOB.\n\n\n\nThis sketch is configured with a parameter \u003ci\u003ek\u003c/i\u003e, which controls the size and error of the sketch.\nIt must be even and in the range [4, 1024], inclusive. Value of 12 roughly corresponds to 1% relative\nerror guarantee at 95% confidence.\n\n\n```sql\n-- Lets simulate a temperature sensor\nCREATE TABLE readings(temp integer);\n\nINSERT INTO readings(temp) select unnest(generate_series(1, 10));\n\n-- Create a sketch by aggregating id over the readings table.\nSELECT datasketch_req_rank(datasketch_req(16, temp), 5, true) from readings;\n┌──────────────────────────────────────────────────────────────────────────┐\n│ datasketch_req_rank(datasketch_req(16, \"temp\"), 5, CAST('t' AS BOOLEAN)) │\n│                                  double                                  │\n├──────────────────────────────────────────────────────────────────────────┤\n│                                                                      0.5 │\n└──────────────────────────────────────────────────────────────────────────┘\n\n-- Put some more readings in at the high end.\nINSERT INTO readings(temp) values (10), (10), (10), (10);\n\n-- Now the rank of 5 is moved down.\nSELECT datasketch_req_rank(datasketch_req(16, temp), 5, true) from readings;\n┌──────────────────────────────────────────────────────────────────────────────────────┐\n│ datasketch_req_rank(datasketch_req(16, \"temp\"), 5, CAST('t' AS BOOLEAN)) │\n│                                        double                                        │\n├──────────────────────────────────────────────────────────────────────────────────────┤\n│                                                                  0.35714285714285715 │\n└──────────────────────────────────────────────────────────────────────────────────────┘\n\n-- Lets get the cumulative distribution function from the sketch.\nSELECT datasketch_req_cdf(datasketch_req(16, temp), [1,5,9], true) from readings;\n┌────────────────────────────────────────────────────────────────────────────────────────────────┐\n│ datasketch_req_cdf(datasketch_req(16, \"temp\"), main.list_value(1, 5, 9), CAST('t' AS BOOLEAN)) │\n│                                            int32[]                                             │\n├────────────────────────────────────────────────────────────────────────────────────────────────┤\n│ [0, 0, 0, 1]                                                                                   │\n└────────────────────────────────────────────────────────────────────────────────────────────────┘\n\n-- The sketch can be persisted and updated later when more data\n-- arrives without having to rescan the previously aggregated data.\nSELECT datasketch_req(16, temp) from readings;\ndatasketch_req(16, \"temp\") =  \\x02\\x01\\x11\\x08\\x10\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x...\n```\n\n##### Aggregate Functions\n\n**`datasketch_req(INTEGER, DOUBLE | FLOAT | sketch_req) -\u003e sketch_req_[type]`**\n\nThe first argument is the the value of K for the sketch.\n\nThis same aggregate function can perform a union of multiple sketches.\n\n##### Scalar Functions\n\n**`datasketch_req_rank(sketch_req, value, BOOLEAN) -\u003e DOUBLE`**\n\nReturns an approximation to the normalized rank of the given item from 0 to 1, inclusive.\n\nThe third argument if true means the weight of the given item is included into the rank.\n\n-----\n\n**`datasketch_req_quantile(sketch_req, DOUBLE) -\u003e DOUBLE`**\n\nReturns an approximation to the data item associated with the given rank\nof a hypothetical sorted version of the input stream so far.\n\nThe third argument if true means the weight of the given item is included into the rank.\n\n-----\n\n\n**`datasketch_req_pmf(sketch_req, value[], BOOLEAN) -\u003e double[]`**\n\nReturns an approximation to the Probability Mass Function (PMF) of the input stream\ngiven a set of split points (items).\n\nThe resulting approximations have a probabilistic guarantee that can be obtained from the\n`datasketch_req_normalized_rank_error(sketch, true)` function.\n\nThe second argument is a list of \u003ci\u003em\u003c/i\u003e unique, monotonically increasing items\nthat divide the input domain into \u003ci\u003em+1\u003c/i\u003e consecutive disjoint intervals (bins).\n\nThe third argument if true the rank of an item includes its own weight, and therefore\nif the sketch contains items equal to a split point, then in PMF such items are\nincluded into the interval to the left of split point. Otherwise they are included into the interval\nto the right of split point.\n\n-----\n\n**`datasketch_req_cdf(sketch_req, value[], BOOLEAN) -\u003e double[]`**\n\nReturns an approximation to the Cumulative Distribution Function (CDF), which is the\ncumulative analog of the PMF, of the input stream given a set of split points (items).\n\nThe second argument is a list of \u003ci\u003em\u003c/i\u003e unique, monotonically increasing items\nthat divide the input domain into \u003ci\u003em+1\u003c/i\u003e consecutive disjoint intervals.\n\nThe third argument if true the rank of an item includes its own weight, and therefore\nif the sketch contains items equal to a split point, then in PMF such items are\nincluded into the interval to the left of split point. Otherwise they are included into the interval\nto the right of split point.\n\nThe reesult is an array of m+1 double values, which are a consecutive approximation to the CDF\nof the input stream given the split_points. The value at array position j of the returned\nCDF array is the sum of the returned values in positions 0 through j of the returned PMF\narray. This can be viewed as array of ranks of the given split points plus one more value\nthat is always 1.\n\n-----\n\n**`datasketch_req_k(sketch_req) -\u003e USMALLINT`**\n\nReturn the value of K for the passed sketch.\n\n-----\n\n**`datasketch_req_is_empty(sketch_req) -\u003e BOOLEAN`**\n\nReturns if the sketch is empty.\n\n-----\n\n**`datasketch_req_n(sketch_req) -\u003e UBIGINT`**\n\nReturn the length of the input stream\n\n-----\n\n**`datasketch_req_is_estimation_mode(sketch_req) -\u003e BOOLEAN`**\n\nReturn if the sketch is in estimation mode\n\n-----\n\n**`datasketch_req_num_retained(sketch_req) -\u003e BOOLEAN`**\n\nReturn the number of items in the sketch\n\n-----\n\n**`datasketch_req_min_item(sketch_req) -\u003e value`**\n\nReturn the smallest item in the sketch.\n\n-----\n\n**`datasketch_req_max_item(sketch_req) -\u003e value`**\n\nReturn the largest item in the sketch.\n\n-----\n\n**`datasketch_req_normalized_rank_error(sketch_req, BOOLEAN) -\u003e DOUBLE`**\n\nGets the normalized rank error for this sketch. Constants were derived as the best fit to 99 percentile\nempirically measured max error in thousands of trials.\n\nThe second argument if true returns the \"double-sided\" normalized rank error.\nOtherwise, it is the \"single-sided\" normalized rank error for all the other queries.\n\n### Approximate Distinct Count\n\nThese sketch type provide fast and memory-efficient cardinality estimation.\n\n#### HyperLogLog - \"`hll`\"\n\nThis sketch type contains a set of very compact implementations of Phillipe Flajolet’s HyperLogLog (HLL) but with significantly improved error behavior and excellent speed performance.\n\nIf the use case for sketching is primarily counting uniques and merging, the HyperLogLog sketch is the 2nd highest performing in terms of accuracy for storage space consumed (the new CPC sketch developed by Kevin J. Lang now beats it).\n\nNeither HLL nor CPC sketches provide means for set intersections or set differences.\n\nThe values that can be aggregated by the CPC sketch are:\n\n* `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT`, `FLOAT`, `DOUBLE`, `UTINYINT`, `USMALLINT`, `UINTEGER`, `UBIGINT`, `VARCHAR`, `BLOB`\n\nThe HLL sketch is returned as a type `sketch_hll` which is equal to a BLOB.\n\n##### Example\n\n```sql\n-- This table will contain the items where we are interested in knowing\n-- how many unique item ids there are.\nCREATE TABLE items(id integer);\n\n-- Insert the same ids twice to demonstrate the sketch only counts distinct items.\nINSERT INTO items(id) select unnest(generate_series(1, 100000));\nINSERT INTO items(id) select unnest(generate_series(1, 100000));\n\n-- Create a sketch by aggregating id over the items table,\n-- use the smallest possible sketch by setting K to 8, better\n-- accuracy comes with larger values of K\nSELECT datasketch_hll_estimate(datasketch_hll(8, id)) from items;\n┌────────────────────────────────────────────────┐\n│ datasketch_hll_estimate(datasketch_hll(8, id)) │\n│                     double                     │\n├────────────────────────────────────────────────┤\n│                              99156.23039496985 │\n└────────────────────────────────────────────────┘\n\n-- The sketch can be persisted and updated later when more data\n-- arrives without having to rescan the previously aggregated data.\nSELECT datasketch_hll(8, id) from items;\ndatasketch_hll(8, id) = \\x0A\\x01\\x07\\x08\\x00\\x10\\x06\\x02\\x00\\x00\\x00\\x00\\x00\\x00...\n```\n\n##### Aggregate Functions\n\n**`datasketch_hll(INTEGER, HLL_SUPPORTED_TYPE) -\u003e sketch_hll`**\n\nThe first argument is the base two logarithm of the number of bins in the sketch, which affects memory used. The second parameter is the value to aggregate into the sketch.\n\n-----\n\n**`datasketch_hll_union(INTEGER, sketch_hll) -\u003e sketch_hll`**\n\nThe first argument is the base two logarithm of the number of bins in the sketch, which affects memory used. The second parameter is the sketch to aggregate via a union operation.\n\n##### Scalar Functions\n\n**`datasketch_hll_estimate(sketch_hll) -\u003e DOUBLE`**\n\nGet the estimated number of distinct elements seen by this sketch\n\n-----\n\n**`datasketch_hll_lower_bound(sketch_hll, integer) -\u003e DOUBLE`**\n\nReturns the approximate lower error bound given the specified number of standard deviations.\n\n-----\n\n**`datasketch_hll_upper_bound(sketch_hll, integer) -\u003e DOUBLE`**\n\nReturns the approximate lower error bound given the specified number of standard deviations.\n\n-----\n\n**`datasketch_hll_describe(sketch_hll) -\u003e VARCHAR`**\n\nReturns a human readable summary of the sketch.\n\n-----\n\n**`datasketch_hll_is_empty(sketch_hll) -\u003e BOOLEAN`**\n\nReturns if the sketch is empty.\n\n-----\n\n**`datasketch_hll_lg_config_k(sketch_hll) -\u003e UTINYINT`**\n\nReturns the base two logarithm for the number of bins in the sketch.\n\n#### Compressed Probability Counting - \"`cpc`\"\n\nThis is an implementations of [Kevin J. Lang’s CPC sketch1](https://arxiv.org/abs/1708.06839). The stored CPC sketch can consume about 40% less space than a HyperLogLog sketch of comparable accuracy. Nonetheless, the HLL and CPC sketches have been intentially designed to offer different tradeoffs so that, in fact, they complement each other in many ways.\n\nThe CPC sketch has better accuracy for a given stored size then HyperLogLog, HyperLogLog has faster serialization and deserialization times than CPC.\n\nSimilar to the HyperLogLog sketch, the primary use-case for the CPC sketch is for counting distinct values as a stream, and then merging multiple sketches together for a total distinct count\n\nNeither HLL nor CPC sketches provide means for set intersections or set differences.\n\nThe values that can be aggregated by the CPC sketch are:\n\n* `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT`, `FLOAT`, `DOUBLE`, `UTINYINT`, `USMALLINT`, `UINTEGER`, `UBIGINT`, `VARCHAR`, `BLOB`\n\nThe CPC sketch is returned as a type `sketch_cpc` which is equal to a BLOB.\n\n##### Example\n\n```sql\n-- This table will contain the items where we are interested in knowing\n-- how many unique item ids there are.\nCREATE TABLE items(id integer);\n\n-- Insert the same ids twice to demonstrate the sketch only counts distinct items.\nINSERT INTO items(id) select unnest(generate_series(1, 100000));\nINSERT INTO items(id) select unnest(generate_series(1, 100000));\n\n-- Create a sketch by aggregating id over the items table,\n-- use the smallest possible sketch by setting K to 4, better\n-- accuracy comes with larger values of K\nSELECT datasketch_cpc_estimate(datasketch_cpc(4, id)) from items;\n┌────────────────────────────────────────────────┐\n│ datasketch_cpc_estimate(datasketch_cpc(4, id)) │\n│                     double                     │\n├────────────────────────────────────────────────┤\n│                             104074.26344655872 │\n└────────────────────────────────────────────────┘\n\n-- The sketch can be persisted and updated later when more data\n-- arrives without having to rescan the previously aggregated data.\nSELECT datasketch_cpc(4, id) from items;\ndatasketch_cpc(4, id) = \\x04\\x01\\x10\\x04\\x0A\\x12\\xCC\\x93\\xD0\\x00\\x00\\x00\\x03\\x00\\x00\\x00\\x0C]\\xAD\\x019\\x9B\\xFA\\x04+\\x00\\x00\\x00\n```\n\n##### Aggregate Functions\n\n**`datasketch_cpc(INTEGER, CPC_SUPPORTED_TYPE) -\u003e sketch_cpc`**\n\nThe first argument is the base two logarithm of the number of bins in the sketch, which affects memory used. The second parameter is the value to aggregate into the sketch.\n\n-----\n\n**`datasketch_cpc_union(INTEGER, sketch_cpc) -\u003e sketch_cpc`**\n\nThe first argument is the base two logarithm of the number of bins in the sketch, which affects memory used. The second parameter is the sketch to aggregate via a union operation.\n\n##### Scalar Functions\n\n**`datasketch_cpc_estimate(sketch_cpc) -\u003e DOUBLE`**\n\nGet the estimated number of distinct elements seen by this sketch\n\n-----\n\n**`datasketch_cpc_lower_bound(sketch_cpc, integer kappa) -\u003e DOUBLE`**\n\nReturns the approximate lower error bound given a parameter kappa (1, 2 or 3).\nThis parameter is similar to the number of standard deviations of the normal distribution and corresponds to approximately 67%, 95% and 99% confidence intervals.\n\n-----\n\n**`datasketch_cpc_upper_bound(sketch_cpc, integer kappa) -\u003e DOUBLE`**\n\nReturns the approximate upper error bound given a parameter kappa (1, 2 or 3).\nThis parameter is similar to the number of standard deviations of the normal distribution and corresponds to approximately 67%, 95% and 99% confidence intervals.\n\n-----\n\n**`datasketch_cpc_describe(sketch_cpc) -\u003e VARCHAR`**\n\nReturns a human readable summary of the sketch.\n\n-----\n\n**`datasketch_cpc_is_empty(sketch_cpc) -\u003e BOOLEAN`**\n\nReturns if the sketch is empty.\n\n\n## Building\n### Managing dependencies\nDuckDB extensions uses VCPKG for dependency management. Enabling VCPKG is very simple: follow the [installation instructions](https://vcpkg.io/en/getting-started) or just run the following:\n```shell\ngit clone https://github.com/Microsoft/vcpkg.git\n./vcpkg/bootstrap-vcpkg.sh\nexport VCPKG_TOOLCHAIN_PATH=`pwd`/vcpkg/scripts/buildsystems/vcpkg.cmake\n```\n\n### Build steps\nNow to build the extension, run:\n```sh\nmake\n```\nThe main binaries that will be built are:\n```sh\n./build/release/duckdb\n./build/release/test/unittest\n./build/release/extension/datasketches/datasketches.duckdb_extension\n```\n- `duckdb` is the binary for the duckdb shell with the extension code automatically loaded.\n- `unittest` is the test runner of duckdb. Again, the extension is already linked into the binary.\n- `datasketches.duckdb_extension` is the loadable binary as it would be distributed.\n\n## Running the extension\nTo run the extension code, simply start the shell with `./build/release/duckdb`.\n\n\n## Running the tests\nDifferent tests can be created for DuckDB extensions. The primary way of testing DuckDB extensions should be the SQL tests in `./test/sql`. These SQL tests can be run using:\n```sh\nmake test\n```\n\n### Installing the deployed binaries\nTo install your extension binaries from S3, you will need to do two things. Firstly, DuckDB should be launched with the\n`allow_unsigned_extensions` option set to true. How to set this will depend on the client you're using. Some examples:\n\nCLI:\n```shell\nduckdb -unsigned\n```\n\nPython:\n```python\ncon = duckdb.connect(':memory:', config={'allow_unsigned_extensions' : 'true'})\n```\n\nNodeJS:\n```js\ndb = new duckdb.Database(':memory:', {\"allow_unsigned_extensions\": \"true\"});\n```\n\nSecondly, you will need to set the repository endpoint in DuckDB to the HTTP url of your bucket + version of the extension\nyou want to install. To do this run the following SQL query in DuckDB:\n```sql\nSET custom_extension_repository='bucket.s3.eu-west-1.amazonaws.com/\u003cyour_extension_name\u003e/latest';\n```\nNote that the `/latest` path will allow you to install the latest extension version available for your current version of\nDuckDB. To specify a specific version, you can pass the version instead.\n\nAfter running these steps, you can install and load your extension using the regular INSTALL/LOAD commands in DuckDB:\n```sql\nINSTALL datasketches\nLOAD datasketches\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FQuery-farm%2Fdatasketches","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FQuery-farm%2Fdatasketches","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FQuery-farm%2Fdatasketches/lists"}