{"id":15350572,"url":"https://github.com/maropu/datasketches-spark","last_synced_at":"2025-08-19T15:09:57.455Z","repository":{"id":142186039,"uuid":"371224565","full_name":"maropu/datasketches-spark","owner":"maropu","description":"Data Sketches for Apache Spark","archived":false,"fork":false,"pushed_at":"2022-12-22T12:59:55.000Z","size":6410,"stargazers_count":22,"open_issues_count":2,"forks_count":4,"subscribers_count":5,"default_branch":"spark-3.2","last_synced_at":"2025-06-05T08:01:14.714Z","etag":null,"topics":["approximate-computing","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maropu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-27T02:33:42.000Z","updated_at":"2025-01-09T10:38:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"f0cdf778-5e50-4b2b-ba45-48f3aefa67fb","html_url":"https://github.com/maropu/datasketches-spark","commit_stats":{"total_commits":32,"total_committers":1,"mean_commits":32.0,"dds":0.0,"last_synced_commit":"50b3a613cb9518e786bd34d4187b07e510aacc94"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/maropu/datasketches-spark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maropu%2Fdatasketches-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maropu%2Fdatasketches-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maropu%2Fdatasketches-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maropu%2Fdatasketches-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maropu","download_url":"https://codeload.github.com/maropu/datasketches-spark/tar.gz/refs/heads/spark-3.2","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maropu%2Fdatasketches-spark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271173376,"owners_count":24711667,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-19T02:00:09.176Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["approximate-computing","spark"],"created_at":"2024-10-01T11:58:42.907Z","updated_at":"2025-08-19T15:09:57.430Z","avatar_url":"https://github.com/maropu.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License](http://img.shields.io/:license-Apache_v2-blue.svg)](https://github.com/maropu/datasketches-spark/blob/master/LICENSE)\n[![Build and test](https://github.com/maropu/datasketches-spark/workflows/Build%20and%20test/badge.svg)](https://github.com/maropu/datasketches-spark/actions?query=workflow%3A%22Build+and+test%22)\n\nThis repository provies Apache DataSketches experimental adapters for Apache Spark.\nPlease visit [the main website](https://datasketches.apache.org/) for more DataSketches information.\n\n## Quantile Sketches\n\nLike the built-in percentile estimation function (`approx_percentile`),\nthis plugin enalbes you to use an alternative function (`approx_percentile_ex`) to estimate percentiles\nin a theoretically-meageable and very compact way:\n\n```\n$ git clone https://github.com/maropu/datasketches-spark.git\n$ cd datasketches-spark\n$ ./bin/pyspark\n\nWelcome to\n      ____              __\n     / __/__  ___ _____/ /__\n    _\\ \\/ _ \\/ _ `/ __/  '_/\n   /__ / .__/\\_,_/_/ /_/\\_\\   version 3.2.0\n      /_/\n\nUsing Python version 3.7.11 (default, Jul 27 2021 07:03:16)\nSparkSession available as 'spark'.\nDataSketches APIs available as built-in functions.\n\n# This example uses the individual household electric power consumption data set in the UCI Machine Learning Repository:\n# - https://archive.ics.uci.edu/ml/datasets/Individual+household+electric+power+consumption\n\u003e\u003e\u003e df = spark.read.format(\"csv\").option(\"header\", True).option(\"sep\", \";\").load(\"household_power_consumption.txt\").selectExpr(\"to_date(Date, 'dd/MM/yyyy') AS Date\", \"CAST(Global_active_power AS double) Global_active_power\")\n\u003e\u003e\u003e df.show(5)\n+----------+-------------------+\n|      Date|Global_active_power|\n+----------+-------------------+\n|2006-12-16|              4.216|\n|2006-12-16|               5.36|\n|2006-12-16|              5.374|\n|2006-12-16|              5.388|\n|2006-12-16|              3.666|\n+----------+-------------------+\nonly showing top 5 rows\n\n\u003e\u003e\u003e df.describe().show(5, False)\n+-------+-------------------+\n|summary|Global_active_power|\n+-------+-------------------+\n|count  |2049280            |\n|mean   |1.0916150365005453 |\n|stddev |1.0572941610939872 |\n|min    |0.076              |\n|max    |11.122             |\n+-------+-------------------+\n\n\u003e\u003e\u003e df.selectExpr(\"percentile(Global_active_power, 0.95) percentile\", \"approx_percentile(Global_active_power, 0.95) approx_percentile\", \"approx_percentile_ex(Global_active_power, 0.95) approx_percentile_ex\").show()\n+----------+-----------------+--------------------+\n|percentile|approx_percentile|approx_percentile_ex|\n+----------+-----------------+--------------------+\n|     3.264|            3.264|                3.25|\n+----------+-----------------+--------------------+\n```\n\nMoreover, this plugin provies functionalities to accumulate quantile summaries for each time interval and\nestimate quantile values over specific intervals later just like [the Snowflake built-in functions](https://docs.snowflake.com/en/user-guide/querying-approximate-percentile-values.html):\n\n```\n\u003e\u003e\u003e import pyspark.sql.functions as f\n\u003e\u003e\u003e summaries = df.groupBy(f.window(\"Date\", \"1 week\")).agg(f.expr(\"approx_percentile_accumulate(Global_active_power) AS summaries\"))\n\u003e\u003e\u003e summaries.show(3, 50)\n+------------------------------------------+--------------------------------------------------+\n|                                    window|                                         summaries|\n+------------------------------------------+--------------------------------------------------+\n|{2006-12-14 09:00:00, 2006-12-21 09:00:00}|[04 01 11 28 0C 00 07 00 AA 1D 00 00 00 00 00 0...|\n|{2009-12-03 09:00:00, 2009-12-10 09:00:00}|[04 01 11 28 0C 00 05 00 9E 05 00 00 00 00 00 0...|\n|{2009-10-22 09:00:00, 2009-10-29 09:00:00}|[04 01 11 28 0C 00 07 00 60 27 00 00 00 00 00 0...|\n+------------------------------------------+--------------------------------------------------+\nonly showing top 3 rows\n\n# Correct percentile of the `Global_active_power` column\nscala\u003e df.where(\"Date between '2007-06-01' and '2010-01-01'\").selectExpr(\"percentile(Global_active_power, 0.95) correct\").show()\n+-------+\n|correct|\n+-------+\n|  3.236|\n+-------+\n\n# Estimated percentile of the `Global_active_power` column\n\u003e\u003e\u003e df = summaries.where(\"window.start \u003e '2007-06-01' and window.end \u003c '2010-01-01'\").selectExpr(\"approx_percentile_combine(summaries) merged\")\n\u003e\u003e\u003e df.selectExpr(\"approx_percentile_estimate(merged, 0.95) percentile\").show()\n+----------+\n| estimated|\n+----------+\n|      3.25|\n+----------+\n\n\u003e\u003e\u003e df.selectExpr(\"approx_pmf_estimate(merged, 4) pmf\").show(1, False)\n+--------------------------------------------------------------------------------------+\n|pmf                                                                                   |\n+--------------------------------------------------------------------------------------+\n|[0.9250280810398008, 0.07003322180158443, 0.004825778691690984, 1.1291846692380381E-4]|\n+--------------------------------------------------------------------------------------+\n```\n\n### Configurations\n\n| Property Name | Default | Meaning |\n| ---- | ---- | ---- |\n| spark.sql.dataSketches.quantiles.sketchImpl | REQ | A sketch implementation used in quantile estimation functions. |\n| spark.sql.dataSketches.quantiles.kll.k | 200 | Specifies the parameter `k` for the quantile sketch implementation named `KLL`, `KllFloatsSketch`. |\n| spark.sql.dataSketches.quantiles.req.k | 12 | Specifies the parameter `k` for the quantile sketch implementation named `REQ`, `ReqSketch`. |\n| spark.sql.dataSketches.quantiles.mergeable.k | 128 | Specifies the parameter `k` for the quantile sketch implementation named `MERGEABLE`, `DoubleSketch`. |\n\n## Frequent Item Sketches\n\nA class of “Heavy Hitters” algorithms enables you to approximately identify the “heaviest”\nor “most frequently occurring” items in an input column:\n\n```\n# This example uses the e-commerce data from UK retailer in the Kaggle data set:\n# - https://www.kaggle.com/carrie1/ecommerce-data\n\u003e\u003e\u003e df = spark.read.format(\"csv\").option(\"header\", True).load(\"data.csv\").selectExpr(\"Country\", \"Description\")\n\u003e\u003e\u003e df.show(5, False)\n+--------------+-----------------------------------+\n|Country       |Description                        |\n+--------------+-----------------------------------+\n|United Kingdom|WHITE HANGING HEART T-LIGHT HOLDER |\n|United Kingdom|WHITE METAL LANTERN                |\n|United Kingdom|CREAM CUPID HEARTS COAT HANGER     |\n|United Kingdom|KNITTED UNION FLAG HOT WATER BOTTLE|\n|United Kingdom|RED WOOLLY HOTTIE WHITE HEART.     |\n+--------------+-----------------------------------+\nonly showing top 5 rows\n\n\u003e\u003e\u003e df.selectExpr(\"count(Description)\", \"approx_count_distinct(Description)\").show()\n+------------------+----------------------------------+\n|count(Description)|approx_count_distinct(Description)|\n+------------------+----------------------------------+\n|            540455|                              4361|\n+------------------+----------------------------------+\n\n# Correct item counts of the `Description` column\n\u003e\u003e\u003e df.groupBy(\"Description\").count().orderBy(col(\"count\").desc()).show(7, False)\n+----------------------------------+-----+\n|Description                       |count|\n+----------------------------------+-----+\n|WHITE HANGING HEART T-LIGHT HOLDER|2369 |\n|REGENCY CAKESTAND 3 TIER          |2200 |\n|JUMBO BAG RED RETROSPOT           |2159 |\n|PARTY BUNTING                     |1727 |\n|LUNCH BAG RED RETROSPOT           |1638 |\n|ASSORTED COLOUR BIRD ORNAMENT     |1501 |\n|SET OF 3 CAKE TINS PANTRY DESIGN  |1473 |\n+----------------------------------+-----+\nonly showing top 7 rows\n\n# Estimated item counts of the `Description` column\n\u003e\u003e\u003e df.selectExpr(\"inline(approx_freqitems(Description))\").show(7, False)\n+----------------------------------+--------+\n|item                              |estimate|\n+----------------------------------+--------+\n|WHITE HANGING HEART T-LIGHT HOLDER|2369    |\n|REGENCY CAKESTAND 3 TIER          |2200    |\n|JUMBO BAG RED RETROSPOT           |2159    |\n|PARTY BUNTING                     |1752    |\n|LUNCH BAG RED RETROSPOT           |1638    |\n|SET OF 3 CAKE TINS PANTRY DESIGN  |1562    |\n|ASSORTED COLOUR BIRD ORNAMENT     |1504    |\n+----------------------------------+--------+\n```\n\nTo pre-compute summaries for each country and estimate frequent items in some of them,\nyou can use similar functions to the quantile sketch ones:\n\n```\n\u003e\u003e\u003e import pyspark.sql.functions as f\n\u003e\u003e\u003e summaries = df.groupBy(\"Country\").agg(f.expr(\"approx_freqitems_accumulate(Description) As summaries\"))\n\u003e\u003e\u003e summaries.show(3)\n+---------+--------------------+\n|  Country|           summaries|\n+---------+--------------------+\n|   Sweden|[04 01 0A 0A 09 0...|\n|Singapore|[04 01 0A 0A 08 0...|\n|  Germany|[04 01 0A 0A 0A 0...|\n+---------+--------------------+\nonly showing top 3 rows\n\n# Correct item counts of the `Description` column\n\u003e\u003e\u003e df.where(\"Country IN ('United Kingdom', 'Germany', 'Spain')\").selectExpr(\"inline(approx_freqitems(Description))\").show(10, False)\n+----------------------------------+--------+\n|item                              |estimate|\n+----------------------------------+--------+\n|WHITE HANGING HEART T-LIGHT HOLDER|2283    |\n|JUMBO BAG RED RETROSPOT           |2042    |\n|REGENCY CAKESTAND 3 TIER          |1965    |\n|PARTY BUNTING                     |1647    |\n|LUNCH BAG RED RETROSPOT           |1488    |\n|ASSORTED COLOUR BIRD ORNAMENT     |1439    |\n|SET OF 3 CAKE TINS PANTRY DESIGN  |1355    |\n|LUNCH BAG  BLACK SKULL.           |1308    |\n|NATURAL SLATE HEART CHALKBOARD    |1252    |\n|PACK OF 72 RETROSPOT CAKE CASES   |1240    |\n+----------------------------------+--------+\nonly showing top 10 rows\n\n# Estimated item counts of the `Description` column\n\u003e\u003e\u003e df = summaries.where(\"Country IN ('United Kingdom', 'Germany', 'Spain')\").selectExpr(\"approx_freqitems_combine(summaries) merged\")\n\u003e\u003e\u003e df.selectExpr(\"inline(approx_freqitems_estimate(merged))\").show(10, False)\n+----------------------------------+---------+\n|item                              |estimated|\n+----------------------------------+---------+\n|WHITE HANGING HEART T-LIGHT HOLDER|2292     |\n|JUMBO BAG RED RETROSPOT           |2042     |\n|REGENCY CAKESTAND 3 TIER          |1965     |\n|PARTY BUNTING                     |1678     |\n|LUNCH BAG RED RETROSPOT           |1488     |\n|ASSORTED COLOUR BIRD ORNAMENT     |1442     |\n|SET OF 3 CAKE TINS PANTRY DESIGN  |1437     |\n|PAPER CHAIN KIT 50'S CHRISTMAS    |1310     |\n|LUNCH BAG  BLACK SKULL.           |1309     |\n|SPOTTY BUNTING                    |1307     |\n+----------------------------------+---------+\n```\n\n### Configurations\n\n| Property Name | Default | Meaning |\n| ---- | ---- | ---- |\n| spark.sql.dataSketches.freqItems.maxMapSize | 1024 | Specifies the physical size of the internal hash map managed by this sketch and must be a power of 2. The maximum capacity of this internal hash map is 0.75 times * maxMapSize. Both the ultimate accuracy and size of this sketch are functions of maxMapSize. |\n\n## Distinct Count Sketches\n\nLike the built-in distinct count estimation function (`approx_count_distinct`),\nthis plugin enalbes you to use an alternative function (`approx_count_distinct_ex`) to estimate\nthe distinct number of an input column in a more precise way:\n\n```\n# This example uses the BitcoinHeist data set in the UCI Machine Learning Repository:\n# - https://archive.ics.uci.edu/ml/datasets/BitcoinHeistRansomwareAddressDataset\n\u003e\u003e\u003e df = spark.read.format(\"csv\").option(\"header\", True).load(\"BitcoinHeistData.csv\").selectExpr(\"year\", \"address\")\n\u003e\u003e\u003e df.show(5, False)\n+----+----------------------------------+\n|year|address                           |\n+----+----------------------------------+\n|2017|111K8kZAEnJg245r2cM6y9zgJGHZtJPy6 |\n|2016|1123pJv8jzeFQaCV4w644pzQJzVWay2zcA|\n|2016|112536im7hy6wtKbpH1qYDWtTyMRAcA2p7|\n|2016|1126eDRw2wqSkWosjTCre8cjjQW8sSeWH7|\n|2016|1129TSjKtx65E35GiUo4AYVeyo48twbrGX|\n+----+----------------------------------+\nonly showing top 5 rows\n\n\u003e\u003e\u003e df.selectExpr(\"count(address)\").show()\n+--------------+\n|count(address)|\n+--------------+\n|       2916697|\n+--------------+\n\n\u003e\u003e\u003e df.selectExpr(\"count(distinct address)\", \"approx_count_distinct(address)\", \"approx_count_distinct_ex(address)\").show()\n+-----------------------+------------------------------+---------------------------------+\n|count(DISTINCT address)|approx_count_distinct(address)|approx_count_distinct_ex(address)|\n+-----------------------+------------------------------+---------------------------------+\n|                2631095|                       2422325|                          2645708|\n+-----------------------+------------------------------+---------------------------------+\n```\n\nTo pre-compute summaries for each year and estimate the distinct count of addresses over specific years,\nyou can use similar functions to the other two sketch ones:\n\n```\n\u003e\u003e\u003e import pyspark.sql.functions as f\n\u003e\u003e\u003e summaries = df.groupBy(\"year\").agg(expr(\"approx_count_distinct_accumulate(address) AS summaries\"))\n\u003e\u003e\u003e summaries.show()\n+----+--------------------+\n|year|           summaries|\n+----+--------------------+\n|2016|[06 01 10 0B 04 1...|\n|2012|[06 01 10 0B 04 1...|\n|2017|[06 01 10 0B 04 1...|\n|2014|[06 01 10 0B 04 1...|\n|2013|[06 01 10 0B 04 1...|\n|2018|[06 01 10 0B 04 1...|\n|2011|[06 01 10 0B 04 1...|\n|2015|[06 01 10 0B 04 1...|\n+----+--------------------+\n\n# Correct distinct number of the `address` column\n\u003e\u003e\u003e df.where(\"year IN ('2014', '2015', '2016')\").selectExpr(\"count(distinct address) correct\").show()\n+--------+\n| correct|\n+--------+\n| 1057136|\n+--------+\n\n# Estimated distinct number of the `address` column\n\u003e\u003e\u003e val df = summaries.where(\"year IN ('2014', '2015', '2016')\").selectExpr(\"approx_count_distinct_combine(summaries) AS merged\")\n\u003e\u003e\u003e df.selectExpr(\"approx_count_distinct_estimate(merged) estimated\").show()\n+----------+\n| estimated|\n+----------+\n|   1063420|\n+----------+\n```\n\n### Configurations\n\n| Property Name | Default | Meaning |\n| ---- | ---- | ---- |\n| spark.sql.dataSketches.distinctCnt.sketchImpl | CPC | A sketch implementation used in distinct count estimation functions. |\n| spark.sql.dataSketches.distinctCnt.cpc.lgK | 11 | Specifies the parameter `lgK` for the distinct count sketch implementation named `CPC`, `CpcSketch`. |\n| spark.sql.dataSketches.distinctCnt.hll.lgK | 12 | Specifies the parameter `lgK` for the distinct count sketch implementation named `HLL`, `HllSketch`. |\n\n## TODO\n\n - Checks performance differences between the built-in funtion and DataSketches ones.\n\n## Bug reports\n\nIf you hit some bugs and have requests, please leave some comments on [Issues](https://github.com/maropu/datasketches-spark/issues)\nor Twitter ([@maropu](http://twitter.com/#!/maropu)).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaropu%2Fdatasketches-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaropu%2Fdatasketches-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaropu%2Fdatasketches-spark/lists"}