{"id":15044858,"url":"https://github.com/absaoss/spark-hofs","last_synced_at":"2025-10-24T14:32:15.801Z","repository":{"id":40366744,"uuid":"167398899","full_name":"AbsaOSS/spark-hofs","owner":"AbsaOSS","description":"Scala API for Apache Spark SQL high-order functions  ","archived":false,"fork":false,"pushed_at":"2023-08-04T06:19:15.000Z","size":76,"stargazers_count":14,"open_issues_count":0,"forks_count":2,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-01-31T02:36:12.567Z","etag":null,"topics":["high-order-functions","scala","spark","sql"],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AbsaOSS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-24T16:25:10.000Z","updated_at":"2023-03-11T10:19:51.000Z","dependencies_parsed_at":"2024-09-25T01:55:03.953Z","dependency_job_id":"b8683538-c40a-4639-aba6-b98188f81286","html_url":"https://github.com/AbsaOSS/spark-hofs","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fspark-hofs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fspark-hofs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fspark-hofs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fspark-hofs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AbsaOSS","download_url":"https://codeload.github.com/AbsaOSS/spark-hofs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237990631,"owners_count":19398466,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["high-order-functions","scala","spark","sql"],"created_at":"2024-09-24T20:51:08.768Z","updated_at":"2025-10-24T14:32:10.738Z","avatar_url":"https://github.com/AbsaOSS.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# spark-hofs\nApache Spark 2.4.0 introduced high-order functions as a part of SQL expressions. These new functions are accessible\nonly via textual representation of Spark SQL.\n\nThis library makes the high-order functions accessible also for Dataframe/Dataset Scala API to get type safety when\nusing the functions. \n\n\u003e **Warning**\n\u003e Starting from Spark 3.2.1 the high-order functions are available in the Scala API natively. The library is still compiled\nfor Scala 2.12 and Scala 2.13 and compatible with Spark 3, but it is for backwards compatibility only and we recommend\nmigrating from `spark-hofs` to the native Spark API.\n\n## Usage\n\nReference the library\n\n\u003ctable\u003e\n\u003ctr\u003e\u003cth\u003eScala 2.11\u003c/th\u003e\u003cth\u003eScala 2.12\u003c/th\u003e\u003cth\u003eScala 2.13\u003c/th\u003e\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"center\"\u003e\n\u003ca href = \"https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-hofs_2.11\"\u003e\u003cimg src = \"https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-hofs_2.11/badge.svg\" alt=\"Maven Central\"\u003e\u003c/a\u003e\u003cbr\u003e\n\u003c/td\u003e\n\u003ctd align=\"center\"\u003e\n\u003ca href = \"https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-hofs_2.12\"\u003e\u003cimg src = \"https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-hofs_2.12/badge.svg\" alt=\"Maven Central\"\u003e\u003c/a\u003e\u003cbr\u003e\n\u003c/td\u003e\n\u003ctd align=\"center\"\u003e\n\u003ca href = \"https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-hofs_2.13\"\u003e\u003cimg src = \"https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-hofs_2.13/badge.svg\" alt=\"Maven Central\"\u003e\u003c/a\u003e\u003cbr\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\u003cpre\u003egroupId: za.co.absa\u003cbr\u003eartifactId: spark-hofs_2.11\u003cbr\u003eversion: 0.5.0\u003c/pre\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\u003cpre\u003egroupId: za.co.absa\u003cbr\u003eartifactId: spark-hofs_2.12\u003cbr\u003eversion: 0.5.0\u003c/pre\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\u003cpre\u003egroupId: za.co.absa\u003cbr\u003eartifactId: spark-hofs_2.13\u003cbr\u003eversion: 0.5.0\u003c/pre\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nPlease, use the table below to determine what version of spark-hofs to use for Spark compatibility.\n\n| spark-hofs version |  Scala version   | Spark version |\n|:------------------:|:----------------:|:-------------:|\n|       0.1.0        |       2.11       |     2.4.0     |\n|       0.2.0        |       2.11       |     2.4.1     |\n|       0.3.x        |       2.11       |     2.4.2     |\n|       0.4.x        |    2.11, 2.12    |     2.4.3+    |\n|       0.5.x        | 2.11, 2.12, 2.13 |     2.4.3+    |\n\nImport Scala API of the high-order functions into your scope.\n```scala\nimport za.co.absa.spark.hofs._\n```\n\n## Functions\n\n### Transform\nThe **transform** function is an equivalent to the *map* function from functional programming. It takes a column of\narrays as the first argument and projects every element in each array with using a function passed as the second argument.\n```scala\nscala\u003e df.withColumn(\"output\", transform('input, x =\u003e x + 1)).show\n+------------+------------+\n|       input|      output|\n+------------+------------+\n|[1, 4, 5, 7]|[2, 5, 6, 8]|\n+------------+------------+\n```\nIf the logic of the projection function requires information about the element position of a given array,\nthe **transform** function can pass an index starting from 0 to the projection function as the second argument.\n```scala\nscala\u003e df.withColumn(\"output\", transform('input, (x, i) =\u003e x + i)).show\n+------------+-------------+\n|       input|       output|\n+------------+-------------+\n|[1, 4, 5, 7]|[1, 5, 7, 10]|\n+------------+-------------+\n```\nBy default, the lambda variable representing the element will be seen as `elm` and the lambda variable representing\nthe index as `idx` in Spark execution plans.\n```scala\nscala\u003e df.withColumn(\"output\", transform('input, (x, i) =\u003e x + i)).explain(true)\n== Parsed Logical Plan ==\n'Project [input#8, transform('input, lambdafunction(('elm + 'idx), 'elm, 'idx, false)) AS output#45]\n+- Project [value#6 AS input#8]\n   +- LocalRelation [value#6]\n\n== Analyzed Logical Plan ==\ninput: array\u003cint\u003e, output: array\u003cint\u003e\nProject [input#8, transform(input#8, lambdafunction((lambda elm#51 + lambda idx#52), lambda elm#51, lambda idx#52, false)) AS output#45]\n+- Project [value#6 AS input#8]\n   +- LocalRelation [value#6]\n\n...\n```\nNames of the lambda variables can be changed by passing extra argument to the **transform** function.\n```scala\nscala\u003e df.withColumn(\"output\", transform('input, (x, i) =\u003e x + i, \"myelm\", \"myidx\")).explain(true)\n== Parsed Logical Plan ==\n'Project [input#8, transform('input, lambdafunction(('myelm + 'myidx), 'myelm, 'myidx, false)) AS output#53]\n+- Project [value#6 AS input#8]\n   +- LocalRelation [value#6]\n\n== Analyzed Logical Plan ==\ninput: array\u003cint\u003e, output: array\u003cint\u003e\nProject [input#8, transform(input#8, lambdafunction((lambda myelm#59 + lambda myidx#60), lambda myelm#59, lambda myidx#60, false)) AS output#53]\n+- Project [value#6 AS input#8]\n   +- LocalRelation [value#6]\n   \n...   \n```\n\n### Filter\nThe **filter** function takes a column of arrays as the first argument and eliminates all elements that do not satisfy \nthe predicate that is passed as the second argument.\n```scala\nscala\u003e df.withColumn(\"output\", filter('input, x =\u003e x % 2 === 1)).show\n+------------------+---------+\n|             input|   output|\n+------------------+---------+\n|[1, 2, 4, 5, 7, 8]|[1, 5, 7]|\n+------------------+---------+\n```\nThe lambda variable within the predicate will be seen as `elm` in Spark execution plans. This name can be changed by\npassing the third argument to the **filter** function. \n\n### Aggregate\nThe **aggregate** function is an equivalent of the *foldLeft* function from functional programming. The method takes\na column of arrays and a column of zero elements as first two arguments. The next argument is a binary function merging\na zero element and all elements from an input array into one element. The first argument of the merging function is\nan accumulated value and the second one is an element of given iteration.\n```scala\nscala\u003e df.withColumn(\"output\", aggregate('input, 'zero, (acc, x)  =\u003e acc + x)).show\n+------------------+----+------+\n|             input|zero|output|\n+------------------+----+------+\n|[1, 2, 4, 5, 7, 8]| 100|   127|\n+------------------+----+------+\n```\nIf an user wants to transform the reduced value before returning the result, the user can pass a function performing\nthe transformation logic as the fourth argument.\n```scala\nscala\u003e df.withColumn(\"output\", aggregate('input, 'zero, (acc, x)  =\u003e acc + x, y =\u003e concat(y, y))).show\n+------------------+----+------+\n|             input|zero|output|\n+------------------+----+------+\n|[1, 2, 4, 5, 7, 8]| 100|127127|\n+------------------+----+------+\n```\n\nThe lambda variable representing the accumulator will be seen as `acc` and the lambda variable representing the element \nas `elm` in Spark execution plans. The names can be changed by passing extra arguments to the **aggregate** function.\n\n### Zip With\nThe **zip_with** function takes two columns of arrays as the first two arguments and performs element-wise merge into\na single column of arrays. The third argument ia a function taking one element from each array at the same\nposition and specifying the merge logic. If one array is shorter, null elements are appended this array to be the same \nlength as the longer array.\n```scala\nscala\u003e df.withColumn(\"output\", zip_with('input1, 'input2, (x, y) =\u003e x + y)).show\n+---------------+-------------+---------------+\n|         input1|       input2|         output|\n+---------------+-------------+---------------+\n|[1, 2, 4, 5, 7]|[2, 4, 8, 12]|[3, 6, 12, 17,]|\n+---------------+-------------+---------------+\n```\nThe lambda variables indicating input elements to the merging function will be seen as `left` and `right` in\nSpark execution plans. The names can be changed by passing extra arguments to the **zip_with** function.\n\n## How to generate Code coverage report\n```sbt\nsbt ++{matrix.scala} jacoco -DSPARK_VERSION={matrix.spark}\n```\nCode coverage will be generated on path:\n```\n{project-root}/spark-hofs/target/scala-{scala_version}/jacoco/report/html\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabsaoss%2Fspark-hofs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabsaoss%2Fspark-hofs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabsaoss%2Fspark-hofs/lists"}