{"id":15044856,"url":"https://github.com/fqaiser94/mse","last_synced_at":"2025-08-22T09:42:48.531Z","repository":{"id":57443671,"uuid":"234994220","full_name":"fqaiser94/mse","owner":"fqaiser94","description":"Make Structs Easy (MSE)","archived":false,"fork":false,"pushed_at":"2020-06-22T20:18:18.000Z","size":174,"stargazers_count":19,"open_issues_count":1,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-31T02:36:19.168Z","etag":null,"topics":["nested","pyspark","python","scala","spark","struct"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fqaiser94.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-20T01:17:56.000Z","updated_at":"2024-08-20T14:21:28.000Z","dependencies_parsed_at":"2022-09-05T09:40:13.445Z","dependency_job_id":null,"html_url":"https://github.com/fqaiser94/mse","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fqaiser94%2Fmse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fqaiser94%2Fmse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fqaiser94%2Fmse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fqaiser94%2Fmse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fqaiser94","download_url":"https://codeload.github.com/fqaiser94/mse/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237990629,"owners_count":19398465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nested","pyspark","python","scala","spark","struct"],"created_at":"2024-09-24T20:51:08.615Z","updated_at":"2025-02-09T17:31:04.514Z","avatar_url":"https://github.com/fqaiser94.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Tests Passing](https://github.com/fqaiser94/mse/workflows/CI/badge.svg)\n![Build Status](https://github.com/fqaiser94/mse/workflows/CD/badge.svg)\n[![Maven Central](https://img.shields.io/maven-central/v/com.github.fqaiser94/mse_2.11)](https://search.maven.org/artifact/com.github.fqaiser94/mse_2.11)\n[![PyPI version shields.io](https://img.shields.io/pypi/v/mse.svg)](https://pypi.python.org/pypi/mse/)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/fqaiser94/mse/blob/master/LICENSE)\n\n# Make Structs Easy (MSE)\n\nThis library adds `withField`, `withFieldRenamed`, and `dropFields` methods to the Column class allowing users to easily add, rename, and drop fields inside of StructType columns. \nThe signature and behaviour of these methods is intended to be similar to their Dataset equivalents, namely the `withColumn`, `withColumnRenamed`, and `drop` methods.\n\nThe methods themselves are backed by efficient Catalyst Expressions and as a result, should provide better performance than equivalent UDFs. \nWhile this library uses Scala's implicit conversion technique to \"monkey patch\" the methods on to the Column class, \nthere is an on-going effort to add these methods natively to the Column class in the Apache Spark SQL project. \nYou can follow along with the progress of this initiative in [SPARK-22231](https://issues.apache.org/jira/browse/SPARK-22231).\n\nIf you find this project useful, please consider supporting it by giving a star!\n\n# Supported Spark versions\n\nMSE should work without any further requirements on Spark/PySpark 2.4.x. \nThe library is available for both Scala versions 2.11 and 2.12.\nThe library is available for Python 3.x.\n\n# Installation\n\n## Scala\n\nStable releases of MSE are published to Maven Central. \nAs such, you can pull in the current stable release by simply adding a library dependency to your project for the correct version.\nFor example, for an SBT project, simply add the following line to your `build.sbt`:\n\n```\nlibraryDependencies += \"com.github.fqaiser94\" %% \"mse\" % \"0.2.4\"\n```\n\nFor other types of projects (e.g. Maven, Gradle), see the installation instructions at this [link](https://search.maven.org/artifact/com.github.fqaiser94/mse_2.11). \n\n## Python\n\nStable releases of MSE are published to PyPi.\nYou will also need to provide your PySpark application/s with the path to the MSE jar which you can get from [here](https://search.maven.org/artifact/com.github.fqaiser94/mse_2.11). For example: \n\n```bash\npip install mse\ncurl https://repo1.maven.org/maven2/com/github/fqaiser94/mse_2.11/0.2.4/mse_2.11-0.2.4.jar --output mse.jar\npyspark --jars mse.jar\n```\n\nIf you get errors like `TypeError: 'JavaPackage' object is not callable`, this usually indicates that you haven't \nprovided PySpark with the correct path to the MSE jar.   \n\n# Usage \n\nTo bring in to scope the (implicit) Column methods in Scala, use:   \n\n```scala\nimport com.github.fqaiser94.mse.methods._\n```\n\nTo bring in to scope the (implicit) Column methods in Python, use:\n\n```python\nfrom mse import *\n```\n\nThe rest of the example code shown below is written in Scala although equivalent Python code would look very similar.  \n\nYou can now use these methods to manipulate fields in a top-level StructType column: \n\n```scala\nimport org.apache.spark.sql._\nimport org.apache.spark.sql.types._\n\n// Generate some example data\nval structLevel1 = spark.createDataFrame(sc.parallelize(\n  Row(Row(1, null, 3)) :: Nil),\n  StructType(Seq(\n    StructField(\"a\", StructType(Seq(\n      StructField(\"a\", IntegerType),\n      StructField(\"b\", IntegerType),\n      StructField(\"c\", IntegerType))))))).cache\n      \nstructLevel1.show\n// +-------+                                                                       \n// |      a|\n// +-------+\n// |[1,, 3]|\n// +-------+\n\nstructLevel1.printSchema\n// root\n//  |-- a: struct (nullable = true)\n//  |    |-- a: integer (nullable = true)\n//  |    |-- b: integer (nullable = true)\n//  |    |-- c: integer (nullable = true)\n\n// add new field to top level struct\nstructLevel1.withColumn(\"a\", 'a.withField(\"d\", lit(4))).show\n// +----------+\n// |         a|\n// +----------+\n// |[1,, 3, 4]|\n// +----------+\n\n// replace field in top level struct\nstructLevel1.withColumn(\"a\", 'a.withField(\"b\", lit(2))).show\n// +---------+\n// |        a|\n// +---------+\n// |[1, 2, 3]|\n// +---------+\n\n// rename field in top level struct\nstructLevel1.withColumn(\"a\", 'a.withFieldRenamed(\"b\", \"z\")).printSchema\n// root\n//  |-- a: struct (nullable = true)\n//  |    |-- a: integer (nullable = true)\n//  |    |-- z: integer (nullable = true)\n//  |    |-- c: integer (nullable = true)\n\n// drop field in top level struct\nstructLevel1.withColumn(\"a\", 'a.dropFields(\"b\")).show\n// +------+\n// |     a|\n// +------+\n// |[1, 3]|\n// +------+\n```\n\nYou can also use these methods to manipulate fields in nested StructType columns: \n\n```scala\n// Generate some example data  \nval structLevel2 = spark.createDataFrame(sc.parallelize(\n    Row(Row(Row(1, null, 3))) :: Nil),\n    StructType(Seq(\n      StructField(\"a\", StructType(Seq(\n        StructField(\"a\", StructType(Seq(\n          StructField(\"a\", IntegerType),\n          StructField(\"b\", IntegerType),\n          StructField(\"c\", IntegerType)))))))))).cache\n          \nstructLevel2.show\n// +---------+\n// |        a|\n// +---------+\n// |[[1,, 3]]|\n// +---------+\n\nstructLevel2.printSchema\n// |-- a: struct (nullable = true)\n// |    |-- a: struct (nullable = true)\n// |    |    |-- a: integer (nullable = true)\n// |    |    |-- b: integer (nullable = true)\n// |    |    |-- c: integer (nullable = true)\n\n// add new field to nested struct\nstructLevel2.withColumn(\"a\", 'a.withField(\n  \"a\", $\"a.a\".withField(\"d\", lit(4)))).show\n// +------------+\n// |           a|\n// +------------+\n// |[[1,, 3, 4]]|\n// +------------+\n\n// replace field in nested struct\nstructLevel2.withColumn(\"a\", $\"a\".withField(\n  \"a\", $\"a.a\".withField(\"b\", lit(2)))).show\n// +-----------+\n// |          a|\n// +-----------+\n// |[[1, 2, 3]]|\n// +-----------+\n    \n// rename field in nested struct\nstructLevel2.withColumn(\"a\", 'a.withField(\n  \"a\", $\"a.a\".withFieldRenamed(\"b\", \"z\"))).printSchema\n// |-- a: struct (nullable = true)\n// |    |-- a: struct (nullable = true)\n// |    |    |-- a: integer (nullable = true)\n// |    |    |-- z: integer (nullable = true)\n// |    |    |-- c: integer (nullable = true)\n    \n// drop field in nested struct\nstructLevel2.withColumn(\"a\", 'a.withField(\n  \"a\", $\"a.a\".dropFields(\"b\"))).show\n// +--------+\n// |       a|\n// +--------+\n// |[[1, 3]]|\n// +--------+\n```\n\nYou can also manipulate **deeply** nested StructType columns using the aforementioned patterns \nbut it can be a little annoying to write out the full chain. For this scenario, this library also provides a helper method, \nnamely `add_struct_field`. You can use this method to add, rename, and drop deeply nested fields as shown below: \n\n```scala\n// Generate some example data  \nval structLevel3 = spark.createDataFrame(sc.parallelize(\n    Row(Row(Row(Row(1, null, 3)))) :: Nil),\n    StructType(Seq(\n      StructField(\"a\", StructType(Seq(\n        StructField(\"a\", StructType(Seq(\n          StructField(\"a\", StructType(Seq(\n              StructField(\"a\", IntegerType),\n              StructField(\"b\", IntegerType),\n              StructField(\"c\", IntegerType))))))))))))).cache\n              \nstructLevel3.show\n//+-----------+\n//|          a|\n//+-----------+\n//|[[[1,, 3]]]|\n//+-----------+\n\nstructLevel3.printSchema\n//root\n// |-- a: struct (nullable = true)\n// |    |-- a: struct (nullable = true)\n// |    |    |-- a: struct (nullable = true)\n// |    |    |    |-- a: integer (nullable = true)\n// |    |    |    |-- b: integer (nullable = true)\n// |    |    |    |-- c: integer (nullable = true)\n\n// add new field to deeply nested struct\nstructLevel3.withColumn(\"a\", add_struct_field(\"a.a.a\", \"d\", lit(4))).show\n// +--------------+                                                                \n// |             a|\n// +--------------+\n// |[[[1,, 3, 4]]]|\n// +--------------+\n\n// replace field in deeply nested struct\nstructLevel3.withColumn(\"a\", add_struct_field(\"a.a.a\", \"b\", lit(2))).show\n// +-------------+\n// |            a|\n// +-------------+\n// |[[[1, 2, 3]]]|\n// +-------------+\n    \n// rename field in deeply nested struct\nstructLevel3.withColumn(\"a\", add_struct_field(\"a.a\", \"a\", $\"a.a.a\".withFieldRenamed(\"b\", \"z\"))).printSchema\n// root\n//  |-- a: struct (nullable = true)\n//  |    |-- a: struct (nullable = true)\n//  |    |    |-- a: struct (nullable = true)\n//  |    |    |    |-- a: integer (nullable = true)\n//  |    |    |    |-- z: integer (nullable = true)\n//  |    |    |    |-- c: integer (nullable = true)\n\n// drop field in deeply nested struct\nstructLevel3.withColumn(\"a\", add_struct_field(\"a.a\", \"a\", $\"a.a.a\".dropFields(\"b\"))).show\n// +----------+\n// |         a|\n// +----------+\n// |[[[1, 3]]]|\n// +----------+\n\n// add, rename, and drop fields in deeply nested struct\nval result = structLevel3.withColumn(\"a\", add_struct_field(\"a.a\", \"a\", $\"a.a.a\".dropFields(\"b\").withFieldRenamed(\"c\", \"b\").withField(\"c\", lit(4))))\nresult.show\n// +-------------+\n// |            a|\n// +-------------+\n// |[[[1, 3, 4]]]|\n// +-------------+\n\nresult.printSchema\n// root\n//  |-- a: struct (nullable = true)\n//  |    |-- a: struct (nullable = true)\n//  |    |    |-- a: struct (nullable = true)\n//  |    |    |    |-- a: integer (nullable = true)\n//  |    |    |    |-- b: integer (nullable = true)\n//  |    |    |    |-- c: integer (nullable = false)\n``` \n\nAnother common use-case is to perform these operations on arrays of structs. \nTo do this using the Scala APIs, we recommend combining the functions in this library with the functions provided in [spark-hofs](https://github.com/AbsaOSS/spark-hofs/):\n\n```scala\nimport org.apache.spark.sql._\nimport org.apache.spark.sql.types._\nimport za.co.absa.spark.hofs._\nimport com.github.fqaiser94.mse.methods._\n\n// Generate some example data\nval arrayOfStructs = spark.createDataFrame(sc.parallelize(\n    Row(List(Row(1, null, 3), Row(4, null, 6))) :: Nil),\n    StructType(Seq(\n      StructField(\"array\", ArrayType(\n        StructType(Seq(\n          StructField(\"a\", IntegerType),\n          StructField(\"b\", IntegerType), \n          StructField(\"c\", IntegerType)))))))).cache\n          \narrayOfStructs.show\n// +------------------+\n// |             array|\n// +------------------+\n// |[[1,, 3], [4,, 6]]|\n// +------------------+\n\narrayOfStructs.printSchema\n// root\n//  |-- array: array (nullable = true)\n//  |    |-- element: struct (containsNull = true)\n//  |    |    |-- a: integer (nullable = true)\n//  |    |    |-- b: integer (nullable = true)\n//  |    |    |-- c: integer (nullable = true)\n\n// add new field to each struct element of array \narrayOfStructs.withColumn(\"array\", transform($\"array\", elem =\u003e elem.withField(\"d\", lit(\"hello\")))).show(false)\n// +--------------------------------+\n// |array                           |\n// +--------------------------------+\n// |[[1,, 3, hello], [4,, 6, hello]]|\n// +--------------------------------+\n\n// replace field in each struct element of array\narrayOfStructs.withColumn(\"array\", transform($\"array\", elem =\u003e elem.withField(\"b\", elem.getField(\"a\") + 1))).show(false)\n// +----------------------+\n// |array                 |\n// +----------------------+\n// |[[1, 2, 3], [4, 5, 6]]|\n// +----------------------+\n\n// rename field in each struct element of array\narrayOfStructs.withColumn(\"array\", transform($\"array\", elem =\u003e elem.withFieldRenamed(\"b\", \"z\"))).printSchema\n// root\n//  |-- array: array (nullable = true)\n//  |    |-- element: struct (containsNull = true)\n//  |    |    |-- a: integer (nullable = true)\n//  |    |    |-- z: integer (nullable = true)\n//  |    |    |-- c: integer (nullable = true)\n\n// drop field in each Struct element of array\narrayOfStructs.withColumn(\"array\", transform($\"array\", elem =\u003e elem.dropFields(\"b\"))).show(false)\n// +----------------+\n// |array           |\n// +----------------+\n// |[[1, 3], [4, 6]]|\n// +----------------+\n```\n\n# SQL installation and usage\n\nThe underlying Catalyst Expressions are SQL compatible. \nUnfortunately, Spark only added public APIs for plugging in custom Catalyst Expressions into the FunctionRegistry in Spark 3.0.0 \n(which is at the time of writing is still in preview). You can find a project with an example of how to do this [here](https://github.com/fqaiser94/mse-sql-example). \n\n# Catalyst Optimization Rules\n\nWe also provide some Catalyst optimization rules that can be plugged into a Spark session to get even better performance. This is as simple as including the following two lines of code at the start of your Scala Spark program:  \n\n```scala\nimport org.apache.spark.sql.catalyst.optimizer.SimplifyStructExpressions\nspark.experimental.extraOptimizations = SimplifyStructExpressions.rules\n``` \n\nSpark will use these optimization rules to internally rewrite queries in a more optimal fashion. \nFor example, consider the following query and its corresponding physical plan: \n\n```scala\nval query = structLevel1.withColumn(\"a\", 'a.withField(\"d\", lit(4)).withField(\"e\", lit(5)))\n\nquery.explain\n// == Physical Plan ==\n// *(1) Project [add_fields(add_fields(a#1, d, 4), e, 5) AS a#32343]\n// +- InMemoryTableScan [a#1]\n//       +- InMemoryRelation [a#1], StorageLevel(disk, memory, deserialized, 1 replicas)\n//             +- Scan ExistingRDD[a#1]\n```\n\nIf we add the `SimplifyStructExpressions.rules` to our Spark session, we see a slightly different physical plan for the same query:\n\n```scala\nimport org.apache.spark.sql.catalyst.optimizer.SimplifyStructExpressions\nspark.experimental.extraOptimizations = SimplifyStructExpressions.rules\n\nquery.explain\n// == Physical Plan ==\n// *(1) Project [add_fields(a#1, d, e, 4, 5) AS a#32343]\n// +- InMemoryTableScan [a#1]\n//       +- InMemoryRelation [a#1], StorageLevel(disk, memory, deserialized, 1 replicas)\n//             +- Scan ExistingRDD[a#1]\n```\n\nAs you can see, the successive `add_fields` method calls have been collapsed into a single `add_fields` method call.  \n\nTheoretically, this should improve performance but for the most part, you won't notice much difference unless you're doing some particularly intense struct manipulation and/or working with a particularly large dataset.  \n\nUnfortunately, to the best of our knowledge, there is currently no way to plug in custom Catalyst optimization rules directly using the Python APIs. \n\n# Questions/Thoughts/Concerns?\n\nFeel free to submit an issue. \n\n# Instructions for deploying a new release\n\nIncrement version number in `python/setup.py`  \nCreate a new release with appropriately incremented tag\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffqaiser94%2Fmse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffqaiser94%2Fmse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffqaiser94%2Fmse/lists"}