{"id":22384685,"url":"https://github.com/mliarakos/spark-typed-ops","last_synced_at":"2025-10-15T12:31:16.065Z","repository":{"id":55218296,"uuid":"318003662","full_name":"mliarakos/spark-typed-ops","owner":"mliarakos","description":"Lightweight type-safe operations for Spark","archived":false,"fork":false,"pushed_at":"2021-01-06T03:47:44.000Z","size":66,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-08T05:11:07.802Z","etag":null,"topics":["scala","scala-macros","shapeless","spark","spark-scala","spark-sql"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mliarakos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-02T22:04:46.000Z","updated_at":"2024-03-01T23:05:10.000Z","dependencies_parsed_at":"2022-08-14T16:40:52.201Z","dependency_job_id":null,"html_url":"https://github.com/mliarakos/spark-typed-ops","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mliarakos/spark-typed-ops","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mliarakos%2Fspark-typed-ops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mliarakos%2Fspark-typed-ops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mliarakos%2Fspark-typed-ops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mliarakos%2Fspark-typed-ops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mliarakos","download_url":"https://codeload.github.com/mliarakos/spark-typed-ops/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mliarakos%2Fspark-typed-ops/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279078182,"owners_count":26098464,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-15T02:00:07.814Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["scala","scala-macros","shapeless","spark","spark-scala","spark-sql"],"created_at":"2024-12-05T01:18:51.177Z","updated_at":"2025-10-15T12:31:15.729Z","avatar_url":"https://github.com/mliarakos.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark Typed Ops\n\nSpark Typed Ops is a Scala library that provides lightweight type-safe operations for [Spark](https://spark.apache.org/). Perform simple typed [Datasets](https://spark.apache.org/docs/latest/sql-programming-guide.html) operations that compile to efficient DataFrame operations:\n\n```scala\nimport com.github.mliarakos.spark.sql.typed.ops._\n\ncase class User(id: Int, name: String, email: String)\n\n// create column\ncolFrom[User](_.id) \n// compiles to: col(\"id\")\n\nval ds: Dataset[User] = ???\n\n// get a column\nds.colFrom(_.id) \n// compiles to: ds.col(\"id\")\n\n// select columns\nds.selectFrom(_.id, _.name) \n// compiles to: ds.select(ds.col(\"id\"), ds.col(\"name\"))\n\n// project between case classes\ncase class Info(name: String, email: String)\n\nds.project[Info] \n// compiles to: ds.select(ds.col(\"name\"), ds.col(\"email\")).as[Info]\n\n// ... and more!\n```\n\nDataset columns are accessed in a type-safe manner, so errors (e.g. misspelled or non-existent columns) are caught by the compiler. The operations are then converted to equivalent untyped DataFrame operations for improved runtime performance. In addition, the simple approach to specifying columns is easily supported by IDEs for autocompletion and refactoring. Find out more about [getting started](#getting-started), the [motivation](#motivation) behind the library, and its complete [usage](#usage).\n\nThis project was inspired by [Frameless](https://github.com/typelevel/frameless) and [scala-nameof](https://github.com/dwickern/scala-nameof). Its goal is to remain lightweight and not to introduce a new API on top of Spark. As such, it provides only simple operations with type-safe column access. For a more complete type-safe extension to Spark, consider [Frameless](https://github.com/typelevel/frameless).\n\nSpark Typed Ops is currently in pre-release and is looking for feedback. Please comment on this [issue](https://github.com/mliarakos/spark-typed-ops/issues/2) with your thoughts.\n\n## Getting Started\n\nCurrently, Spark Typed Ops is available as a development snapshot. Configure your project to use snapshot releases:\n\n```scala\nresolvers += Resolver.sonatypeRepo(\"snapshots\")\n```\n\nThen, add `spark-typed-ops` as dependency to your project:\n\n```scala\n\"com.github.mliarakos\" %% \"spark-typed-ops\" % \"0.1.0-SNAPSHOT\"\n```\n\nSpark Types Ops intentionally does not have a compile dependency on Spark. This essentially allows you to use any version of Spark Typed Ops with any version of Spark. The following versions are tested with Spark Typed Ops, but others will most likely work:\n\n| Spark Types Ops | Scala | Spark |\n| --- | --- | --- |\n| 0.1.0-SNAPSHOT | 2.11 \u003cbr/\u003e 2.12 | 2.4.7 \u003cbr/\u003e 3.0.1 |\n\n## Motivation\n\nSpark Datasets add type safety to DataFrames, but with a slight trade-off for performance due to the overhead of object serialization and deserialization. There are many common simple use cases where we'd like to avoid the object overhead while maintaining type-safety.\n\nConsider the example of selecting columns from a Dataset as a DataFrame:\n\n```scala\ncase class User(id: Int, name: String, email: String)\nval ds: Dataset[User] = ???\n\n// maintain type-safety, incur object overhead\nval df1 = ds.map(user =\u003e (user.id, user.name))\n\n// lose type-safety, avoid object overhead\nval df2 = ds.select(\"id\", \"name\")\n```\n\nThe first approach using `map` maintains type-safety, but incurs object overhead. The columns are derived from type-safe access of the user object. However, as shown in the explain plan, the user object must be deserialized to be used by the function and the resulting tuple must be serialized:\n\n```\n== Physical Plan ==\n*(1) SerializeFromObject [assertnotnull(input[0, scala.Tuple2, true])._1 AS _1#10, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, assertnotnull(input[0, scala.Tuple2, true])._2, true, false) AS _2#11]\n+- *(1) MapElements \u003cfunction1\u003e, obj#9: scala.Tuple2\n   +- *(1) DeserializeToObject newInstance(class User), obj#8: User\n      +- LocalTableScan \u003cempty\u003e, [id#3, name#4, email#5]\n```\n\nThe second approach using `select` uses unsafe string column names, but avoids object overhead. The column names can only be validated at runtime, not compile time. However, as shown in the explain plan, the columns are directly accessed without having to serialize or deserialize any objects:\n\n```\n== Physical Plan ==\nLocalTableScan \u003cempty\u003e, [id#3, email#5]\n```\n\nIn addition, there are some Dataset methods that don't have type-safe versions at all:\n\n```scala\n// these methods have two versions that present the trade-off\nds.map(user =\u003e (user.id, user.name))\nds.select(\"id\", \"name\")\n\nds.groupByKey(user =\u003e (user.id, user.name))\nds.groupBy(\"id\", \"name\")\n\n// these methods have no type-safe version at all\nds.describe(\"id\", \"name\")\nds.dropDuplicates(\"id\", \"name\")\nds.sort(\"id\", \"name\")\n```\n\nSpark Typed Ops provides Dataset extensions to get both benefits. It converts type-safe Dataset operations to efficient DataFrame operations at compile time. The added type-safety helps prevents errors without sacrificing performance. It also operates mostly at compile time using only the existing Spark API so there's little to no runtime impact.\n\nMost of the operation conversions are implemented using Scala macros, with only the `project` method being implemented with [shapeless](https://github.com/milessabin/shapeless). Macros are the preferred implementation because they have broader IDE support for autocompletion and refactoring.\n\n## Usage\n\nUse Spark Type Ops by importing:\n\n```scala\nimport com.github.mliarakos.spark.sql.typed.ops._\n```\n\nThis provides type-safe versions of several Spark SQL functions for working with columns:\n\n```scala\ncase class User(id: Int, name: String, email: String)\n\n// names\nnameFrom[User](_.id)          // \"id\"\nnamesFrom[User](_.id, _.name) // Seq(\"id\", \"name\")\n\n// columns\n$[User](_.id)                // col(\"id\")\ncolFrom[User](_.id)          // col(\"id\")\ncolsFrom[User](_.id, _.name) // Seq(col(\"id\"), col(\"name\"))\n\n// typed columns\ntypedColFrom[User](_.id)   // col(\"id\").as[Int]\ntypedColFrom[User](_.name) // col(\"name\").as[String]\n```\n\nIt also provides type-safe Dataset extensions for many common methods:\n\n```scala\nval ds: Dataset[User] = ???\n\n// names\nds.nameFrom(_.id)          // \"id\"\nds.namesFrom(_.id, _.name) // Seq(\"id\", \"name\")\n\n// columns\nds.colFrom(_.id)          // ds.col(\"id\")\nds.colsFrom(_.id, _.name) // Seq(ds.col(\"id\"), ds.col(\"name\"))\n\n// select\nds.selectFrom(_.id, _.name)      // ds.select(ds.col(\"id\"), ds.col(\"name\"))\nds.selectFromTyped(_.id, _.name) // ds.select(ds.col(\"id\").as[Int], ds.col(\"name\").as[String])\n\n// other methods\nds.cubeFrom(_.id, _.name)                  // ds.cube(\"id\", \"name\")\nds.describeFrom(_.id, _.name)              // ds.describe(\"id\", \"name\")\nds.dropFrom(_.id, _.name)                  // ds.drop(\"id\", \"name\")\nds.dropDuplicatesFrom(_.id, _.name)        // ds.dropDuplicates(\"id\", \"name\")\nds.groupByFrom(_.id, _.name)               // ds.groupBy(\"id\", \"name\")\nds.orderByFrom(_.id, _.name)               // ds.orderBy(\"id\", \"name\")\nds.rollupFrom(_.id, _.name)                // ds.rollup(\"id\", \"name\")\nds.sortFrom(_.id, _.name)                  // ds.sort(\"id\", \"name\")\nds.sortWithinPartitionsFrom(_.id, _.name)  // ds.sortWithinPartitions(\"id\", \"name\")\nds.withColumnRenamedFrom(_.id, \"recordId\") // ds.withColumnRenamed(\"id\", \"recordId\")\n```\n\nNested columns are supported:\n\n```scala\ncase class Address(street: String, city: String)\ncase class User(id: Int, name: String, address: Address)\nval ds: Dataset[User] = ???\n\ncolFrom[User](_.address.street) // col(\"address.street\")\nds.colFrom(_.address.street)    // ds.col(\"address.street\")\nds.selectFrom(_.address.street) // ds.select(ds.col(\"address.street\"))\n```\n\nThese functions and methods are useful in specifying column expressions and conditions:\n\n```scala\n// User with posts\ncase class User(id: Int, name: String, email: String)\ncase class Post(id: Int, userId: Int, post: String)\n\nval users: Dataset[User] = ???\nval posts: Dataset[Post] = ???\n\n// where conditions\nposts.where(length(colFrom[Post](_.post)) \u003e 10)\n// posts.where(length(col(\"post\")) \u003e 10)\nusers.where(users.colFrom(_.name).like(\"a%\"))\n// users.where(users.col(\"name\").like(\"a%\"))\n\n// sort condition\nposts.sort(length(colFrom[Post](_.post)).desc)\n// posts.sort(length(col(\"post\")).desc)\n\n// join condition\nusers.join(posts, users.colFrom(_.id) === posts.colFrom(_.userId))\n// users.join(posts, users.col(\"id\") === posts.col(\"userId\"))\n\n// select column expression\nusers.select(upper(users.colFrom(_.name)))\n// users.select(upper(users.col(\"name\")))\n\n// new column expression\nposts.withColumn(\"preview\", substring(posts.colFrom(_.post), 0, 10))\n// posts.withColumn(\"preview\", substring(posts.col(\"post\"), 0, 10))\n```\n\nThe Dataset extensions also provide one new method, `project`, that performs a type-safe projection from one case class to another:\n\n```scala\ncase class Info(name: String, email: String)\n\nval info: Dataset[Info] = users.project[Info]\n// ds.select(ds.col(\"name\"), ds.col(\"email\")).as[Info]\n```\n\nThe projection is by column name (regardless of order) and the compiler validates that the needed columns exist and have the correct types.\n\nMany of the Dataset extension methods can be re-written using the name/column methods:\n\n```scala\n// these compile to the same expression\n// ds.orderBy(\"id\", \"name\")\nusers.orderByFrom(_.id, _.name)\nusers.orderBy(users.nameFrom(_.id), users.nameFrom(_.name))\nusers.orderBy(nameFrom[User](_.id), nameFrom[User](_.name))\n\n// these compile to an equivalent expression\nusers.orderBy(users.colFrom(_.id), users.colFrom(_.name))\nusers.orderBy(users.colsFrom(_.id, _.name): _*)\nusers.orderBy(colFrom[User](_.id), colFrom[User](_.name))\nusers.orderBy(colsFrom[User](_.id, _.name): _*)\n```\n\nHowever, using the extension methods is recommended when possible because they reduce boilerplate and help ensure columns are used in the correct context:\n\n```scala\n// will succeed at runtime\n// the columns are derived from the type of the dataset, so they must be valid on the dataset\nusers.orderByFrom(_.id, _.name)\n\n// will fail at runtime\n// although userId is a valid Post column, it is not a valid column on the users dataset\nusers.orderBy(nameFrom[Post](_.userId))\nusers.orderBy(colFrom[Post](_.userId))\n\n// will fail at runtime\n// although id is a valid posts column and is also a present on the users dataset, the column\n// references the posts dataset and is not actually available on the users dataset\nusers.orderBy(posts.colFrom(_.id))\n\n// will succeed at runtime, but is fragile to refactoring\n// id is a valid Post column and is also present on the users dataset, but it succeeds only \n// because the column is specified by name and the names coincidentally match\n// refactoring the name of the column on User or Post will cause failure at runtime\nusers.orderBy(nameFrom[Post](_.id))\n```\n\nIn general, the type-safety added by Spark Typed Ops is for column access, not for column usage. Be careful that columns are used in an appropriate context.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmliarakos%2Fspark-typed-ops","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmliarakos%2Fspark-typed-ops","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmliarakos%2Fspark-typed-ops/lists"}