{"id":20630742,"url":"https://github.com/sysgears/scala-spark-test-utils","last_synced_at":"2025-10-26T22:11:27.564Z","repository":{"id":66053100,"uuid":"306623150","full_name":"sysgears/scala-spark-test-utils","owner":"sysgears","description":null,"archived":false,"fork":false,"pushed_at":"2020-10-26T21:59:44.000Z","size":12,"stargazers_count":6,"open_issues_count":1,"forks_count":1,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-04T17:48:59.441Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sysgears.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-10-23T12:01:06.000Z","updated_at":"2024-11-21T19:01:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"bfc12f95-fa8b-4a7d-8613-2751200a9349","html_url":"https://github.com/sysgears/scala-spark-test-utils","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysgears%2Fscala-spark-test-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysgears%2Fscala-spark-test-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysgears%2Fscala-spark-test-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysgears%2Fscala-spark-test-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sysgears","download_url":"https://codeload.github.com/sysgears/scala-spark-test-utils/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242588421,"owners_count":20154203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T14:09:26.079Z","updated_at":"2025-10-26T22:11:22.520Z","avatar_url":"https://github.com/sysgears.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Test utils for Apache Spark\n\nThis repositories provides some util classes that helps readability of your unit/integration tests for Spark projects.\n\n## DataFrame declaration syntax\n\nUsing `com.sysgears.DataFramesBuilder` you can define your DataFrames in tests with slightly more readable way:\n\n```scala\nimport com.sysgears.DataFrameBuilder._\nimport org.apache.spark.sql.SparkSession\n...\n\nimplicit val spark: SparkSession = ...\n\nval users =\n    ! \"first_name\" | \"last_name\" | \"age\" |\n    ! \"John\"       | \"Johnson\"   | 17    |\n    ! \"Henry\"      | \"Petrovich\" | 18    |\n    ! \"Harry\"      | \"Harrison\"  | 19    |\n```\n\nFirst row is a header - names of columns. Other rows contains data. Types of data is defined by first row.\nDue to unary ! that starts row it has restriction - first column type must not be a boolean. To achieve this, just \nchange order of columns when first column is a `Boolean`.\n\n## DataFrames util to store state.\n\nDataFrames class allow you to mock your DAO that provides data frames for next asserting on it. It has two params:\nformat and path to make it closer to the Spark API.\n\nTo create/get DataFrames object:\n```scala\nimport com.sysgears.DataFrames\n...\n\nnew DataFrames().addReadableTable(\"jdbc\", \"users\", users) // or\nDataFrames.threadLocal.addReadableTable(\"jdbc\", \"users\", users)\n```\n\nNow you can implement your test DAO as next:\n\n```scala\nimport org.apache.spark.sql.DataFrame\nimport org.apache.spark.sql.SparkSession\nimport com.sysgears.DataFrames\n\ntrait UsersDao {\n  def getAll(): DataFrame\n  def save(dataFrame: DataFrame)\n}\n\nclass TestUsersDao(dataFrames: DataFrames) extends UsersDao {\n  override def getAll: DataFrame = dataFrames.read(\"jdbc\", \"users\")\n  override def save(dataFrame:  DataFrame) = dataFrames.write(\"jdbc\",\"users\", dataFrame)\n}\n```\n\nAfter that to get all written `DataFrame`:\n\n```scala\nimport com.sysgears.DataFrames\n...\n\ndataFrames.getWrittenTable(\"jdbc\", \"users\").show() // or\nDataFrames.threadLocal.getWrittenTable(\"jdbc\", \"users\").show()\n```\n\n## Spark stub object\n\n__This feature is experimental__\n\nYou can use `SparkStub` class to use it in tests and without any DAO.\n\n```scala\nval dataFrames = new DataFrames()\nval spark: SparkSession = SparkStub.create(dataFrames)\n```\n\nAs first argument it takes dataFrames object (which is DataFrames.threadLocal by default). Every `load` operation is\nreplaced by `dataFrames.read(\"...\", \"...\")` and every `save` operation on DataSet is replaced by \n`dataFrames.write(\"...\",\"...\", ...)`.\nSo you can use `dataFrames.addReadableTable(\"...\", \"...\", ...)` to add tables to be read by spark, and\nget all written data by using `dataFrames.getWrittenTable(\"...\", \"...\")`\n\n## Cucumber\n\n### Converters\n\nTo convert Cucumber's DataTable to/from Spark's DataFrame you can use `DataSetConverter` with converter methods.\n\n```scala\nimport io.cucumber.java.en.Given\nimport io.cucumber.datatable.DataTable\n\nclass Steps {\n    private implicit val sparkSession: SparkSession = ...\n\n    @Given(\"spark format: {string} for table: {string} has data:\")\n    def setDataFrame(format: String, tableOrPath: String, dataTable: DataTable): Unit = {\n      dataTable.asDataFrame()\n    }\n}\n```\n\nAfter that you will be able to use it like this:\n```gherkin\nGiven spark format: \"jdbc\" for table: \"users\" has data:\n  | first_name STRING | last_name STRING | age INT |\n  | John              | Petrovich        | 17      |\n  | Henry             | Johnson          | 18      |\n  | Harry             | Potter           | 19      |\n```\n\nYou can enumerate default fields as Spark SQL:\n\n```scala\ndataTable.asDataFrame(defaults = \"age INT\")\n```\n\nYou can omit types declaration, if you have some java/scala type that represents this DataFrame:\n\n```scala\ndataTable.asTypeCheckedDataFrame(classOf[User])\ndataTable.asTypeCheckedDataFrame(classOf[User], defaults = \"age\")\n```\n\n### Predefined steps\n\nLibrary already has a bunch of steps might be used to write BDD tests. Here is a list of all available given steps:\n\n```gherkin\nGiven spark format: \"jdbc\" for table: \"users\" has data:\n  | first_name STRING | last_name STRING | age INT |\n  | John              | Petrovich        | 17      |\n\nGiven spark format: \"jdbc\" for table: \"users\" has data with defaults \"age INT\":\n  | first_name STRING | last_name STRING |\n  | John              | Petrovich        |\n\nGiven spark format: \"jdbc\" for table: \"users\" has data as \"com.example.User\":\n  | first_name | last_name | age |\n  | John       | Petrovich | 17  |\n\nGiven spark format: \"jdbc\" for table: \"users\" has data with defaults \"age\" as \"com.example.User\":\n  | first_name | last_name |\n  | John       | Petrovich |\n\nGiven spark format: \"jdbc\" for table: \"users\" has data with defaults as \"com.example.User\":\n  | first_name | last_name |\n  | John       | Petrovich |\n```\n\nand all `then` steps:\n\n```gherkin\nThen spark format: \"jdbc\" for table: \"users\" has data:\n  | first_name STRING | last_name STRING | age INT |\n  | John              | Petrovich        | 17      |\n\nThen spark format: \"jdbc\" for table: \"users\" wrote data with defaults \"age INT\":\n  | first_name STRING | last_name STRING |\n  | John              | Petrovich        |\n\nThen spark format: \"jdbc\" for table: \"users\" wrote data as \"com.example.User\":\n  | first_name | last_name | age |\n  | John       | Petrovich | 17  |\n\nThen spark format: \"jdbc\" for table: \"users\" wrote data with defaults \"age\" as \"com.example.User\":\n  | first_name | last_name |\n  | John       | Petrovich |\n\nThen spark format: \"jdbc\" for table: \"users\" wrote data with defaults as \"com.example.User\":\n  | first_name | last_name |\n  | John       | Petrovich |\n```\n\n`SparkSteps` is integrated with `DataFrames`, so in your test job runner code you should pass it to your job method or \nbind it to your DAO mock or use `SparkStub`.\n`SparkSteps` is integrated by `@Inject` from Guice, and takes 3 params:\n\n* `SparkSession` - required to enable conversion between Cucumber's `DataTAale` and Spark's `DataFrame`\n* `DataFrames` - required to enable it to add available to read tables and assert on written tables\n* `@Named(\"cucumber.spark.datatype-packages\") dataTypesPackages: Array[String]` - \nallows to use short class names in steps. For example, when `dataTypesPackages = Array(\"com.example\", \"com\")` you can \nwrite short class names:\n\n```gherkin\nThen spark format: \"jdbc\" for table: \"users\" wrote data as \"User\":\nThen spark format: \"jdbc\" for table: \"users\" wrote data as \"example.User\":\n```\n\nTo provide this args, lets create module for Cucumber:\n\n```scala\nimport com.google.inject._\nimport com.google.inject.name.Named\nimport com.sysgears.{DataFrames, SparkStub}\nimport org.apache.spark.sql.SparkSession\nimport io.cucumber.core.backend.ObjectFactory\n\nclass TestModule extends AbstractModule {\n  override def configure(): Unit = {}\n\n  @Provides\n  @Singleton\n  def dataFrames() = new DataFrames() // or DataFrames.threadLocal\n\n  @Provides\n  @Singleton\n  def session(dataFrames: DataFrames): SparkSession = SparkStub.create(dataFrames) // or any other session\n\n  @Provides\n  @Named(\"cucumber.spark.datatype-packages\")\n  def dataTypePackages(): Array[String] = Array(\"com.example.demo\")\n}\n```\n\nConfigure Object factory:\n\n```scala\nimport com.google.inject._\nimport io.cucumber.core.backend.ObjectFactory\nimport io.cucumber.guice.{CucumberModules, ScenarioScope}\n\nclass TestObjectFactory extends ObjectFactory {\n  private val injector = Guice.createInjector(\n    Stage.PRODUCTION,\n    CucumberModules.createScenarioModule,\n    new TestModule()\n  )\n\n  override def start(): Unit = injector.getInstance(classOf[ScenarioScope]).enterScope()\n  override def stop(): Unit = injector.getInstance(classOf[ScenarioScope]).exitScope()\n  override def getInstance[T](glueClass: Class[T]): T = injector.getInstance(glueClass)\n  override def addClass(glueClass: Class[_]): Boolean = true\n}\n```\n`/src/resources/META-INF/services/io.cucumber.core.backend.ObjectFactory`:\n```\ncom.example.demo.TestObjectFactory\n```\n\nAnd eventually pass all args to Cucumber (here we are using junit integration):\n\n```scala\nimport io.cucumber.junit.{Cucumber, CucumberOptions}\nimport org.junit.runner.RunWith\n\n@RunWith(classOf[Cucumber])\n@CucumberOptions(\n  objectFactory = classOf[TestObjectFactory],       // object factory we created above\n  glue = Array(\"com.sysgears\", \"com.example.demo\"), // com.sysgears contains step definitions, com.example.demo - just for example\n  features = Array(\"classpath:\")                    // to load all features defined in the root of resources folder\n)\nclass CucumberDemo {}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysgears%2Fscala-spark-test-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsysgears%2Fscala-spark-test-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysgears%2Fscala-spark-test-utils/lists"}