{"id":15494275,"url":"https://github.com/andyglow/spark-option-setter","last_synced_at":"2026-05-11T08:02:41.953Z","repository":{"id":50218134,"uuid":"239643542","full_name":"andyglow/spark-option-setter","owner":"andyglow","description":"Spark's Reader/Writer Frame and Stream type-safe option setter","archived":false,"fork":false,"pushed_at":"2021-06-01T18:13:13.000Z","size":82,"stargazers_count":0,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-12T23:28:57.881Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andyglow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-11T00:35:48.000Z","updated_at":"2021-06-01T18:13:16.000Z","dependencies_parsed_at":"2022-09-24T08:41:10.716Z","dependency_job_id":null,"html_url":"https://github.com/andyglow/spark-option-setter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyglow%2Fspark-option-setter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyglow%2Fspark-option-setter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyglow%2Fspark-option-setter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyglow%2Fspark-option-setter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andyglow","download_url":"https://codeload.github.com/andyglow/spark-option-setter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241494328,"owners_count":19971906,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T08:13:00.578Z","updated_at":"2026-05-11T08:02:36.894Z","avatar_url":"https://github.com/andyglow.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# spark-option-setter\n\n[![Build Status](https://cloud.drone.io/api/badges/andyglow/spark-option-setter/status.svg)](https://cloud.drone.io/andyglow/spark-option-setter)\n[![codecov](https://codecov.io/gh/andyglow/spark-option-setter/branch/master/graph/badge.svg?token=Q37E5LWMIK)](https://codecov.io/gh/andyglow/spark-option-setter)\n[![mvn](https://img.shields.io/badge/dynamic/json.svg?label=mvn\u0026query=%24.response.docs%5B0%5D.latestVersion\u0026url=https%3A%2F%2Fsearch.maven.org%2Fsolrsearch%2Fselect%3Fq%3Dspark-option-setter_2.12%26start%3D0%26rows%3D1)](https://search.maven.org/artifact/com.github.andyglow/spark-option-setter_2.12/)\n\n\u003e since `0.3.1` it is compiled against scala `2.12` for spark `3.x` only\n\n\u003e earlier versions compiled against both scala `2.11` and `2.12` and aimed to cover spark `2.4.x`\n\nWhen it comes to specifying options on spark readers and writers sometimes it may look inconvenient\nwhen you need to work with effects like `Option`, `Either`, `Try`.. Although last 2 might mot be considered as a \ngood candidate, the first, which is `Option` often is taken into the loop.\n\nAnd code becomes messy...\n\n```scala\nval extra1Option: Option[String] = ???\nval extra2Option: Either[String, Long] = ???\nval spark: SparkSession = ???\n\nval reader = spark.read.format(\"parquet\").option(\"foo\", \"bar\")\nextra1Option foreach { reader.option(\"extra-option-1\", _) }\nextra2Option.right foreach { reader.option(\"extra-option-2\", _) }\nval df = reader.load()\n```  \n\nSo with this simple library you can acheave this level of readiness of code\n```scala\nimport com.github.andyglow.spark.options._\n\nval extra1Option: Option[String] = ???\nval extra2Option: Either[String, Long] = ???\nval spark: SparkSession = ???\n\nspark.read\n  .format(\"parquet\")\n  .option(\"foo\", \"bar\")\n  .option(\"extra-option-1\", extra1Option)\n  .option(\"extra-option-2\", extra2Option)\n  .load()\n```    \n\n### Install\n```sbt\nlibraryDependencies += \"com.github.andyglow\" %% \"spark-option-setter\" % \"0.0.1\"\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandyglow%2Fspark-option-setter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandyglow%2Fspark-option-setter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandyglow%2Fspark-option-setter/lists"}