{"id":19168409,"url":"https://github.com/bloomberg/spark-flow","last_synced_at":"2025-05-07T14:41:59.279Z","repository":{"id":66125203,"uuid":"71829679","full_name":"bloomberg/spark-flow","owner":"bloomberg","description":"Library for organizing batch processing pipelines in Apache Spark","archived":false,"fork":false,"pushed_at":"2017-01-04T23:46:00.000Z","size":290,"stargazers_count":41,"open_issues_count":1,"forks_count":13,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-19T23:31:58.929Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bloomberg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-24T20:32:24.000Z","updated_at":"2024-04-07T19:09:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"ec1f323a-d2a6-4bc8-92f2-c6cddd4eacce","html_url":"https://github.com/bloomberg/spark-flow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fspark-flow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fspark-flow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fspark-flow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fspark-flow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bloomberg","download_url":"https://codeload.github.com/bloomberg/spark-flow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252897456,"owners_count":21821439,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T09:42:32.948Z","updated_at":"2025-05-07T14:41:59.255Z","avatar_url":"https://github.com/bloomberg.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# spark-flow\n\nThis is a library for organizing batch processing pipelines in Apache Spark and handling automatic checkpointing of intermediate results. The core type is a DC (Distributed Collection) which is analogous to a Spark Dataset. DCs have an API encompassing RDDs, Datasets, and Dataframes.\n\nA logical pipeline can be constructed entirely lazily without a Spark context. Pass in a Spark context to any DC to get the corresponding Dataset, RDD or Dataframe.\n\n## Building Locally\n    sbt publishLocal\n\n## Include in project\n    libraryDependencies += \"com.bloomberg\" %% \"spark-flow\" % \"0.1.0\"\n\n## Basic Example\n    \n    import com.bloomberg.sparkflow\n    \n    val numbers = sparkflow.parallelize(1 to 10)\n    val filtered = numbers.filter(_ \u003c 3).checkpoint()\n    val doubled = filtered.map(_ * 2)\n    \n    println(doubled.getRDD(sc).sum())\n    \n## Combined Dataframe API\n    import com.bloomberg.sparkflow\n    \n    val cars = sparkflow.read\n      .format(\"csv\")\n      .option(\"header\", \"true\")\n      .load(testFile(\"cars.csv\"))\n\n    val makeModel = cars.select(\"make\", \"model\").checkpoint()\n    \n    makeModel.getDF(sc).show()\n    \n## Larger Example\n\n    object FilmsPipeline {\n  \n      class FilmMain(){\n  \n        val inputFilmRows = sparkflow.read.format(\"csv\").option(\"header\", \"true\")\n          .load(testFile(\"Film_Locations_in_San_Francisco.csv\"))\n          .toDF(FilmsPipeline.columns:_*)\n          .as[FilmsPipeline.InputFilmRow]\n  \n        val filmRows = inputFilmRows.keyBy(_.title)\n          .groupByKey()\n          .map(parseFilmRows)\n          .checkpoint()\n  \n        val actorMovieCount = filmRows.flatMap(filmRow =\u003e filmRow.actors.map((_,1))).reduceByKey(_+_)\n        val topActors = actorMovieCount.sortBy(_._2, ascending = false).map(_._1).take(5)\n  \n        val filmsWithTopActors = filmRows.withResult(topActors).filter{\n          filmRowActors =\u003e {\n            val (filmRow, actors) = filmRowActors\n            filmRow.actors.toSet.intersect(actors.toSet).nonEmpty\n          }\n        }.map(_._1)\n      }\n  \n  \n      val columns = Seq(\n        \"title\",\n        \"release\",\n        \"locations\",\n        \"funFacts\",\n        \"productionCompany\",\n        \"distributor\",\n        \"director\",\n        \"writer\",\n        \"actor1\",\n        \"actor2\",\n        \"actor3\"\n      )\n  \n      case class InputFilmRow(title: String,\n                              release: String,\n                              locations: String,\n                              funFacts: String,\n                              productionCompany: String,\n                              distributor: String,\n                              director: String,\n                              writer: String,\n                              actor1: String,\n                              actor2: String,\n                              actor3: String)\n  \n      case class FilmRow( title: String,\n                          release: String,\n                          locations: Seq[String],\n                          funFacts: String,\n                          productionCompany: String,\n                          distributor: String,\n                          director: String,\n                          writer: String,\n                          actors: Seq[String])\n  \n  \n      val parseFilmRows = (tuple: (String, Seq[InputFilmRow])) =\u003e {\n        val (title, rows) = tuple\n        val firstRow = rows.head\n        val locations = rows.map(_.locations).distinct\n        val actors = rows.flatMap(row =\u003e Seq(row.actor1, row.actor2, row.actor3)).distinct.filter(_ != \"\")\n        FilmRow(\n          firstRow.title,\n          firstRow.release,\n          locations,\n          firstRow.funFacts,\n          firstRow.productionCompany,\n          firstRow.distributor,\n          firstRow.director,\n          firstRow.writer,\n          actors)\n      }\n  \n      def testFile(fileName: String): String = {\n        Thread.currentThread().getContextClassLoader.getResource(fileName).toString\n      }\n  \n    }\n\n\n\n## Upcoming\n* graphx support\n* DAG viewer frontend attached to running process\n* component / pipeline abstractions\n* debug run mode with auto Try wrapped functions and trapped failures\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbloomberg%2Fspark-flow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbloomberg%2Fspark-flow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbloomberg%2Fspark-flow/lists"}