{"id":13640737,"url":"https://github.com/leobenkel/Zparkio","last_synced_at":"2025-04-20T06:34:28.826Z","repository":{"id":40303998,"uuid":"221801743","full_name":"leobenkel/ZparkIO","owner":"leobenkel","description":"Boiler plate framework to use Spark and ZIO together.","archived":false,"fork":false,"pushed_at":"2024-08-12T23:01:45.000Z","size":779,"stargazers_count":175,"open_issues_count":23,"forks_count":31,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-07T09:21:34.257Z","etag":null,"topics":["boiler-plate","functional-programming","helpers","scala","spark","template","zio"],"latest_commit_sha":null,"homepage":"https://github.com/leobenkel/ZparkIO","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leobenkel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"leobenkel","patreon":"leobenkel","open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2019-11-14T23:00:38.000Z","updated_at":"2025-03-17T02:52:04.000Z","dependencies_parsed_at":"2024-09-24T05:40:59.211Z","dependency_job_id":null,"html_url":"https://github.com/leobenkel/ZparkIO","commit_stats":{"total_commits":283,"total_committers":13,"mean_commits":21.76923076923077,"dds":0.7703180212014135,"last_synced_commit":"85cf42197843fd1cdb0a1d531fb43af316d54f5b"},"previous_names":[],"tags_count":54,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leobenkel%2FZparkIO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leobenkel%2FZparkIO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leobenkel%2FZparkIO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leobenkel%2FZparkIO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leobenkel","download_url":"https://codeload.github.com/leobenkel/ZparkIO/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249859688,"owners_count":21335988,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["boiler-plate","functional-programming","helpers","scala","spark","template","zio"],"created_at":"2024-08-02T01:01:14.000Z","updated_at":"2025-04-20T06:34:28.529Z","avatar_url":"https://github.com/leobenkel.png","language":"Scala","funding_links":["https://github.com/sponsors/leobenkel","https://patreon.com/leobenkel","https://www.patreon.com/leobenkel"],"categories":["Misc"],"sub_categories":[],"readme":"![ZparkIO logo](https://raw.githubusercontent.com/leobenkel/ZparkIO/main/assets/ZparkIO_animated.gif)\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![release-badge][]][release]\n[![maven-central-badge][]][maven-search]\n![CI](https://github.com/leobenkel/zparkio/workflows/CI/badge.svg)\n[![BCH compliance](https://bettercodehub.com/edge/badge/leobenkel/ZparkIO?branch=main)](https://bettercodehub.com/)\n[![Coverage Status](https://coveralls.io/repos/github/leobenkel/ZparkIO/badge.svg?branch=main)](https://coveralls.io/github/leobenkel/ZparkIO?branch=main)\n[![Mutation testing badge](https://img.shields.io/endpoint?style=flat\u0026url=https%3A%2F%2Fbadge-api.stryker-mutator.io%2Fgithub.com%2Fleobenkel%2FZparkIO%2Fmain)](https://dashboard.stryker-mutator.io/reports/github.com/leobenkel/ZparkIO/main)\n\n\n\n[release]:              https://github.com/leobenkel/zparkio/releases\n[release-badge]:        https://img.shields.io/github/tag/leobenkel/zparkio.svg?label=version\u0026color=blue\n[maven-search]:         https://search.maven.org/search?q=g:com.leobenkel%20a:zparkio*\n[maven-search-test]:         https://search.maven.org/search?q=g:com.leobenkel%20a:zparkio-test*\n[leobenkel-github-badge]:     https://img.shields.io/badge/-Github-yellowgreen.svg?style=social\u0026logo=GitHub\u0026logoColor=black\n[leobenkel-github-link]:      https://github.com/leobenkel\n[leobenkel-linkedin-badge]:     https://img.shields.io/badge/-Linkedin-yellowgreen.svg?style=social\u0026logo=LinkedIn\u0026logoColor=black\n[leobenkel-linkedin-link]:      https://linkedin.com/in/leobenkel\n[leobenkel-personal-badge]:     https://img.shields.io/badge/-Website-yellowgreen.svg?style=social\u0026logo=data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9JzMwMHB4JyB3aWR0aD0nMzAwcHgnICBmaWxsPSIjMDAwMDAwIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB2ZXJzaW9uPSIxLjEiIHg9IjBweCIgeT0iMHB4IiB2aWV3Qm94PSIwIDAgNjQgNjQiIGVuYWJsZS1iYWNrZ3JvdW5kPSJuZXcgMCAwIDY0IDY0IiB4bWw6c3BhY2U9InByZXNlcnZlIj48Zz48Zz48cGF0aCBkPSJNNDEuNiwyNy4yYy04LjMsMC0xNSw2LjctMTUsMTVzNi43LDE1LDE1LDE1YzguMywwLDE1LTYuNywxNS0xNVM0OS45LDI3LjIsNDEuNiwyNy4yeiBNNTEuNSwzNmgtMy4zICAgIGMtMC42LTEuNy0xLjQtMy4zLTIuNC00LjZDNDguMiwzMi4yLDUwLjIsMzMuOSw1MS41LDM2eiBNNDEuNiwzMS41YzEuMywxLjIsMi4zLDIuNywzLDQuNGgtNkMzOS4zLDM0LjIsNDAuNCwzMi43LDQxLjYsMzEuNXogICAgIE0zNy40LDMxLjNjLTEsMS40LTEuOCwyLjktMi40LDQuNmgtMy4zQzMzLjEsMzMuOSwzNS4xLDMyLjIsMzcuNCwzMS4zeiBNMzAuMyw0NWMtMC4yLTAuOS0wLjQtMS44LTAuNC0yLjhjMC0xLDAuMS0yLDAuNC0yLjkgICAgaDMuOWMtMC4xLDEtMC4yLDEuOS0wLjIsMi45YzAsMC45LDAuMSwxLjksMC4yLDIuOEgzMC4zeiBNMzEuNyw0OC4zSDM1YzAuNiwxLjcsMS40LDMuNCwyLjQsNC44QzM1LDUyLjIsMzMsNTAuNSwzMS43LDQ4LjN6ICAgICBNNDEuNiw1Mi45Yy0xLjMtMS4yLTIuMy0yLjgtMy4xLTQuNWg2LjFDNDQsNTAuMSw0Mi45LDUxLjcsNDEuNiw1Mi45eiBNMzcuNiw0NWMtMC4yLTAuOS0wLjItMS44LTAuMi0yLjhjMC0xLDAuMS0yLDAuMy0yLjloOCAgICBjMC4yLDAuOSwwLjMsMS45LDAuMywyLjljMCwxLTAuMSwxLjktMC4yLDIuOEgzNy42eiBNNDUuOCw1My4xYzEtMS40LDEuOC0zLDIuNC00LjhoMy4zQzUwLjIsNTAuNSw0OC4yLDUyLjIsNDUuOCw1My4xeiBNNDksNDUgICAgYzAuMS0wLjksMC4yLTEuOCwwLjItMi44YzAtMS0wLjEtMi0wLjItMi45aDMuOWMwLjIsMC45LDAuNCwxLjksMC40LDIuOWMwLDEtMC4xLDEuOS0wLjQsMi44SDQ5eiI+PC9wYXRoPjxwYXRoIGQ9Ik0zNCwyNS45Yy0wLjktMC43LTEuOC0xLjMtMi45LTEuOGMyLTIuMSwzLjItNC45LDMuMi03LjljMC02LjMtNS4xLTExLjQtMTEuNC0xMS40UzExLjYsOS45LDExLjYsMTYuMiAgICBjMCwzLjEsMS4yLDUuOSwzLjIsNy45Yy00LjEsMi02LjgsNS40LTcuMSw5LjRsLTAuMywzLjhjMCwyLDcsMy42LDE1LjYsMy42YzAuMiwwLDAuNSwwLDAuNywwQzI0LjIsMzQuMywyOC4yLDI4LjYsMzQsMjUuOXogICAgIE0yMyw4LjhjNC4xLDAsNy40LDMuMyw3LjQsNy40cy0zLjMsNy40LTcuNCw3LjRzLTcuNC0zLjMtNy40LTcuNFMxOC45LDguOCwyMyw4Ljh6Ij48L3BhdGg+PC9nPjwvZz48L3N2Zz4=\u0026logoColor=black\n[leobenkel-personal-link]:      https://leobenkel.com\n[leobenkel-patreon-link]:            https://www.patreon.com/leobenkel\n[leobenkel-patreon-badge]: https://img.shields.io/badge/-Patreon-yellowgreen.svg?style=social\u0026logo=Patreon\u0026logoColor=black\n[maven-central-link]:           https://maven-badges.herokuapp.com/maven-central/com.leobenkel/zparkio_2.11\n[maven-central-badge]:          https://maven-badges.herokuapp.com/maven-central/com.leobenkel/zparkio_2.11/badge.svg\n[maven-central-link-test]:           https://maven-badges.herokuapp.com/maven-central/com.leobenkel/zparkio-test_2.11\n[maven-central-badge-test]:          https://maven-badges.herokuapp.com/maven-central/com.leobenkel/zparkio-test_2.11/badge.svg\n\n\n# ZparkIO\nBoiler plate framework to use [Spark](https://github.com/apache/spark) and [ZIO](https://github.com/zio/zio) together.\n\nThe goal of this framework is to blend Spark and ZIO in an easy to use system for data engineers.\n\nAllowing them to use Spark in a new, faster, more reliable way, leveraging ZIO power.\n\n## Table of Contents\n\n* [What is this library for ?](#what-is-this-library-for-)\n* [More About ZparkIO](#more-about-zparkio)\n    * [Public Presentation](#public-presentation)\n    * [Migrate your Spark Project to ZparkIO](#migrate-your-spark-project-to-zparkio)\n* [Why would you want to use ZIO and Spark together?](#why-would-you-want-to-use-zio-and-spark-together)\n* [How to use?](#how-to-use)\n    * [Include dependencies](#include-dependencies)\n        * [Unit-test](#unit-test)\n    * [How to use in your code?](#how-to-use-in-your-code)\n        * [Main](#main)\n        * [Spark](#spark)\n        * [Command lines](#command-lines)\n        * [Helpers](#helpers)\n        * [Unit test](#unit-test-1)\n* [Examples](#examples)\n    * [Simple example](#simple-example)\n    * [More complex architecture](#more-complex-architecture)\n* [Authors](#authors)\n    * [Leo Benkel](#leo-benkel)\n\nCreated by [gh-md-toc](https://github.com/ekalinin/github-markdown-toc)\n\n## What is this library for ?\n\nThis library will implement all the boiler plate for you to be able to include Spark and ZIO in your ML project.\n\nIt can be tricky to use ZIO to save an instance of Spark to reuse in your code and this library solve all the boilerplate problem for you.\n\n## More About ZparkIO\n\n### Public Presentation\n\nFeel free to look at the slides on [Google Drive](https://docs.google.com/presentation/d/1gyFJpH2mzJ9ghSTsIMrUHWA9rCtSn2ML9ERUFvuYSp8) or on [SlideShare](https://www.slideshare.net/LeoBenkel/2020-0326-meet-up-z-parkio-230980911) presented during the [ScalaSF meetup](https://www.meetup.com/SF-Scala/events/268998404/) on Thursday, March 26, 2020. You can also watch [the presentation on Youtube](https://www.youtube.com/embed/Ov7WZroBkv0?start=507\u0026end=2416).\n\nZparkIO was on `version 0.7.0`, so things might be out of date.\n\n### Migrate your Spark Project to ZparkIO\n[Migrate from Plain Spark to ZparkIO](https://medium.com/@AyoubFakir/migrating-from-a-plain-spark-application-to-zio-with-zparkio-8fcd5f5da6ab)\n\n## Why would you want to use ZIO and Spark together?\n\nFrom my experience, using ZIO/Future in combination with Spark can speed up drastically the performance of your job. The reason being that sources (BigQuery, Postgresql, S3 files, etc...) can be fetch in parallel while the computation are not on hold. Obviously ZIO is much better than Future but it is harder to set up. Not anymore!\n\nSome other nice aspect of ZIO is the error/exception handling as well as the build-in retry helpers. Which make retrying failed task a breath within Spark.\n\n## How to use?\n\nI hope that you are now convinced that ZIO and Spark are a perfect match. Let's see how to use this Zparkio.\n\nOne of the easiest way to use ZparkIO is to use the [giter8 template project](https://github.com/leobenkel/zparkio.g8):\n\n```\nsbt new leobenkel/zparkio.g8\n```\n\n### Include dependencies\n\nFirst include the library in your project:\n\n```sbt\nlibraryDependencies += \"com.leobenkel\" %% \"zparkio\" % \"[SPARK_VERSION]_[VERSION]\"\n```\nWith version being: [![maven-central-badge][]][maven-search] [![release-badge][]][release].\n\nTo checkout out the [Spark Versions](https://github.com/leobenkel/ZparkIO/blob/main/sparkVersions) and the [Version](https://github.com/leobenkel/ZparkIO/blob/main/VERSION).\n\nThis library depends on [Spark](https://github.com/apache/spark), [ZIO](https://github.com/zio/zio) and [Scallop](https://github.com/scallop/scallop). \n\n#### Unit-test\n\nYou can also add \n\n```sbt\nlibraryDependencies += \"com.leobenkel\" %% \"zparkio-test\" % \"[VERSION]\"\n```\nWith version being: [![maven-central-badge-test][]][maven-search-test] .\n\nTo get access to helper function to help you write unit tests.\n\n### How to use in your code?\n\nThere is a [project example](https://github.com/leobenkel/ZparkIO/tree/main/examples/Example1_mini/src) you can look at. But here are the details.\n\n#### Main\n\nThe first thing you have to do is extends the `ZparkioApp` trait. For an example you can look at the [ProjectExample](https://github.com/leobenkel/ZparkIO/tree/main/examples/Example1_mini/src): [Application](https://github.com/leobenkel/ZparkIO/blob/main/examples/Example1_mini/src/main/scala/com/leobenkel/example1/Application.scala).\n\n#### Spark\n\nBy using this architecture, you will have access to `SparkSesion` anywhere in your `ZIO` code, via \n```scala\nimport com.leobenkel.zparkio.Services._\n\nfor {\n  spark \u003c- SparkModule()\n} yield {\n  ???\n}\n```\n\nfor instance you can see its use [here](https://github.com/leobenkel/ZparkIO/blob/cd958947d26996a15f100f397d8c471a07f047d3/examples/Example1_mini/src/main/scala/com/leobenkel/example1/Application.scala#L30).\n\n#### Command lines\n\nYou will also have access to all your command lines automatically parsed, generated and accessible to you via: \n\n[CommandLineArguments](https://github.com/leobenkel/ZparkIO/blob/cd958947d26996a15f100f397d8c471a07f047d3/examples/Example1_mini/src/main/scala/com/leobenkel/example1/Arguments.scala#L11-L22) ;\nit is recommended to make this helper function to make the rest of your code easier to use.\n\nThen using it, [like here](https://github.com/leobenkel/ZparkIO/blob/cd958947d26996a15f100f397d8c471a07f047d3/examples/Example1_mini/src/main/scala/com/leobenkel/example1/Application.scala#L29), is easy.\n\n#### Helpers\n\nIn the [implicits](https://github.com/leobenkel/Zparkio/blob/main/Library/src/main/scala/com/leobenkel/zparkio/implicits.scala) object, that you can include everywhere. You are getting specific helper functions to help streamline your projects.\n\n#### Unit test\n\nUsing this architecture will literally allow you to [run your main as a unit test](https://github.com/leobenkel/ZparkIO/blob/main/examples/Example1_mini/src/test/scala/com/leobenkel/example1/ApplicationTest.scala).\n\n## Examples\n\n### Simple example\n\nTake a look at the simple project example to see example of working code using this library: \n[SimpleProject](https://github.com/leobenkel/ZparkIO/tree/main/examples/Example1_mini).\n\n### More complex architecture\n\nA full-fledged, production-ready project will obviously need more code than the simple example.\nFor this purpose, and upon suggestion of several awesome people, I added a more complex project.\nThis is a WIP and more will be added as I go. \n[MoreComplexProject](https://github.com/leobenkel/ZparkIO/tree/main/examples/Example2_small/src).\n\n## Authors\n\n### Leo Benkel\n\n* [![leobenkel-github-badge][]][leobenkel-github-link]\n* [![leobenkel-linkedin-badge][]][leobenkel-linkedin-link]\n* [![leobenkel-personal-badge][]][leobenkel-personal-link]\n* [![leobenkel-patreon-badge][]][leobenkel-patreon-link]\n\n## Alternatives\n\n* [univalence/zio-spark](https://github.com/univalence/zio-spark)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleobenkel%2FZparkio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleobenkel%2FZparkio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleobenkel%2FZparkio/lists"}