{"id":19506801,"url":"https://github.com/calvinlfer/etl-workflow","last_synced_at":"2025-07-06T22:33:17.797Z","repository":{"id":68736731,"uuid":"127667758","full_name":"calvinlfer/etl-workflow","owner":"calvinlfer","description":"An opinionated way to structure ETL pipelines with a heavy focus on reusability and testing","archived":false,"fork":false,"pushed_at":"2018-10-07T06:26:29.000Z","size":37,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-01T05:07:25.412Z","etag":null,"topics":["cats","etl","functional-programming","scala"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/calvinlfer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-01T20:13:19.000Z","updated_at":"2021-03-20T21:47:50.000Z","dependencies_parsed_at":"2023-03-18T16:38:43.535Z","dependency_job_id":null,"html_url":"https://github.com/calvinlfer/etl-workflow","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/calvinlfer/etl-workflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2Fetl-workflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2Fetl-workflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2Fetl-workflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2Fetl-workflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/calvinlfer","download_url":"https://codeload.github.com/calvinlfer/etl-workflow/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2Fetl-workflow/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263983609,"owners_count":23539558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cats","etl","functional-programming","scala"],"created_at":"2024-11-10T22:38:29.310Z","updated_at":"2025-07-06T22:33:17.744Z","avatar_url":"https://github.com/calvinlfer.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ETL Workflow _(beta)_ #\n\n[![Build Status](https://travis-ci.org/calvinlfer/etl-workflow.svg?branch=master)](https://travis-ci.org/calvinlfer/etl-workflow)\n[![Download](https://api.bintray.com/packages/calvinlfer/maven/etl-workflow/images/download.svg) ](https://bintray.com/calvinlfer/maven/etl-workflow/_latestVersion)\n\n**ETL Workflow** is a simple and *opinionated* way to help you structure type-safe Extract-Transform-Load (**ETL**) \npipelines. This Domain Specific Language (DSL) is flexible enough to create linear pipelines which involve a single \n`Extract` source and `Load` sink \n\n```\nExtract source A ~\u003e Transform A to B ~\u003e Load B (sink 1)\n```\n\nall the way to stitching multiple Extract sources together and flowing the data through to multiple Load sinks\n\n```\n\nExtract source A ~\u003e                               ~\u003e Load D (sink 1)\n                   \\                             /\nExtract source B    ~\u003e Transform (A, B, C) to D ~\u003e   Load D (sink 2)\n                   /                             \\\nExtract source C ~\u003e                               ~\u003e Load D (sink 3)\n\n``` \n\nIt is built on an immutable and functional architecture where side-effects are executed at the end-of-the-world when the \npipeline is run. \n\nThis is intended to be used in conjunction with Spark (especially for doing ETL) in order to minimize boilerplate and \nhave the ability to see an almost whiteboard-like representation of your pipeline.\n\n## Usage ##\n```sbt\nresolvers += Resolver.bintrayRepo(\"calvinlfer\",\"maven\")\n\nlibraryDependencies += \"com.ghostsequence %% \"etl-workflow\" % \u003clatest version info at the top\u003e\"\n```\n\n## Building Blocks ##\n\nAn ETL pipeline consists of the following building blocks:\n\n#### `Extract[A]` ####\nA producer of a single element of data whose type is `A`. This is the start of the ETL pipeline, you can connect this\nto `Transform`ers or to a `Load[A, AStatus]` to create an `ETLPipeline[AStatus]` that can be run.\n\n#### `Transform[A, B]` ####\nA transformer of a an element `A` to `B` you can attach these after an `Extract[A]` or before a `Load[B]`\n\n#### `Load[B, BStatus]` ####\nThe end of the pipeline which takes data `B` flowing through the pipeline and consumes it and produces a status \n`BStatus` which indicates whether consumption happens successfully\n\n#### `ETLPipeline[ConsumeStatus]` ####\nThis represents the fully created ETL pipeline which can be executed using `unsafeRunSync()` to produce a \n`ConsumeStatus` which indicates whether the pipeline has finished successfully.\n\n**Note:** At the end of the day, these building blocks are a reification of values and functions. You can build an \nETL pipeline out of functions and values but it helps to have a Domain Specific Language to increase readability.\n\n## Examples ##\nSee [here](src/main/tut/Examples.md) for examples on how to get started\n\n### Inspiration ###\n\n- [Mario](https://github.com/intentmedia/mario)\n- [Akka Streams](https://doc.akka.io/docs/akka/2.5/stream/index.html)\n- [Monix Observables](https://monix.io)\n\n\n### Release process ###\nMake sure you have the correct [Bintray credentials](http://queirozf.com/entries/publishing-an-sbt-project-onto-bintray-an-example)\nbefore proceeding:\n\n```bash\nsbt release\n```\n\nThis will automatically create a Git Tag and publish the library to Bintray for all Scala versions.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcalvinlfer%2Fetl-workflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcalvinlfer%2Fetl-workflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcalvinlfer%2Fetl-workflow/lists"}