{"id":24534398,"url":"https://github.com/mattlianje/etl4s","last_synced_at":"2026-02-20T05:01:36.194Z","repository":{"id":267358942,"uuid":"899813922","full_name":"mattlianje/etl4s","owner":"mattlianje","description":"Powerful, whiteboard-style ETL","archived":false,"fork":false,"pushed_at":"2026-02-17T05:52:04.000Z","size":27520,"stargazers_count":72,"open_issues_count":3,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-02-17T11:26:03.062Z","etag":null,"topics":["big-data","data-engineering","etl","functional-programming","streaming"],"latest_commit_sha":null,"homepage":"https://etl4s.dev","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mattlianje.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-07T04:51:11.000Z","updated_at":"2026-02-17T05:51:49.000Z","dependencies_parsed_at":"2026-01-07T03:08:36.789Z","dependency_job_id":null,"html_url":"https://github.com/mattlianje/etl4s","commit_stats":null,"previous_names":["mattlianje/etl4s"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/mattlianje/etl4s","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattlianje%2Fetl4s","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattlianje%2Fetl4s/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattlianje%2Fetl4s/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattlianje%2Fetl4s/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mattlianje","download_url":"https://codeload.github.com/mattlianje/etl4s/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattlianje%2Fetl4s/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29641929,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-20T03:21:14.183Z","status":"ssl_error","status_checked_at":"2026-02-20T03:18:24.455Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","data-engineering","etl","functional-programming","streaming"],"created_at":"2025-01-22T11:17:07.854Z","updated_at":"2026-02-20T05:01:36.160Z","avatar_url":"https://github.com/mattlianje.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"pix/etl4s-2.png\" width=\"700\"\u003e\n\u003c/p\u003e\n\n# \u003cimg src=\"pix/etl4s-logo.png\" width=\"50\"\u003e etl4s\n**Powerful, whiteboard-style ETL**\n\nA lightweight, zero-dependency library for writing type-safe, beautiful ✨🍰  data flows in functional Scala. \nBattle-tested at [Instacart](https://www.instacart.com/). Part of [d4](https://github.com/mattlianje/d4).\n\n## Features\n- Declarative, typed pipeline endpoints\n- Use **Etl4s.scala** like a header file\n- Type-safe, compile-time checked\n- [Config-driven](#configuration) by design\n- Easy, monadic composition of pipelines\n- Built-in retry/failure handling\n- Automatic [tracing](#introspection-with-etl4strace)\n- Drop-in [telemetry](#telemetry)\n- [Data lineage](#lineage) visualization\n\n## Installation\n\n**etl4s** is on MavenCentral and cross-built for Scala, 2.12, 2.13, 3.x\n```scala\n\"xyz.matthieucourt\" %% \"etl4s\" % \"1.9.1\"\n```\nOr try in REPL:\n```bash\nscala-cli repl --scala 3 --dep xyz.matthieucourt:etl4s_3:1.9.1\n```\n\nAll you need:\n```scala\nimport etl4s._\n```\n\n## Quick Example\n```scala\nimport etl4s._\n\nval getUser  = Extract(\"John Doe\")\nval getOrder = Extract(\"Order #1234\")\nval combine  = Transform[(String, String), String] { case (user, order) =\u003e\n  s\"$user placed $order\"\n}\nval saveDb    = Load[String, String](s =\u003e { println(s\"DB: $s\"); s })\nval sendEmail = Load[String, Unit](s =\u003e println(s\"Email: $s\"))\n\nval pipeline = (getUser \u0026 getOrder) ~\u003e combine ~\u003e (saveDb \u0026 sendEmail)\n\npipeline.unsafeRun()\n```\n\n## Why etl4s?\n\n- Ultimately, these nodes and pipelines are just reifications of functions and values with a few extra niceties.\n- Chaotic, framework-coupled ETL codebases that grow without an imposed discipline drive dev teams and data orgs to their knees.\n- **etl4s** is a lightweight DSL to enforce discipline, type-safety, and reuse of pure functions - and see [functional ETL](https://maximebeauchemin.medium.com/functional-data-engineering-a-modern-paradigm-for-batch-data-processing-2327ec32c42a) for what it is... and could be.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eBut seriously, why etl4s? Why not raw functions??\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\n- **Clean, easy to edit graphs**: Raw function composition can obscure the high-level flow of data and monadic-stacks, although mightily useful and time-tested, don't impose a strict discipline on assignment and creating new bindings. **etl4s** uses a declarative DSL (`~\u003e`, `\u0026`, `\u0026\u003e`) to define pipelines as explicit, type-safe graphs. This makes your data flows easy to read, reason about, and modify: like a whiteboard diagram.\n\n- **Reusable, typed endpoints**: Pipelines are declarative values with clear contracts (`Node[In, Out]`). Share them across teams as portable, composable components or libs.\n\n- **Built-in resilience and parallelism**: Instead of manually writing boilerplate for error handling and concurrency, **etl4s** provides clean, chainable methods. Add automatic retries with `.withRetry`, handle failures with `.onFailure`, and run tasks in parallel with the `\u0026\u003e` operator, keeping your core logic clean.\n\n- **Automatic state tracking**: Pipeline steps often need to react to upstream events - validation failures, warnings, timeouts, but threading state manually through function calls is painful. **etl4s** uses ThreadLocal `Trace` channels that flow automatically: downstream steps can check `Trace.hasErrors`, `Trace.getLogs`, or `Trace.getElapsedTimeMillis` without any wiring. Call `.unsafeRunTrace()` for full execution details.\n\n- **Metrics by design**: In ETL, metrics aren't just infra-monitoring, they're business logic... especially at the peripheries in Extractors and Loaders. Yet, metric collection is typically bolted on afterwards, or run as side-processes. **etl4s** bakes the `Etl4sTelemetry` interface into every pipeline. Add counters, gauges, and histograms directly in your business logic with `Tel` calls (zero-cost until you provide an implementation). Works with any backend: Prometheus, DataDog, OpenTelemetry.\n\n- **Lineage visualization for free**: Because **etl4s** pipelines are data structures, you can attach metadata and automatically generate lineage diagrams with `.toMermaid` or `.toDot`... impossible with plain functions.\n\n- **Clean configuration and dependency management**: Avoid \"parameter drilling\" configuration objects through nested functions. **etl4s** provides a simple dependency injection system (`.requires` and `.provide`) that automatically infers and injects the minimal required configuration for any part of your pipeline.\n\n\u003c/details\u003e\n\n\n## Core Concepts\n**etl4s** has one core building block:\n```scala\nNode[-In, +Out]\n```\nA Node wraps a lazily-evaluated function `In =\u003e Out`. Chain them with `~\u003e` to build pipelines.\n\nTo improve readability and express intent, **etl4s** defines four aliases: `Extract`, `Transform`, `Load` and `Pipeline`. All behave the same under the hood.\n\n```scala\nval step = Transform[String, Int](_.length)\nstep(\"hello\")  // 5\n```\n\n**Running pipelines:**\n- `pipeline(input)` - call like a function\n- `.unsafeRun(input)` - explicit run\n- `.safeRun(input)` - returns `Try[Out]`\n- `.unsafeRunTrace(input)` - returns `Trace` (logs, timing, errors)\n- `.safeRunTrace(input)` - returns `Trace` with `Try[Out]`\n\n**DI:** Use `.requires` to turn any Node into a `Reader[Config, Node]`. The `~\u003e` operator works between Nodes and Readers. See [Configuration](#configuration).\n\n## Type safety\n**etl4s** won't let you chain together \"blocks\" that don't fit together:\n```scala\n val fiveExtract: Extract[Unit, Int]        = Extract(5)\n val exclaim:     Transform[String, String] = Transform(_ + \"!\")\n\n fiveExtract ~\u003e exclaim\n```\nThe above will not compile with:\n```shell\n-- [E007] Type Mismatch Error: -------------------------------------------------\n4 | fiveExtract ~\u003e exclaim\n  |                ^^^^^^^\n  |                Found:    (exclaim : Transform[String, String])\n  |                Required: Node[Int, Any]\n```\n\n## Operators\n\netl4s uses a few simple operators to build pipelines:\n\n| Operator | Name | Description | Example |\n|----------|------|-------------|---------|\n| `~\u003e` | Connect | Chains operations in sequence | `e1 ~\u003e t1 ~\u003e l1` |\n| `\u0026` | Combine | Group sequential operations with same input | `t1 \u0026 t2` |\n| `\u0026\u003e` | Parallel | Group concurrent operations with same input | `t1 \u0026\u003e t2` |\n| `\u003e\u003e` | Sequence | Runs nodes in order with same input | `p1 \u003e\u003e p2` |\n\n## Configuration\n\nDeclare what each step `.requires`, then `.provide` it later:\n\n```scala\nimport etl4s._\n\ncase class Cfg(key: String)\n\nval A = Extract(\"data\")\nval B = Transform[String, String].requires[Cfg] { cfg =\u003e data =\u003e\n  s\"${cfg.key}: $data\"\n}\n\nval pipeline = A ~\u003e B\n\npipeline.provide(Cfg(\"secret\")).unsafeRun(())  /* \"secret: data\" */\n\n/** NOTE (Scala 2.x)\n  * Use: `Node.requires[Cfg, In, Out](cfg =\u003e in =\u003e out)` syntax\n  */\n```\n\n**etl4s** automatically infers the smallest shared config for your pipeline. Just `.provide` once.\n\nRead more [here](https://mattlianje.github.io/etl4s/config/)\n\n## Parallelizing Tasks\n**etl4s** has an elegant shorthand for grouping and parallelizing operations that share the same input type:\n```scala\n/* Simulate slow IO operations (e.g: DB calls, API requests) */\n\nval e1 = Extract { Thread.sleep(100); 42 }\nval e2 = Extract { Thread.sleep(100); \"hello\" }\nval e3 = Extract { Thread.sleep(100); true }\n```\n\nSequential run of e1, e2, and e3 **(~300ms total)**\n```scala\nval sequential: Extract[Unit, (Int, String, Boolean)] =\n     e1 \u0026 e2 \u0026 e3\n```\n\nParallel run of e1, e2, e3 on their own JVM threads with Scala Futures **(~100ms total, same result, 3X faster)**\n```scala\nimport scala.concurrent.ExecutionContext.Implicits.global\n\nval parallel: Extract[Unit, (Int, String, Boolean)] =\n     e1 \u0026\u003e e2 \u0026\u003e e3\n```\n\nMix sequential and parallel execution (first two parallel (~100ms), then third (~100ms)):\n```scala\nval mixed = (e1 \u0026\u003e e2) \u0026 e3\n```\n\nFull example of a parallel pipeline:\n```scala\nval consoleLoad: Load[String, Unit] = Load(println(_))\nval dbLoad:      Load[String, Unit] = Load(x =\u003e println(s\"DB Load: ${x}\"))\n\nval merge = Transform[(Int, String, Boolean), String] { case (i, s, b) =\u003e\n    s\"$i-$s-$b\"\n  }\n\nval pipeline =\n  (e1 \u0026\u003e e2 \u0026\u003e e3) ~\u003e merge ~\u003e (consoleLoad \u0026\u003e dbLoad)\n```\n\n## Handling Failures\n\n#### `withRetry`\nRetry failed operations:\n```scala\nimport etl4s._\n\nvar n = 0\nval A = Transform[Int, String] { x =\u003e\n  n += 1\n  if (n \u003c 3) throw new RuntimeException(\"fail\")\n  else \"ok\"\n}.withRetry(maxAttempts = 3, initialDelayMs = 10)\n\nExtract(42) ~\u003e A  /* Succeeds on 3rd attempt */\n```\n\n#### `onFailure`\nCatch exceptions and recover:\n```scala\nimport etl4s._\n\nval A = Extract[Unit, String](_ =\u003e throw new RuntimeException(\"Boom!\"))\n  .onFailure(e =\u003e s\"Error: ${e.getMessage}\")\n\nA.unsafeRun(())  /* Returns \"Error: Boom!\" */\n```\n\n## Conditional Branching\n\nRoute data through different pipelines with `If`, `ElseIf`, and `Else`:\n\n```scala\nval pipeline = extractUser\n  .If(_.tier == \"premium\")      (validateUser ~\u003e enrichUser ~\u003e toPremiumOffer)\n  .ElseIf(_.tier == \"standard\") (validateUser ~\u003e toStandardOffer)\n  .Else                         (toGuestNotice)\n```\n\nBranch on config only with `IfCtx`/`ElseIfCtx`:\n```scala\nval pipeline = sourceReader\n  .IfCtx(_.isBackfill)(backfillBranch)\n  .ElseIfCtx(_.isDryRun)(dryRunBranch)\n  .Else(normalBranch)\n```\n\nPlain `Node` branches are automatically lifted to `Reader` when mixed with config-aware branches - no manual wrapping needed.\n\nRead more [here](https://mattlianje.github.io/etl4s/branching/).\n\n## Side Effects\nUse `.tap()` for side effects without disrupting pipeline flow:\n\n```scala\nimport etl4s._\n\nval A: Extract[Any, List[String]] = Extract(_ =\u003e List(\"a.txt\", \"b.txt\"))\n                                       .tap(files =\u003e println(s\"Processing: $files\"))\n\nval B = Transform[List[String], Int](_.size)\n\nA ~\u003e B\n```\n\nChain side effects with `\u003e\u003e`:\n```scala\nval logStart = Node { println(\"Starting...\") }\nval logEnd   = Node { println(\"Done!\") }\n\nval pipeline = logStart \u003e\u003e (A ~\u003e B) \u003e\u003e logEnd\npipeline.unsafeRun()\n```\n\n## Tracing\nNodes can access and update their runtime state with ThreadLocal channels spawned for free. All state is automatically shared across your entire pipeline. Read more [here](https://mattlianje.github.io/etl4s/trace/)\n\n```scala\nval A = Transform[String, Int] { s =\u003e\n  if (s.isEmpty) Trace.error(\"empty\")\n  s.length\n}\n\nval B = Transform[Int, String] { n =\u003e\n  if (Trace.hasErrors) \"FALLBACK\" else s\"len: $n\"  \n}\n\n(A ~\u003e B).unsafeRun(\"\")  /* \"FALLBACK\" */\n```\n\n## Telemetry\netl4s provides a minimal `Etl4sTelemetry` interface for observability. All pipeline run methods automatically look for this interface in implicit scope.\n\n`Tel` is etl4s's telemetry API object with the same method names as the trait for consistency. All `Tel` calls are no-ops by default - zero overhead until you provide an implementation.\n\n```scala\nval A = Transform[List[String], Int] { data =\u003e\n  Tel.withSpan(\"op\") {\n    Tel.addCounter(\"n\", data.size)\n    Tel.setGauge(\"v\", data.size.toDouble)\n    data.map(_.length).sum\n  }\n}\n\n/* By default Tel calls are no-ops (zero cost) */\nA.unsafeRun(data)\n\n/* Implement Etl4sTelemetry for your backend */\nimplicit val telemetry: Etl4sTelemetry = MyPrometheusProvider()\nA.unsafeRun(data) /* metrics flow to Prometheus */\n```\n\nThe `Etl4sTelemetry` interface has just 4 methods: `withSpan`, `addCounter`, `setGauge`, `recordHistogram`\nwhich cover 95% of observability needs.\n\n`unsafeRunTrace` captures all `Tel` calls as structured `TelemetryData` with OTLP-compatible spans and metrics:\n```scala\nval trace = pipeline.unsafeRunTrace(data)\ntrace.spans              // List[TelSpan] with traceId, parentSpanId, timing\ntrace.counterTotals      // Map[String, Long]\ntrace.toOtelJson         // OTLP JSON for OpenTelemetry collectors\n```\n\nRead more in the [Telemetry guide](https://mattlianje.github.io/etl4s/opentelemetry/).\n\n## Lineage\n\nTrack data lineage and visualize pipeline dependencies. Attach metadata to any Node or Reader then call `.toDot`, `.toJson` or `.toMermaid`\non individual instances or on Sequences:\n\n```scala\nval A = Node[String, String](identity)\n  .lineage(\n    name = \"A\",\n    inputs = List(\"s1\", \"s2\"),\n    outputs = List(\"s3\"), \n    schedule = \"0 */2 * * *\"\n  )\n\nval B = Node[String, String](identity)\n  .lineage(\n    name = \"B\",\n    inputs = List(\"s3\"),\n    outputs = List(\"s4\", \"s5\")\n  )\n```\n\nExport lineage as JSON, DOT (Graphviz), or Mermaid diagrams:\n\n```scala\nSeq(A, B).toJson\nSeq(A, B).toDot\n```\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"pix/graphviz-example.svg\" width=\"500\"\u003e\n\u003c/p\u003e\n\n```scala\nSeq(A, B).toMermaid\n```\n```mermaid\ngraph LR\n    classDef pipeline fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000\n    classDef dataSource fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000\n    classDef cluster fill:#e8f5e8,stroke:#2e7d32,stroke-width:2px,color:#000\n\n    A[\"A\u003cbr/\u003e(0 */2 * * *)\"]\n    B[\"B\"]\n    s1([\"s1\"])\n    s2([\"s2\"])\n    s3([\"s3\"])\n    s4([\"s4\"])\n    s5([\"s5\"])\n\n    s1 --\u003e A\n    s2 --\u003e A\n    A --\u003e s3\n    s3 --\u003e B\n    B --\u003e s4\n    B --\u003e s5\n    A -.-\u003e B\n    linkStyle 6 stroke:#ff6b35,stroke-width:2px\n\n    class A pipeline\n    class B pipeline\n    class s1 dataSource\n    class s2 dataSource\n    class s3 dataSource\n    class s4 dataSource\n    class s5 dataSource\n```\n\n**etl4s** automatically infers dependencies by matching output -\u003e input sources. Nodes don't need to be connected with `~\u003e` for lineage tracking. Explicit dependencies via `upstreams` also supported.\n\n## Examples\n\n#### Chain two pipelines\nSimple UNIX-pipe style chaining of two pipelines:\n```scala\nimport etl4s._\n\nval p1 = Pipeline((i: Int) =\u003e i.toString)\nval p2 = Pipeline((s: String) =\u003e s + \"!\")\n\nval p3 = p1 ~\u003e p2\n```\n\n#### Complex chaining\nConnect the output of two pipelines to a third:\n```scala\nimport etl4s._\n\nval namePipeline = Pipeline(\"John Doe\")\nval agePipeline  = Pipeline(30)\nval toUpper      = Transform[String, String](_.toUpperCase)\nval consoleLoad  = Load[String, Unit](println(_))\n\nval combined =\n  for {\n    name \u003c- namePipeline\n    age \u003c- agePipeline\n    _ \u003c- Extract(s\"$name | $age\") ~\u003e toUpper ~\u003e consoleLoad\n  } yield ()\n```\n\n## Real-world examples\n**etl4s** works great with anything:\n- Spark / Flink / Beam\n- ETL / Streaming\n- Distributed Systems\n- Local scripts\n- Big Data workflows\n- Web-server dataflows\n\n## Inspiration\n- Debasish Ghosh's [Functional and Reactive Domain Modeling](https://www.manning.com/books/functional-and-reactive-domain-modeling)\n- [Akka Streams DSL](https://doc.akka.io/libraries/akka-core/current/stream/stream-graphs.html#constructing-graphs)\n- Various Rich Hickey talks\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmattlianje%2Fetl4s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmattlianje%2Fetl4s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmattlianje%2Fetl4s/lists"}