{"id":15060791,"url":"https://github.com/tharwaninitin/etlflow","last_synced_at":"2025-02-28T04:12:01.300Z","repository":{"id":42872804,"uuid":"226845693","full_name":"tharwaninitin/etlflow","owner":"tharwaninitin","description":"EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for running complex Auditable workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases, SFTP servers, On-Prem Systems and more.","archived":false,"fork":false,"pushed_at":"2024-08-26T05:34:23.000Z","size":14047,"stargazers_count":44,"open_issues_count":0,"forks_count":12,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-02-21T03:07:29.768Z","etag":null,"topics":["aws","bigquery","dataproc","etl","etl-framework","etl-pipeline","gcp","gcs","redis","s3","scala","spark","zio"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tharwaninitin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-09T10:32:23.000Z","updated_at":"2024-11-22T16:41:28.000Z","dependencies_parsed_at":"2024-08-26T06:49:11.953Z","dependency_job_id":"7e129a7e-5833-480f-b08c-83c0c0e25211","html_url":"https://github.com/tharwaninitin/etlflow","commit_stats":null,"previous_names":["tharwaninitin/etljobs"],"tags_count":39,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tharwaninitin%2Fetlflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tharwaninitin%2Fetlflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tharwaninitin%2Fetlflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tharwaninitin%2Fetlflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tharwaninitin","download_url":"https://codeload.github.com/tharwaninitin/etlflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241098935,"owners_count":19909447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","bigquery","dataproc","etl","etl-framework","etl-pipeline","gcp","gcs","redis","s3","scala","spark","zio"],"created_at":"2024-09-24T23:04:38.945Z","updated_at":"2025-02-28T04:12:01.283Z","avatar_url":"https://github.com/tharwaninitin.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# EtlFlow\n\n[![License](http://img.shields.io/:license-Apache%202-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0.txt)\n[![EtlFlow CI](https://github.com/tharwaninitin/etlflow/actions/workflows/ci.yml/badge.svg)](https://github.com/tharwaninitin/etlflow/actions/workflows/ci.yml)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-core_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-core)\n[![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-core_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-core_2.12)\n\n**EtlFlow** is an ecosystem of functional libraries in Scala based on ZIO for running complex **_Auditable_** jobs/workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases and more.\n\nBelow are some important features of this library, some of which come from ZIO.\n\n- **Universal**. It provides a consistent way to interact with different services/products across cloud platforms like GCP, AWS, Azure, and On-Premises systems.\n- **Functional**. Rapidly compose complex **jobs/workflows** from simple tasks.\n- **Auditable**. Build jobs/workflows that provides auditability by default for multiple backends.\n- **Resource-safe**. Build jobs/workflows that never leak resources (including threads!), even when they fail.\n- **Compile Time DI**. Build jobs/workflows that allows resolving dependencies at compile time.\n- **Fibers**. Built on non-blocking fibers that never waste or leak resources, which lets you build scalable, resilient, and reactive applications\n- **Concurrent and Asynchronous**. Easily build concurrent asynchronous or synchronous jobs/workflows without deadlocks, race conditions, or complexity.\n- **Type-safe**. Use the full power of the Scala compiler to catch bugs at compile time.\n- **Testable**. Inject test services into your job/workflow for fast, deterministic, and type-safe testing.\n- **Resilient**. Build jobs/workflows that never lose errors, and which respond to failure locally and flexibly.\n\n[//]: # (__Library Documentation__  https://tharwaninitin.github.io/etlflow/site/docs)\n[//]: \u003c\u003e (__Scala Test Coverage Report__  https://tharwaninitin.github.io/etlflow/testcovrep/)\n\n## Examples\n* [Core Module](examples/examplecore):         \n  In this example project, you can explore core features of etlflow, Task and Audit API.\n* [GCP Module (GCS, DataProc, BigQuery tasks)](examples/examplegcp):         \n  In this example project, you can explore GCP tasks.\n* [K8S Module (K8S tasks)](examples/examplek8s):         \n  In this example project, you can explore different Kubernetes tasks.\n* [Spark Module (Spark tasks)](examples/examplespark):         \n  In this example project, you can explore different Apache Spark tasks.\n\n## Modules Dependency Graph\n\n![ModuleDepGraph](moduleDep.png)\n\n| Module | Latest Version                                                                                                                                                                                         |                                                                                                                                                       Documentation | Scala Versions                                                                                                                                                                                           | Java Version | \n|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------:|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|\n| Core   | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-core_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-core)   |         [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-core_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-core_3) | [![etlflow-core Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-core/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-core)    | Java 8 +     |\n| GCP    | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-gcp_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-gcp)     |           [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-gcp_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-gcp_3) | [![etlflow-gcp Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-gcp/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-gcp)       | Java 8 +     |\n| JDBC   | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-jdbc_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-jdbc)   |         [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-jdbc_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-jdbc_3) | [![etlflow-jdbc Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-jdbc/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-jdbc)    | Java 8 +     |\n| Http   | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-http_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-http)   |         [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-http_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-http_3) | [![etlflow-http Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-http/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-http)    | Java 11 +    |\n| K8S    | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-k8s_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-k8s)     |           [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-k8s_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-k8s_3) | [![etlflow-k8s Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-k8s/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-k8s)       | Java 8 +     |\n| Email  | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-email_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-email) |       [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-email_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-email_3) | [![etlflow-email Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-email/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-email) | Java 8 +     |\n| AWS    | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-aws_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-aws)     |           [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-aws_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-aws_3) | [![etlflow-aws Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-aws/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-aws)       | Java 8 +     |\n| FTP    | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-ftp_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-ftp)     |           [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-ftp_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-ftp_3) | [![etlflow-ftp Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-ftp/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-ftp)       | Java 8 +     |\n| Redis  | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-redis_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-redis) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-redis_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-redis_2.12) | [![etlflow-redis Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-redis/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-redis) | Java 8 +     |\n| Spark  | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-spark_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-spark) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-spark_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-spark_2.12) | [![etlflow-spark Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-spark/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-spark) | Java 8 +     |\n\n## Requirements and Installation\nThis project is compiled with scala versions 2.12.17, 2.13.10, 3.3.0\n\nAvailable via [maven central](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-core).\nAdd the below latest release as a dependency to your project\n\n[![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-core_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-core)\n\n__SBT__\n```scala\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-core\" % \"1.7.3\"\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-gcp\" % \"1.7.3\"\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-jdbc\" % \"1.7.3\"\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-spark\" % \"1.7.3\"\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-k8s\" % \"1.7.3\"\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-http\" % \"1.7.3\"\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-ftp\" % \"1.7.3\"\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-redis\" % \"1.7.3\"\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-aws\" % \"1.7.3\"\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-email\" % \"1.7.3\"\n```\n__Maven__\n```\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.tharwaninitin\u003c/groupId\u003e\n    \u003cartifactId\u003eetlflow-core_2.12\u003c/artifactId\u003e\n    \u003cversion\u003e1.7.3\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n# Etlflow Modules\n\u003c!-- TOC --\u003e\n- [Etlflow Modules](#etlflow-modules)\n  - [Core](#core)\n    - [Task API](#task-api)\n    - [Audit API](#audit-api)\n    - [Job API](#job-api)\n  - [GCP](#gcp)\n    - [Dataproc](#dataproc)\n  - [JDBC](#jdbc)\n  - [K8S](#k8s)\n  - [Http](#http)\n  - [Email](#email)\n  - [Aws](#aws)\n  - [Redis](#redis)\n  - [Spark](#spark)\n\u003c!-- /TOC --\u003e\n\n## Core\nThe core module provides **Task** and **Audit** APIs, which are used by all tasks in different modules. It also provides a **Job** API that facilitates grouping multiple tasks together to leverage **auditing** and **logging** capabilities at the job/workflow level.\n### Task API\nBelow is the simplest example of **creating** a **Task** and **running** it using EtlFlow.\nThis example uses the noop audit backend, which does nothing. This is useful when you want to test a task that requires an audit backend to be passed in.\n\n```scala\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-core\" % \"1.7.3\"\n```\n\n```scala\nimport etlflow.audit.Audit\nimport etlflow.task.GenericTask\nimport zio._\n\nobject Job1 extends ZIOAppDefault {\n\n  def executeTask(): Task[Unit] = ZIO.logInfo(s\"Hello EtlFlow Task\")\n\n  val genericTask1: GenericTask[Any, Unit] = GenericTask(\n    name = \"Generic Task\",\n    task = executeTask()\n  )\n\n  val task1: RIO[Audit, Unit] = genericTask1.toZIO\n\n  override def run: Task[Unit] = task1.provide(etlflow.audit.noop)\n}\n```\n### Audit API\nEtlFlow provides an auditing interface that can be used to track the execution of tasks and jobs (collections of tasks). \nThe auditing interface is integrated with the Task Interface. \nEach task uses this interface to maintain the state of all tasks in the job/workflow in the backend of choice for end-to-end auditability. \nCurrently, there are audit backend implementations available for BigQuery, MySQL, and Postgres. \nAudit has a simple and concise interface, which makes it quite easy to add any new backend.\n```scala\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-core\" % \"1.7.3\"\nlibraryDependencies += \"com.github.tharwaninitin\" %% \"etlflow-jdbc\" % \"1.7.3\"\n```\n```scala\nimport etlflow.task.{GenericTask, DBQueryTask}\nimport etlflow.model.Credential.JDBC\nimport zio._\n\nobject Job2 extends ZIOAppDefault {\n  \n  private val task1 = GenericTask(\n    name = \"Generic Task 1\",\n    task = ZIO.logInfo(s\"Task 1\")\n  )\n\n  private val task2 = GenericTask(\n    name = \"Generic Task 2\",\n    task = ZIO.logInfo(s\"Task 2\")\n  )\n\n  val job = for {\n    _ \u003c- task1.toZIO\n    _ \u003c- task2.toZIO\n  } yield ()\n\n  private val cred = JDBC(sys.env(\"DB_URL\"), sys.env(\"DB_USER\"), sys.env(\"DB_PWD\"), sys.env(\"DB_DRIVER\"))\n\n  override def run: Task[Unit] = job.provide(etlflow.audit.DB(cred))\n}\n```\nHere's a snapshot of data for the `task_run` table after this job has run:\n\n| task_run_id | job_run_id | task_name | task_type   | metadata | status  | created_at              | modified_at             |\n|-------------|------------|-----------|-------------|----------|---------|-------------------------|-------------------------|\n| 1           | 100        | Task 1    | GenericTask | {}       | SUCCESS | 2023-07-13 10:00:00 UTC | 2023-07-13 11:00:00 UTC |\n| 2           | 100        | Task 2    | GenericTask | {}       | RUNNING | 2023-07-13 12:00:00 UTC | 2023-07-13 13:00:00 UTC |\n\n### Job API\nJob API enables grouping multiple tasks together for auditing capabilities at the job level, below is the example of creating a JobApp and running it using EtlFlow.\nBy default, it uses noop audit layer but here we are using JDBC layer to persist auditing information in database.\n```scala\nimport etlflow._\nimport etlflow.audit.Audit\nimport etlflow.task._\nimport zio._\n\nobject MyJobApp extends JobApp {\n\n  private val cred = JDBC(sys.env(\"DB_URL\"), sys.env(\"DB_USER\"), sys.env(\"DB_PWD\"), sys.env(\"DB_DRIVER\"))\n  \n  override val auditLayer: Layer[Throwable, Audit] = etlflow.audit.DB(cred)\n\n  private val task1 = GenericTask(\n    name = \"Task_1\",\n    task = ZIO.logInfo(s\"Hello EtlFlow Task\")\n  )\n\n  def job(args: Chunk[String]): RIO[audit.Audit, Unit] = task1.toZIO\n}\n```\n\nHere's a snapshot of data for the `job_run` and `task_run` table after this job has run:\n\n| job_run_id | job_name | metadata | status  | created_at              | modified_at             |\n|------------|----------|----------|---------|-------------------------|-------------------------|\n| 1          | MyJobApp | {}       | SUCCESS | 2023-07-13 10:00:00 UTC | 2023-07-13 11:00:00 UTC |\n\n\n| task_run_id | job_run_id | task_name | task_type   | metadata | status  | created_at              | modified_at             |\n|-------------|------------|-----------|-------------|----------|---------|-------------------------|-------------------------|\n| 1           | 1          | Task 1    | GenericTask | {}       | SUCCESS | 2023-07-13 10:00:00 UTC | 2023-07-13 11:00:00 UTC |\n\n## GCP\n```shell\n# To run all below GCP examples set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the location of the service account json key. \nexport GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json\n```\n### Dataproc\n```scala\nimport etlflow.task._\nimport gcp4zio.dp._\nimport etlflow.audit\nimport etlflow.audit.Audit\nimport zio._\n\nval gcpProject: String = \"GCP_PROJECT\"\nval gcpRegion: String  = \"GCP_REGION\"\nval dpCluster: String  = \"DP_CLUSTER\"\nval dpEndpoint: String = \"DP_ENDPOINT\"\nval dpBucket: String   = \"DP_BUCKET\"\n\nval createCluster = DPCreateTask(\"DPCreateTask\", dpCluster, ClusterProps(dpBucket))\nval deleteCluster = DPDeleteTask(\"DPDeleteTask\", dpCluster)\n\nval args      = List(\"1000\")\nval mainClass = \"org.apache.spark.examples.SparkPi\"\nval libs      = List(\"file:///usr/lib/spark/examples/jars/spark-examples.jar\")\nval conf      = Map(\"spark.executor.memory\" -\u003e \"1g\", \"spark.driver.memory\" -\u003e \"1g\")\n\nval sparkJob = DPSparkJobTask(\"DPSparkJobTask\", args, mainClass, libs, conf)\n\nval programGCP: RIO[DPJob with DPCluster with Audit, Unit] = for {\n  _ \u003c- createCluster.toZIO\n  _ \u003c- sparkJob.toZIO\n  _ \u003c- deleteCluster.toZIO\n} yield ()\n\nval dpJobLayer = DPJob.live(dpCluster, gcpProject, gcpRegion, dpEndpoint)\nval dpClusterLayer = DPCluster.live(gcpProject, gcpRegion, dpEndpoint)\n\nprogramGCP.provide(dpJobLayer,dpClusterLayer,audit.noop)\n```\nCheck [this](examples/examplegcp/src/main/scala/examples/Job1GCP.scala) for complete example.\n\n## JDBC\n```scala\n// Todo\n```\n\n## K8S\nThis module depends on kubernetes official Java client library version 18.0.1\n```scala\nimport etlflow.task._\nimport etlflow.k8s._\nimport etlflow.audit\nimport etlflow.audit.Audit\nimport zio._\n\nval jobName: String = \"hello\"\n\nval programK8S: RIO[K8S with Audit, Unit] = for {\n  _ \u003c- K8SJobTask(\n    name = \"CreateKubeJobTask\",\n    jobName = jobName,\n    image = \"busybox:1.28\",\n    command = Some(List(\"/bin/sh\", \"-c\", \"sleep 5; ls /etc/key; date; echo Hello from the Kubernetes cluster\"))\n  ).toZIO\n  _ \u003c- K8STrackJobTask(\"TrackKubeJobTask\", jobName).toZIO\n  _ \u003c- K8SJobLogTask(\"GetKubeJobLogTask\", jobName).toZIO\n  _ \u003c- K8SDeleteJobTask(\"DeleteKubeJobTask\", jobName).toZIO\n} yield ()\n\nprogramK8S.provide(K8S.live(),audit.noop)\n```\nCheck [this](examples/examplek8s/src/main/scala/examples/Job1K8S.scala) for complete example.\n## Http\n```scala\n// Todo\n```\n## Email\n```scala\n// Todo\n```\n## AWS\n```scala\n// Todo\n```\n## Redis\n```scala\n// Todo\n```\n## Spark\n```scala\n// Todo\n```\n\n## Contributions\nPlease feel free to add issues to report any bugs or to propose new features.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftharwaninitin%2Fetlflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftharwaninitin%2Fetlflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftharwaninitin%2Fetlflow/lists"}