Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with etl-framework

A curated list of projects in awesome lists tagged with etl-framework .

https://github.com/elastic/logstash

Logstash - transport and process your logs, events, or other data

etl-framework java jruby logging real-time-processing streaming

Last synced: 16 Dec 2025

https://github.com/cloudquery/cloudquery

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

airbyte attack-surface-management aws azure bigquery cspm data data-analysis data-collection data-engineering data-integration elt etl etl-framework gcp github-api go google kubernetes sql

Last synced: 16 May 2026

https://github.com/noflo/noflo

Flow-based programming for JavaScript

etl-framework fbp flow-based-programming nocode noflo

Last synced: 14 May 2025

https://github.com/DAGWorks-Inc/hamilton

Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

dag data-analysis data-engineering data-science dataframe etl etl-framework etl-pipeline feature-engineering hacktoberfest lineage llmops machine-learning mlops orchestration pandas python rag software-engineering

Last synced: 26 Mar 2025

https://github.com/singer-io/getting-started

This repository is a getting started guide to Singer.

data-analysis etl etl-framework python singer

Last synced: 14 May 2025

https://github.com/marsupialtail/quokka

Making data lake work for time series

data-lake-analytics distributed etl-framework mlops sql

Last synced: 07 Oct 2025

https://github.com/stitchfix/hamilton

A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton

dag data-engineering data-platform data-science dataframe etl etl-framework etl-pipeline feature-engineering featurization hamilton hamiltonian machine-learning numpy pandas python software-engineering stitch-fix

Last synced: 29 Sep 2025

https://github.com/cinchoo/choetl

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

avro cinchoo-etl csharp csv dotnet etl etl-framework flat json keyvalue parquet parquet-files parser reader writer xml yaml

Last synced: 12 Apr 2025

https://github.com/Cinchoo/ChoETL

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

avro cinchoo-etl csharp csv dotnet etl etl-framework flat json keyvalue parquet parquet-files parser reader writer xml yaml

Last synced: 14 Mar 2025

https://github.com/apache/seatunnel-web

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

apache data-integration data-pipeline etl-framework high-performance offline real-time seatunnel sql-engine

Last synced: 14 May 2025

https://github.com/yotpoltd/metorikku

A simplified, lightweight ETL Framework based on Apache Spark

big-data distributed-computing etl etl-framework etl-pipeline scala spark sql

Last synced: 06 Apr 2025

https://github.com/seanharr11/etlalchemy

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

database etl etl-framework migrations python sqlalchemy

Last synced: 14 Jan 2026

https://github.com/flow-php/flow

Flow PHP - data processing framework

etl etl-framework etl-pipeline

Last synced: 31 Mar 2025

https://github.com/data-dot-all/dataall

A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.

aws aws-glue aws-lake-formation aws-s3 data data-science etl-framework lakeformation lakehouse redshift

Last synced: 29 Jul 2025

https://github.com/velocitybolt/open-extract

Structured Data Extractor for AI Agents. Search your documents or the web for specific data and get it back in JSON or Markdown in a single tool call.

agent-tools ai autogen context-aware context-aware-structured-outputs crewai etl etl-automation etl-framework langchain langgraph llm openai python rag structured-outputs unstructured-data

Last synced: 03 Apr 2025

https://github.com/frankframework/frankframework

The Frank!Framework is an easy-to-use, stateless integration framework which allows (transactional) messages to be modified and exchanged between different systems.

batch-processing bpmn community-driven data-flow elt-framework erp erp-framework etl-framework framework integration integration-framework integration-platform java low-code low-code-development-platform low-code-platform open-source self-hosted system-integration workflow-engine

Last synced: 23 May 2026

https://github.com/bitwiseinc/hydrograph

A visual ETL development and debugging tool for big data

apache-spark big-data cascading etl etl-framework

Last synced: 24 Oct 2025

https://github.com/BitwiseInc/Hydrograph

A visual ETL development and debugging tool for big data

apache-spark big-data cascading etl etl-framework

Last synced: 03 Apr 2025

https://github.com/halestudio/hale

(Spatial) data harmonisation with hale»studio (formerly HUMBOLDT Alignment Editor)

data-harmonisation database eclipse-rcp etl etl-framework geospatial-data gml groovy hale hale-studio humboldt-alignment-editor inspire java scala transformation xml

Last synced: 13 Apr 2025

https://github.com/restarone/violet_rails

an app engine for your business. Seamlessly implement business logic with a powerful API. Out of the box CMS, blog, forum and email functionality. Developer friendly & easily extendable for your next SaaS/XaaS project. Built with Rails 6, Devise, Sidekiq & PostgreSQL

blog cms ember emberjs etl-automation etl-framework etl-pipeline forum multi-tenancy multitenancy rails ruby ruby-on-rails rubyonrails saas saas-boilerplate template violet-rails wordpress-replacement xaas

Last synced: 04 Jan 2026

https://github.com/contextdata/vectoretl

Build super simple end-to-end data & ETL pipelines for your vector databases and Generative AI applications

cohere data datapipeline etl etl-framework etl-pipeline openai pinecone python qdrant qdrant-vector-database unstructured vector-database weaviate

Last synced: 09 Apr 2025

https://github.com/geopython/stetl

Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.

data-conversion etl etl-framework gis gml inspire osgeo pipeline python streaming-etl transformations

Last synced: 07 Jul 2025

https://github.com/ContextData/VectorETL

Build super simple end-to-end data & ETL pipelines for your vector databases and Generative AI applications

cohere data datapipeline etl etl-framework etl-pipeline openai pinecone python qdrant qdrant-vector-database unstructured vector-database weaviate

Last synced: 22 Sep 2025

https://github.com/maxim2266/csvplus

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

csv csv-format etl etl-framework etl-pipeline fluent-interface go go-csv stream-processing

Last synced: 14 Apr 2025

https://github.com/MassStreetAnalytics/etl-framework

A framework for moving data into a data warehouse.

data-warehouse etl etl-components etl-framework etl-pipeline python sql sqlserver

Last synced: 20 Jul 2025

https://github.com/tharwaninitin/etlflow

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for running complex Auditable workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases, SFTP servers, On-Prem Systems and more.

aws bigquery dataproc etl etl-framework etl-pipeline gcp gcs redis s3 scala spark zio

Last synced: 28 Feb 2025

https://github.com/stellar/stellar-etl

Stellar ETL will enable real-time analytics on the Stellar network

bitcoin blockchain data-analysis ethereum etl-framework etl-pipeline stellar stellar-lumens stellar-network

Last synced: 02 Apr 2026

https://github.com/nhl/link-move

A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.

cayenne etl etl-framework

Last synced: 14 Jan 2026

https://github.com/otiai10/hotsub

Command line tool to run batch jobs concurrently with ETL framework on AWS or other cloud computing resources

aws batch-job bioinformatics cwl cwl-workflow docker docker-machine etl-framework gcp wdl wdl-workflow workflow workflow-engine

Last synced: 16 Oct 2025

https://github.com/cefriel/chimera

Composable Semantic Transformation Pipelines

apache-camel etl-framework mediator pipelines rdf semantic-web

Last synced: 11 Jan 2026

https://github.com/rpsft/etlbox

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

dotnet dotnet-core etl etl-framework

Last synced: 14 Jun 2025

https://github.com/TriplyDB/Documentation

Documentation for the TriplyDB and TriplyETL products

etl-framework etl-pipeline graph-database linked-data production-systems semantic-web

Last synced: 03 Apr 2025

https://github.com/san089/spark_packaged_project

This project contains pyspark jobs to create data pipelines and shows how to distribute the project package on Cluster.

data-pipeline etl etl-framework etl-pipeline job pyspark spark

Last synced: 15 Apr 2025

https://github.com/apache/seatunnel-datasource-sdk

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

apache data-integration data-pipeline etl-framework high-performance offline real-time seatunnel sql-engine

Last synced: 09 Apr 2025

https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics

Building a next-generation hybrid data pipeline architecture that combines the power of Microsoft Fabric, Azure Cloud, and Power BI. This pipeline is engineered to tackle the challenges of real-time data ingestion, multi-layered processing, and analytics, delivering business-critical insights.

azure azure-data-factory azure-fabric bi-analytics big-data-analytics big-data-projects cloud-data-warehouse cloud-dataflow data-analytics data-engineering data-engineering-pipeline data-engineering-project data-pipeline-monitoring data-science data-visualization data-warehouse etl etl-framework etl-pipeline

Last synced: 14 Apr 2026

https://github.com/drsnowbird/openrefine

OpenRefine Docker for Data ETL/ELT

big-data docker etl-framework openrefine

Last synced: 10 Apr 2025

https://github.com/vh-d/Rflow

Rflow is a general-purpose workflow management framework for R

data-processing database dataflow etl etl-framework r reproducibility rlang rstats rstats-package workflow-management

Last synced: 30 Jul 2025

https://github.com/DrSnowbird/openrefine

OpenRefine Docker for Data ETL/ELT

big-data docker etl-framework openrefine

Last synced: 07 Apr 2025

https://github.com/jesufemi-o/hacking-apache-iceberg

Tequila: My journey to apache iceberg. You'll definitely need your pinch of salt! opinions are mine!!

etl-framework iceberg lakehouse-platform

Last synced: 25 Feb 2026

https://github.com/vigneshss-07/google-cloud-professional-data-engineer-acompleteguide

This Repo contains all study, lab and supportive materials for Udemy course on "Google Cloud Professional Data Engineer - A Complete Guide".

big-data bigquery cloud-computing dataengineering elt-pipeline etl-framework gcp-services gcp-storage google-cloud machine-learning

Last synced: 06 Mar 2026

https://github.com/sanjinkurelic/antennadistribution

Antenna Distribution is a project that shows how to run business analysis tools on a set of a data.

business-analytics business-intelligence dwh etl etl-automation etl-framework mssql olap olap-cube powerbi ssas ssis

Last synced: 13 Jul 2025

https://github.com/vh-d/RETL

R package for ETL

etl etl-framework transformations

Last synced: 30 Jul 2025

https://github.com/lumio-medical/lumio-etl

A library providing utilities for building lumio-flow based ETLs

etl etl-framework java workflow

Last synced: 14 Jan 2026

https://github.com/leadtechie/bambooconnect

Bamboo Connect is a lightweight ETL (Extract, Transform, Load) library with examples and templates. It enables developers to quickly extract, transform, reconcile and then load resulting data securely. This avoids time consuming manual error prone tasks.

etl etl-framework etl-pipeline pandas python

Last synced: 25 Jan 2026

https://github.com/knitli/recoco

CocoIndex, but usable from Rust. Modular, feature-gated, edge-deployable. 220 crates instead of 820.

cocoindex dataflow etl-framework rust

Last synced: 01 Apr 2026

https://github.com/dadananjesha/azuredataengine

AzureDataEngine is a robust, scalable batch processing data architecture built on the Azure platform. It efficiently extracts, transforms, and loads massive datasets for machine learning applications, leveraging Azure Blob Storage, PostgreSQL, Databricks, and Key Vault to ensure reliability and maintainability.

azure batch-processing blob-storage databricks etl etl-framework key-vault postgresql-database spark vnet

Last synced: 15 Apr 2026

https://github.com/jesufemi-o/grokking-dlt-kafka

repo showing example of how to go from proof of concept to prod-ready setup with dlt for kafka sources

batch-processing dlthub etl-framework kafka

Last synced: 17 Aug 2025

https://github.com/goldstrike77/ansible-role-linux-logstash

Ansible role for install and configure Logstash on Linux operating system.

ansible ansible-role elk etl-framework logging logstash real-time-processing streaming

Last synced: 10 May 2026

https://github.com/chubbard/gratum

A simplified ETL engine for groovy.

etl etl-framework etl-pipeline groovy

Last synced: 11 Jun 2025

https://github.com/thomasms/sss

Stupidly Simple Storage with python decorators

dataframe decorators etl etl-framework storage

Last synced: 29 Mar 2025

https://github.com/neelanjan-chakraborty/operateoptima

Operate Optima: High-speed Apache Spark ETL pipeline that cuts processing time by 25 min, adopted by 3 business units, featured on Woke AI Blog. Includes clean modular architecture, benchmarks, and deployment-ready code.

apache-spark databricks etl etl-automation etl-framework etl-pipeline pyspark python

Last synced: 17 Apr 2026

https://github.com/anyparser/anyparser_core

Anyparser Python SDK for RAG/ETL Pipelines - File Content Extraction. Supports extraction from various file formats including PDF, Microsoft Office documents, OCR/Image to Text, Audio to Text, and Website to Text.

cache-augmented-generation crawler crewai etl-framework etl-pipeline knowledge-graph knowledgebase langchain langgraph llamaindex ms-office n8n ocr openai pdf python rag retrieval-augmented-generation search-engine typescript

Last synced: 05 Oct 2025

https://github.com/eDatos/coetl

ETL Console is the Canary Islands Government's corporate solution for registering and executing ETL files developed using Pentaho Data Integration technology

console-application data etl etl-framework etl-pipeline java official-statistics open-data pentaho-data pentaho-data-integration pentaho-kettle pentaho-server statistics web

Last synced: 14 Jul 2026

https://github.com/ianturton/shetland

Shetland: A python DSL to handle ETL with OGR

etl etl-framework geospatial ogr2ogr python

Last synced: 25 Mar 2025

https://github.com/amitnema/spark-etl-framework

A generic, multimodule framework for building scalable ETL (Extract, Transform, Load) processes using Apache Spark and Java. This framework is designed to be cloud-agnostic, maintainable, testable, and near-production-ready.

etl etl-framework etl-pipeline general-purpose generalization java-11 maven spark-3x spark-submit

Last synced: 18 May 2026

https://github.com/datacoolie/datacoolie

Metadata-driven ETL framework for portable data pipelines across Polars, Spark, Fabric, Databricks, and AWS.

aws data-engineering databricks delta-lake etl etl-framework iceberg lakehouse metadata-driven microsoft-fabric pipeline polars pyspark spark

Last synced: 28 Apr 2026

https://github.com/jbn/vaquero

A Python library for iterative and interactive data wrangling at laptop-scale.

data data-analysis data-cleaning data-mining dirty-data elt etl etl-framework

Last synced: 10 Jun 2026

https://github.com/vim89/flowforge

Let's be honest - most data pipeline frameworks treat types as suggestions. Config files are strings. Schemas are "validated" at runtime. Data quality is an afterthought. So, let's do differently

archetype data data-contracts data-engineering data-pipelines data-quality data-science database dataengineering datapipeline etl etl-framework pipelines scala scalability spark spark-sql spark-streaming

Last synced: 14 Apr 2026

https://github.com/kanchishimono/psetl

Simple and extensible PySpark ETL framework

apache-spark etl etl-framework etl-pipeline pyspark spark

Last synced: 17 May 2026

https://github.com/bala-1409/sql-projects

The repository contains Structured Query Language (SQL) Scripts. The Multiple SQL scripts for various projects which includes data cleaning, data pre-processing, data processing, data transformation and insights gaining through Query Language.

data-analysis data-mining data-science data-transformation database eda etl-framework exploratory-data-analysis microsoft-sql-server query-language sql sql-server sql-server-database sql-server-management-studio

Last synced: 27 Feb 2026

https://github.com/giacomolat/etl-workflow

This project uses Pentaho Data Integration to build a data warehouse (DWH). It includes the sales fact table and two dimensional tables (dim_product and dim_payment). The ETL process implements incremental delta to optimize the extraction and loading of new data.

database dataengineering dwh etl etl-framework etl-pipeline pentaho pentaho-data-integration pgadmin4 postgres postgresql postgresql-database sql

Last synced: 16 Apr 2026

https://github.com/shridhar1504/sql-projects

The repository contains Structured Query Language (SQL) Scripts. The Multiple SQL scripts for various projects which includes data cleaning, data pre-processing, data processing, data transformation and insights gaining through Query Language.

data-analysis data-mining data-science data-transformation eda etl-framework microsoft-sql-server query-language sql sql-server sql-server-management-studio sqlqueries

Last synced: 09 Mar 2026

https://github.com/harish876/forge

Opiniated Framework to write ETL Pipelines controlled by a central config store.

etl-framework python

Last synced: 19 May 2026

https://github.com/imjuliengaupin/bourne

A Python-based framework for modular, AI-ready data pipeline orchestration featuring plug-and-play agents, real-time terminal dashboards, and Pydantic-powered schema validation.

agentic-ai agentic-workflow agents ai etl-framework github-actions orchestration-framework pydantic-v2 python3

Last synced: 18 Apr 2026