Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with data-lake

A curated list of projects in awesome lists tagged with data-lake .

https://github.com/dlt-hub/dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

data data-engineering data-lake data-loading data-warehouse elt extract load python transform

Last synced: 01 Apr 2026

https://github.com/apache/kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

data-lake hacktoberfest hadoop hive jdbc kubernetes spark spark-sql sql thrift

Last synced: 13 May 2025

https://github.com/bytedance/bitsail

BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

big-data data-integration data-lake data-pipeline data-synchronization flink high-performance real-time

Last synced: 15 May 2025

https://github.com/teradata/kylo

Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

data-lake hadoop kylo nifi spark teradata

Last synced: 15 May 2025

https://github.com/Teradata/kylo

Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

data-lake hadoop kylo nifi spark teradata

Last synced: 06 Apr 2025

https://github.com/pixelsdb/pixels

An efficient storage and compute engine for both on-prem and cloud-native data analytics.

cloud-database column-store data-lake data-warehouse database olap

Last synced: 03 May 2026

https://github.com/canner/wren-engine

🤖 The Semantic Engine for Model Context Protocol(MCP) Clients and AI Agents 🔥

agent agentic-ai ai business-intelligence data data-analysis data-analytics data-lake data-warehouse hacktoberfest llm mcp mcp-server semantic semantic-layer sql

Last synced: 10 Mar 2026

https://github.com/uber/marmaray

Generic Data Ingestion & Dispersal Library for Hadoop

avro-schema data-lake hadoop ingest-data schema-format spark

Last synced: 23 Mar 2025

https://github.com/gigapi/gigapi

GigAPI is a Timeseries lakehouse for real-time data and sub-second queries, powered by DuckDB OLAP + Parquet Query Engine, Compactor w/ Cloud-Native Storage. Drop-in FDAP alternative ⭐

api clickhouse-server data-lake database datalake duckdb duckdb-api duckdb-server ducklake fdap gigapipe golang lakehouse olap parquet qryn query-engine rest-api s3 sql

Last synced: 05 Oct 2025

https://github.com/awslabs/amazon-s3-find-and-forget

Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)

amazon-s3 aws big-data ccpa data data-erasure data-lake gdpr parquet privacy right-to-be-forgotten s3

Last synced: 04 Apr 2025

https://github.com/maxi-k/btrblocks

BtrBlocks: Efficient Columnar Compression for Data Lakes (SIGMOD 2023 Paper)

compression data-lake databases research

Last synced: 09 Apr 2025

https://github.com/azure/usql

U-SQL Examples and Issue Tracking

azure big-data data-lake u-sql

Last synced: 12 Apr 2025

https://github.com/garystafford/tickit-data-lake-demo

Resources for video demonstrations and blog posts related to DataOps on AWS

airflow aws data-lake dataops devops redshift

Last synced: 07 May 2025

https://github.com/azure/azuredatalake

Samples and Docs for Azure Data Lake Store and Analytics

azure big-data data-lake

Last synced: 09 Apr 2025

https://github.com/Canner/wren-engine

🤖 The semantic engine for LLMs, bringing semantic context to AI agents. 🔥

business-intelligence data data-analysis data-analytics data-lake data-warehouse hacktoberfest llm semantic semantic-layer sql

Last synced: 01 Apr 2025

https://github.com/smart-data-lake/smart-data-lake

Smart Automation Tool for building modern Data Lakes and Data Pipelines

data-lake data-pipelines deltalake hadoop hive scala smart-data-lake spark transform-data

Last synced: 13 Apr 2025

https://github.com/dominikhei/Local-Data-LakeHouse

Sample Data Lakehouse deployed in Docker containers using Apache Iceberg, Minio, Trino and a Hive Metastore. Can be used for local testing.

apache-iceberg data-lake data-lakehouse hive-metastore lakehouse minio trino

Last synced: 07 May 2026

https://github.com/camunda-community-hub/zeeqs

GraphQL API for Zeebe data

data-lake graphql zeebe zeebe-tool

Last synced: 04 Oct 2025

https://github.com/OElesin/querypal

Web UI for Amazon Athena

analytics aws aws-athena data data-lake sql

Last synced: 30 Jul 2025

https://github.com/matsmoll/aligned

The DBT of ML, as Aligned describes data dependencies in ML systems, and reduce technical data debt

ai data-contracts data-lake datacontracts dbt feature-engineering feature-store ml ml-ops mlops

Last synced: 13 Aug 2025

https://github.com/realdatadriven/etlx

ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.

data-engineering data-lake data-lakehouse data-quality data-quality-checks data-quality-monitoring data-science duckdb elt elt-pipeline etl etl-elt-pipelines etl-pipeline object-storage relational-databases report report-automation s3 s3-storage

Last synced: 30 Apr 2026

https://github.com/imsanjoykb/python-mysql-operation

This Python MySQL Repo shows you how to use MySQL Connector Python to access MySQL databases. You will learn how to connect to MySQL database and perform common database operations such as SELECT, INSERT, UPDATE, & DELETE in Python.

data-lake database database-operation database-programming mysql-server python-mysql python-mysql-connector

Last synced: 20 Jun 2025

https://github.com/ec-europa/eubfr-data-lake

EU Budget for Results - Data Lake

data-lake

Last synced: 29 Apr 2025

https://github.com/sap-samples/hana-cloud-relational-data-lake-onboarding

This is an end-to-end onboarding sample for SAP HANA Cloud, relational data lake. It shows how to create schema, load data, and execute queries.

data-lake hana-cloud sample sap-hana

Last synced: 29 Jan 2026

https://github.com/apache/kyuubi-docker

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

data-lake hadoop hive jdbc kubernetes spark spark-sql sql thrift

Last synced: 19 Oct 2025

https://github.com/wherobots/havasu

The spatial table format for spatial lakehouse

apache-sedona data-lake geoparquet havasu iceberg raster

Last synced: 05 Feb 2026

https://github.com/AuFeld/Data_Engineering_Projects

A collection of data engineering projects: data modeling, ETL pipelines, data lakes, infrastructure configuration on AWS, data warehousing, containerization, and a dashboard to monitor data pipeline KPIs

airflow aws cassandra data-engineering data-lake data-warehouse docker emr etl-pipeline infrastructure-as-code infrastructure-setup postgresql python redshift s3 spark

Last synced: 29 Jul 2025

https://github.com/treeverse/lakefs-hooks

a simple lakeFS webhook for pre-commit and pre-merge validation of data objects

data-engineering data-lake lakefs

Last synced: 23 Oct 2025

https://github.com/bmsuisse/lakeapi

API for distributing Data Lake Data

api columnar data-lake deltalake duckdb fastapi lakehouse polars rest-api sql

Last synced: 18 Feb 2026

https://github.com/maris-development/beacon

A high-performance climate data lake supporting subsetting for zarr, netcdf, parquet, arrow ipc, csv and bbf

blue-cloud2026 data-access data-lake docker open-science rest-api

Last synced: 29 Jun 2026

https://github.com/jlsilva01/adls-azure

Procedimento para criação de um Azure Data Lake Storage usando Terraform, através de uma assinatura MS Learn Sandbox

adlsgen2 azure azurecli data-lake terraform

Last synced: 24 Jun 2025

https://github.com/txn2/mcp-data-platform

A semantic data platform MCP server that composes multiple data tools with bidirectional cross-injection - tool responses automatically include critical context from other services.

data-analysis data-lake data-warehouse golang golang-library mcp mcp-server

Last synced: 31 May 2026

https://github.com/ibm-cloud/nodejs-data-lake-dashboard

Sample and tutorial that creates interactive dashboards using: Dynamic Dashboard Embedded, Cloud Object Storage, SQL Query, DB2 Warehouse and AppID.

cloud data-lake db2 db2-warehouse ibm-cloud ibm-cloud-solutions tutorial

Last synced: 05 Mar 2026

https://github.com/datasphere-oss/datasphere

DataSphere is the first open-source cloud-native data observability platform that helps you trace the whole data infrastructure in your warehouses, lakes and databases.

cloud-native daas data-analytics data-governance data-lake data-management data-observability datamesh datasphere warehouse

Last synced: 14 Jul 2025

https://github.com/DataDrivenGit/Music-Streaming-App-using-AWS-ETL

Implemented Data Warehouse, Data Lake on AWS and Data modeling with Postgres and Apache Cassandra, Also used Apache Airflow to create data pipeline

airflow-operators cassandra data-lake data-pipelines datawarehouse postgres python3 sql

Last synced: 20 Jul 2025

https://github.com/codewell/data-kale

The Simple Data Lake - Data Kale

data data-lake python

Last synced: 25 May 2026

https://github.com/cds-snc/data-lake

Infrastructure for the Platform Data Lake

aws data-lake terraform

Last synced: 12 Feb 2026

https://github.com/shrikantnaidu/data-lakehouse-with-delta-lake

Setting up a data lakehouse with delta lake using docker

data-lake delta-lake docker-compose pyspark

Last synced: 07 Feb 2026

https://github.com/bahbosque/delta-to-iceberg-aws-glue

Tool to migrate Delta Lake tables to Apache Iceberg using AWS Glue and S3

apache-iceberg aws aws-glue-data-catalog data-lake delta-lake migration-tool open-source spark

Last synced: 03 Jul 2025

https://github.com/pirate-emperor/bigdata-pipeline

BigData Pipeline is a local testing environment for experimenting with various storage solutions (RDB, HDFS), query engines (Trino), schedulers (Airflow), and ETL/ELT tools (DBT). It supports MySQL, Hadoop, Hive, Kudu, and more.

airflow airflow-dags airflow-docker big-data data-lake data-lakestore data-warehouse dbt dbt-core distributed-computing docker docker-compose hadoop hive hiveql kudu mysql mysql-server trino trino-cli

Last synced: 08 Feb 2026

https://github.com/san089/yelp_project

This project is to create a Data lake for Yelp data-set and further using the it to create an Analytical Sandbox Data Science purpose and also creating a data warehouse for reporting purpose.

data-lake data-pipeline etl etl-pipeline ingestion load pyspark recommender-system redshift

Last synced: 21 Apr 2026

https://github.com/santiagortiiz/snowflake-data-pipelines

EPAM's Snowflake hands-on lab. We built a pipeline to read and load data from S3 into Snowflake, developed an ETL workflow to clean the data and stored it in a data warehouse with the 3NF and Star schemas for data mart analysis.

business-intelligence data-lake data-pipelines data-warehouse etl snowflake streams

Last synced: 26 Jun 2025

https://github.com/supakunz/book-revenue-pipeline

A ready-to-use Docker-based template for data engineering projects, featuring a complete stack with Apache Airflow, Spark, and MinIO for building scalable data pipelines.

apache-airflow apache-spark data-engineering data-lake delta-lake docker docker-compose etl-pipeline minio pyspark python s3 template-boilerplate

Last synced: 05 Apr 2026

https://github.com/narius2030/imcp-support-blinders

This project focuses on image captioning by creating two primary models: DarkNetLM and DarkNetVG2. Both models leverage the CSP DarkNet53 architecture as the backbone of YOLOv8 for feature extraction from images. Combining with Transformers or LSTM to generating captions.

computer-vision data-lake image-captioning large-language-model mobile-app

Last synced: 30 Jun 2026

https://github.com/mikeacosta/cowculate

Herd management ETL and analytics processing

apache-spark aws cloudformation data-lake etl kinesis-stream redshift s3

Last synced: 20 Apr 2026

https://github.com/anuveyatsu/cloudflare-data-fabric

Cloudflare Data Fabric: Use Cloudflare's global infrastructure to build a flexible, resilient framework for data solutions.

cloudflare data data-lake fabric lakehouse mesh

Last synced: 29 Jun 2026

https://github.com/banknatchapol/sparkify-data-lake

Build an ETL pipeline for a data lake hosted on AWS S3.

aws data-lake

Last synced: 12 Apr 2026

https://github.com/toch/oasis

A DIY data lake 🌴

data-lake oasis

Last synced: 07 Mar 2026

https://github.com/najuzilu/dl-spark

Building a Data Lake with Spark

aws-emr aws-s3 data-engineering data-lake etl-pipeline spark

Last synced: 03 May 2026

https://github.com/narius2030/datalake-solution-imcp

This project involved the development and implementation of a Data Lake architecture to support an AI model capable of generating image captions. The architecture was designed to efficiently ingest, process, and centralized store large volumes of image and text data.

data-lake docker-container etl-pipeline fastapi medallion-architecture mlops nosql-database object-storage

Last synced: 15 Apr 2026

https://github.com/vermicida/data-lake

Data Lake, the code corresponding the project #4 of the Udacity's Data Engineer Nanodegree Program

aws-s3 data-engineering data-lake etl-pipeline python spark

Last synced: 18 Feb 2026

https://github.com/morgan-sell/usa-tourism-etl

Coalesced and transformed various data sources to create a comprehensive data lake for the USA tourism sector.

aws data-engineering data-lake emr-cluster etl-pipeline python spark

Last synced: 16 Apr 2026

https://github.com/stevehoober254/dataengineer-portfolio

📊 End-to-end ETL pipelines, Airflow DAGs, notebook-driven analytics & data warehousing

airflow analytics big-data dagster data-engineering data-lake data-pipelines etl python spark

Last synced: 18 Apr 2026

https://github.com/dataspoclab/dataspoc-lens

Virtual warehouse — SQL + Jupyter + AI over cloud Parquet via DuckDB

cli data data-engineering data-lake duckdb etl parquet python singer sql

Last synced: 20 Apr 2026

https://github.com/jaimealruiz/proyecto-tfg

Diseño e Implementación de interconexión entre LLM y Espacios de Datos

data-lake mcp mcp-client mcp-server rag

Last synced: 08 Sep 2025

https://github.com/josecsotomorales/pyspark-datalake

Data Lake with PySpark and AWS S3

data-lake pyspark python spark

Last synced: 27 Mar 2025

https://github.com/jibbs1703/tickit-data-lake

This repository demonstrates the creation of a robust, 3-tier data lake using an Orchestrator and AWS resources. It collects data from an on-premises NoSQL database and loads it into a SQL database in AWS.

aws-glue aws-glue-crawler aws-glue-data-catalog aws-redshift aws-s3 boto3 data-lake database etl-pipeline medallion-architecture mongodb precommit-hooks

Last synced: 20 Mar 2025

https://github.com/gares95/datalake-spark

This repository consist of a project to build an ETL pipeline for a data lake hosted on S3 using Spark. This project is based on Udacity's template.

data-lake datalake udacity udacity-data-engineer-nanodegree

Last synced: 11 Jan 2026

https://github.com/renatoelho/data-lake

o Data Lake é um conceito/arquitetura de armazenamento de dados em vez de uma ferramenta específica. É um repositório centralizado que permite armazenar dados brutos ou semiestruturados de diversas fontes, como bancos de dados, logs, dispositivos IoT e muito mais.

big-data data-engineering data-lake docker docker-compose duckdb jupyter-notebook minio python s3

Last synced: 10 Apr 2026

https://github.com/aymendaoudi/evcharging

Simulation, Ingestion and ETL-ing data of millions of EV charging sessions by thousands of EVs in thousands of stations around the world.

batch-processing data-engineering data-lake data-warehouse kafka kafka-connect lake-fs minio mongodb postgresql python3 simpy spark stream-processing

Last synced: 12 Mar 2025

https://github.com/dominique-jacque/nba-data-lake

NBA Data Lake Repository contains the setup_nba_data_lake.py script, which automates the creation of a data lake for NBA analytics using AWS services. The script integrates Amazon S3, AWS Glue, and Amazon Athena, and sets up the infrastructure needed to store and query NBA-related data.

amazon-athena api aws-glue cloudshell data-lake iam s3

Last synced: 07 Mar 2026

https://github.com/daniel-jcvv/daniel-jcvv

👨‍💻 Data Engineer | 3+ years enterprise experience with Telcel & Citi Banamex Develop ETL pipelines, data governance, and cloud solutions. Building scalable data architectures and automated workflows for Fortune 500 clients. Tech Stack: Python, SQL Server, Oracle, Apache Airflow, PySpark

agentic-ai apache-airflow apache-kafka apache-spark automation business-intelligence citi-bank-apis data-analysis data-engineering data-lake data-warehouse etl-pipeline medallion-architecture mlops n8n-workflow python rag sql-server

Last synced: 15 Apr 2026

https://github.com/manuel-lang/data-lake-with-spark

Project Data Lake as part of Udacity's Data Engineering Nanodegree

data-engineering data-lake etl-pipeline s3 spark udacity udacity-data-engineer-nanodegree

Last synced: 13 May 2026

https://github.com/paznera/php-minio-obj-store

cli (Laravel prompts) example usage of btrfs-like storage behind s3 interface (minio) and in-memory(redis) indexing with object metadata

btrfs buckets data-lake laravel minio php redis s3 snapshots

Last synced: 10 Apr 2026

https://github.com/julientoucoula17/journeytosnowflakeexpert

Learning and practice in order to become an expert.

data-applications data-engineering data-lake data-warehouse sql

Last synced: 19 Feb 2026