Ecosyste.ms: Awesome

https://github.com/narius2030/molisa-data-warehouse-integration

Extract data from many databases of Labor, Invalids and Social Affairs sectors and convert to appropriate structure and format, then upload to shared data warehouse and data mart. Thanks to that, people of state agencies can easily retrieve and analyze data based on the compiled data warehouse.

apache-airflow apache-spark api-rest data-pipeline data-warehousing medallion-architecture postgresql

Last synced: 12 Oct 2024

https://github.com/datawithbaraa/sql-modern-warehouse-and-analytics

A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.

data-analysis data-analytics data-cleaning data-engineering data-lake data-lakehouse data-science data-warehouse data-warehousing database datalake datascience datawarehouse datawarehousing etl medallion-architecture pipeline sql sql-query sql-server

Last synced: 23 Dec 2024

https://github.com/ocramz/podium

database lakehouse lakehouse-engine medallion-architecture

Last synced: 05 Dec 2024

https://github.com/jibbs1703/tickit-data-lake

This repository demonstrates the creation of a robust, 3-tier data lake using AWS resources.

aws-glue aws-glue-crawler aws-glue-data-catalog aws-s3 data-lake database etl-pipeline medallion-architecture

Last synced: 25 Nov 2024

https://github.com/narius2030/datalake-solution-imcp

This project involved the development and implementation of a Data Lake architecture to support an AI model capable of generating image captions. The architecture was designed to efficiently ingest, process, and centralized store large volumes of image and text data.

data-lake docker-container etl-pipeline fastapi medallion-architecture mlops nosql-database object-storage

Last synced: 03 Dec 2024

https://github.com/datawithbaraa/sql-data-warehouse-project

A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.

data-analysis data-analytics data-cleaning data-engineering data-lakehouse data-science data-warehouse data-warehousing datalake datascience datawarehouse datawarehousing etl etl-job etl-pipeline medallion-architecture sql sql-query sql-server sqlserver

Last synced: 31 Dec 2024

https://github.com/duyanh711/robustdatapipelineformoderndataengineering

In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our cloud provider.

apache-spark azure databricks dbt medallion-architecture

Last synced: 16 Nov 2024