Projects in Awesome Lists tagged with airflow-dag

https://github.com/san089/goodreads_etl_pipeline

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

airflow airflow-dag apache-airflow apache-spark data-engineering data-engineering-pipeline data-lake data-migration emr-cluster etl-framework etl-job etl-pipeline goodreads-data-pipeline livy python redshift s3 scheduler spark warehouse

Last synced: 16 May 2025

https://github.com/allegro/bigflow

A Python framework for data processing on GCP.

airflow-dag beam bigquery composer dag dataflow dataproc gcp python python-framework workflows

Last synced: 08 Apr 2025

https://github.com/angadsingh/airflow-ditto

An airflow DAG transformation framework

airflow airflow-dag aws azure dataflow emr extensible framework graph-algorithms graph-manipulation hdinsight isomorphism livy networkx spark yarn

Last synced: 23 Apr 2025

https://github.com/nathadriele/airflow-kpi-insertion-pipeline

The script automates the collection and insertion of KPIs related to transaction time and storage usage in a Data Warehouse, using Apache Airflow. It calculates the time elapsed since the last transaction and the percentage of storage usage, recording this data periodically in specific tables.

airflow-dag data-engineering data-ingestion data-kpi-scheduler data-warehouse etl-pipeline kpi-data-pipeline python-script storage-usage transaction-metrics

Last synced: 24 Mar 2025

https://github.com/ignacionmiranda/trigger-dag-run-params-demo

Apache Airflow demo project that setup 3 DAGs to explain how to pass parameters from a DAG to a triggered DAG.

airflow-dag airflow-dags airflow-operator airflow-operators apache-airflow python python3

Last synced: 14 Mar 2025

https://github.com/ckongala/airflow

Apache Airflow

airflow airflow-dag apache-airflow

Last synced: 01 Mar 2025

https://github.com/lepiaf/airflow-training

Small project to play around Apache Airflow and ETL

airflow airflow-dag

Last synced: 29 Mar 2025

https://github.com/adrianmarino/datitos

Datitos - TP2 with steroids

airflow-dag apache-airflow classification deep-learning optuna python pytorch

Last synced: 18 Mar 2025

https://github.com/shegzimus/de_nasa_neow_pipeline

Airflow powered ETL pipeline for moving Near-Earth-Object data from NASA to Google Cloud

airflow-dag airflow-operator airflow-providers bigquery celery-redis docker docker-compose docker-container google-cloud-platform googlecloudstorage nasa-api

Last synced: 22 Mar 2025

https://github.com/wlopezm-unal/reddit_project_airflow_aws

This project focuses on implementing an ETL pipeline using Apache Airflow to efficiently extract data from Reddit, transform it as needed, and load it into an AWS S3 bucket. The use of Airflow allows for robust orchestration of the data workflow, ensuring that each step of the ETL process is executed in a reliable and repeatable manner.

airflow airflow-dag airflow-docker aws docker-compose git python terraform