Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with dags

A curated list of projects in awesome lists tagged with dags .

https://github.com/apache/burr

Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.

ai burr chatbot-framework dags generative-ai graphs hacktoberfest llmops llms mlops persistent-data-structure state-machine state-management visibility

Last synced: 08 Jul 2025

https://github.com/astronomer/dag-factory

Construct Apache Airflow DAGs Declaratively via YAML configuration files

airflow apache-airflow dags python

Last synced: 13 May 2025

https://github.com/DAGWorks-Inc/burr

Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.

ai burr chatbot-framework dags generative-ai graphs hacktoberfest llmops llms mlops persistent-data-structure state-machine state-management visibility

Last synced: 16 Apr 2025

https://github.com/astronomer/astro-sdk

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

airflow apache-airflow bigquery dags data-analysis data-science elt etl gcs pandas postgres python s3 snowflake sql sqlite workflows

Last synced: 13 Apr 2025

https://github.com/kaxil/airflowctl

A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects

airflow apache-airflow cli dags python

Last synced: 08 Apr 2025

https://github.com/flyaflya/causact

causact: R package to accelerate computational Bayesian inference workflows in R through interactive visualization of models and their output.

bayesian-inference dags posterior-probability probabilistic-graphical-models probabilistic-programming rstats-package

Last synced: 23 Feb 2026

https://github.com/integrated-inferences/causalqueries

Bayesian inference from binary causal models

bayes causal dags mixedmethods stan

Last synced: 13 Feb 2026

https://github.com/integrated-inferences/CausalQueries

Bayesian inference from binary causal models

bayes causal dags mixedmethods stan

Last synced: 26 Apr 2025

https://github.com/alvarocavalcante/airflow-parse-bench

Stop creating bad DAGs! Use this tool to measure and compare the parse time of your DAGs, identify bottlenecks, and optimize your Airflow environment for better performance.

airflow apache-airflow dags data-engineering python python3

Last synced: 11 Sep 2025

https://github.com/astronomer/astro-provider-ray

This provider contains operators, decorators and triggers to send a ray job from an airflow task

apache-airflow dags mlops-workflow mlpipelines workflow

Last synced: 06 Jul 2025

https://github.com/andrewheiss/closing-backdoors-dags

Use regression, inverse probability weighting, and matching to close confounding backdoors and find causation in observational data

causal-inference dags do-calculus inverse-probability-weights matching

Last synced: 21 Jan 2026

https://github.com/badal-io/gcp-airflow-foundations

Opinionated framework based on Airflow 2.0 for building pipelines to ingest data into a BigQuery data warehouse

airflow apache-airflow bigquery dags data-engineering data-pipeline etl-pipeline

Last synced: 24 Mar 2025

https://github.com/thomwright/balamb

🌱 Concurrently run a set of dependent, asynchronous tasks with type-safe dependencies

concurrent dag dags data data-seeding dependency-injection di seed seeding tasks

Last synced: 07 May 2025

https://github.com/polidea/dag-checks

The dag-checks consist of checks that can help you in maintaining your Apache Airflow instance.

airflow ci-cd dags debugging maintenance

Last synced: 01 May 2025

https://github.com/kaushalprajapatikp/mlops_data_science_pipeline

A modular MLOps pipeline for end-to-end machine learning: automated data validation, feature engineering, multi-model training, MLflow/DagsHub tracking, and FastAPI deployment. Containerized, extensible, and production-ready for real-world projects. Explore code, docs, and notebooks for reproducible ML workflows.

dags machine-learning mlflow mlops mlops-workflow pipeline python3

Last synced: 18 Jul 2026

https://github.com/tuancamtbtx/airflow-on-k8s

Setup Airflow and Implement Some Operators Deployed on K8s

airflow dags k8s python3

Last synced: 21 Jan 2026

https://github.com/tuanai-vireox/airflow-on-k8s

Setup Airflow and Implement Some Operators Deployed on K8s

airflow dags k8s python3

Last synced: 20 Apr 2026

https://github.com/ychaaby/integration-and-analysis-of-crypto-data

Integration and Analysis of Crypto Data with Apache Airflow, Tableau, Snowflake, AWS S3/Lambda

airflow api aws aws-s3 dags etl pipeline

Last synced: 14 Apr 2026

https://github.com/priyanshubiswas-tech/airflow_dbt_superset_project

End-to-end ITSM data engineering pipeline using PostgreSQL, DBT, Airflow, and Superset. Covers ingestion, cleaning, transformation, orchestration, and visualization, validated across Docker Toolbox and Docker Desktop environments.

apache-airflow apache-superset dags data-analysis dbt docker etl etl-automation etl-pipeline postgresql

Last synced: 07 May 2025

https://github.com/ahmedfarag9/airbyte-airflow-api-intergration

Api integration using Airbyte/Airflow Mega Docker Cluster - Airbyte handles the Api connection & Airflow does the orchestration.

airbyte airflow api aws-ec2 celery dags docker docker-compose linux makefile python redis shell-script ssh

Last synced: 14 Apr 2026

https://github.com/goamegah/flowstate

End-To-End Real-time Road Traffic Monitoring Spark Structured Streaming solution

airflow aws aws-kinesis dags kafka postgresql realtime scala spark-core spark-sql spark-streaming streaming

Last synced: 06 May 2026

https://github.com/seilylook/airflow-introduction

Airflow Introduction Practice

airflow dags docker postgresql

Last synced: 10 Apr 2026

https://github.com/shiningflash/apache-airflow-workflow-manager

Simple Apache Airflow setup with both standalone and Docker-Compose workflows for quick orchestration experiments.

airflow airflow-dags apache-airflow dags docker python workflow

Last synced: 17 May 2026

https://github.com/fistgang/airflow-example

An introduction to Airflow, setting up a local environment and writing my first DAG (lazy)

airflow dags

Last synced: 06 Nov 2025

https://github.com/abhijeetdasbakshi/ecommerce-insights

A Dockerized end-to-end project that combines unsupervised machine learning for customer segmentation with scalable data pipelines. It uses MongoDB for data ingestion, Scikit-learn for clustering, Airflow for orchestration, and Streamlit for interactive visualization — enabling actionable insights into e-commerce

airflow airflow-dags ci-cd-pipeline clustering dags data data-pipelines docker docker-compose docker-container dockerfile git great-expectations kafka mongodb pca-analysis postgresql pyspark t-sne umap-learn

Last synced: 04 Apr 2026

https://github.com/vaxdata22/cities-weather-s3-snowflake-slack-notif-etl-by-airflow-on-ec2

This is my second industry-level ETL project. This data pipeline orchestration uses Apache Airflow on AWS EC2. It demonstrates how to build an ETL data pipeline that would extract data (JSON) from the OpenWeatherMap API, transform it, dump it as CSV in S3 bucket, then copy it to destination tables in Snowflake DW and send Slack notification.

apache-airflow aws-ec2 business-intelligence dags data-warehousing etl-pipeline openweathermap-api orchestration python3 slack-webhook snowflake sql

Last synced: 26 May 2026

https://github.com/nineinfra/nineinfra-dags

A collection of the airflow dags for the NineInfra Cluster

airflow dags

Last synced: 05 Sep 2025

https://github.com/vaxdata22/customer-churn-data-analytics-etl-pipeline-by-airflow-on-ec2

This is an end-to-end AWS Cloud ETL project. This orchestration uses Apache Airflow on AWS EC2 as well as AWS Glue. It demonstrates how to build ETL pipeline that would perform data transform using Glue job/crawler as well as loading into a Redshift table. It also shows how to connect Amazon Athena to Glue Data Catalog, and Power BI to Redshift.

amazon-athena amazon-redshift apache-airflow aws-ec2 aws-glue aws-s3 business-intelligence customer-churn-analytics dags data-visualization etl-pipeline orchestration power-bi python3

Last synced: 18 Jun 2025

https://github.com/vaxdata22/zillow-rapid-api-end-to-end-etl-data-pipeline-by-airflow-on-ec2

This is an end-to-end AWS Cloud ETL project. This data pipeline orchestration uses Apache Airflow on AWS EC2 as well as AWS Lambda. It demonstrates how to build ETL data pipeline that would perform data transformation using Lambda function as well as loading into a Redshift cluster table. The data would then be visualized using Amazon QuickSight.

amazon-quicksight amazon-redshift apache-airflow aws-ec2 aws-lambda aws-s3 business-intelligence dags data-visualization etl-pipeline orchestration python3 rapid-api zillow-house-listings

Last synced: 19 May 2026

https://github.com/alvarocavalcante/airflow-custom-deferrable-dataflow-operator

Start your Dataflow jobs execution directly from the Triggerer without going to the Worker!

airflow airflow-dags airflow-operators dags data-engineering dataflow gcp python python3

Last synced: 11 Sep 2025

https://github.com/felipemchdev/data-dbt-batch

End-to-end batch pipeline using Olist data with Bronze/Silver/Gold layers,

airflow dags data-engineering dbt docker duckdb great-expectations pipeline

Last synced: 11 Jun 2026

https://github.com/markphamm/learn-airflow

A hands-on repository for learning Apache Airflow, covering DAG creation, scheduling, task dependencies, and XComs. Includes practical examples, best practices, and real-world use cases to master workflow automation. 🚀

airflow dags dockers

Last synced: 12 Apr 2025

https://github.com/stateset/stateset-airflow-dags

Stateset DAGS for Ops and Scheduled Workflows

dags

Last synced: 22 Jul 2026

https://github.com/wcharczuk/temporalflow

A workflow to process graphs of activities in parallel.

airflow-dags dags graphs incremental temporal

Last synced: 12 Apr 2025

https://github.com/vaxdata22/city-weather-and-s3file-rds-s3-bigquery-etl-by-airflow-on-ec2

This is my third AWS Cloud ETL project. This data pipeline orchestration uses Apache Airflow on AWS EC2. It demonstrates how to build an ETL data pipeline that would perform data extraction to a database in parallel to a loading process into the same database, join the tables, copy joined data to S3 and finally copy the S3 file to BigQuery DW.

apache-airflow aws-ec2 aws-rds-postgres aws-s3 bigquery business-intelligence dags data-warehousing etl-pipeline openweathermap-api orchestration python3 sql

Last synced: 21 May 2026

https://github.com/vaxdata22/redfin-analytics-etl-data-engg-pipeline-by-airflow-on-ec2

This is an end-to-end AWS Cloud ETL project. This data pipeline orchestration uses Apache Airflow on AWS EC2 as well as Snowpipe. It demonstrates how to build ETL data pipeline that would perform data transformation using Python on Apache Airflow as well as automatic ingestion into Snowflake data warehouse via Snowpipe. Also features Power BI.

apache-airflow aws-ec2 aws-s3 business-intelligence dags data-visualization etl-pipeline orchestration power-bi python3 redfin snowflake snowpipe sqs-queue

Last synced: 29 Apr 2026

https://github.com/dmarks84/coursework_capstone_full_data_engineering

Final Project for IBM Data Engineering & Python Professional Certificate -- Applied all skills and methods utilized in the series of courses for this certification

apache-airflow apache-hadoop apache-kafka apache-spark api beautifulsoup cassandra dags etl mongodb nosql pandas plotly postgresql python scipy seaborn sql

Last synced: 25 Feb 2026

https://github.com/vaxdata22/city-weather-and-s3file-rds-s3-bigquery-by-airflow-on-ec2

This is my third industry-level ETL project. This data pipeline orchestration uses Apache Airflow on AWS EC2. It demonstrates how to build an ETL data pipeline that would perform data extraction to a database in parallel to a loading process into the same database, join the tables, copy joined data to S3 and finally copy the S3 file to BigQuery DW.

apache-airflow aws-ec2 aws-rds-postgres aws-s3 bigquery business-intelligence dags data-warehousing etl-pipeline openweathermap-api orchestration python3 sql

Last synced: 18 Mar 2025

https://github.com/danhenriquex/kubernetes-for-data-engineering

Data Engineering with Kubernetes and Airflow

airflow dags kubernetes python

Last synced: 20 Jan 2026

https://github.com/dpbm/qcop

An AI model to predict the output of a quantum cirucit

airflow apache-airflow compose dags docker docker-compose make pillow python pytorch qiskit qiskit-aer quantum-computing

Last synced: 29 Apr 2026

https://github.com/yevheniidatsenko/goit-de-hw-07

❇️ Hometask - Data Engineering (Apache Airflow)

airflow dags docker goit-de-hw-07 mysql

Last synced: 01 May 2026

https://github.com/vaxdata22/redfin-analytics-etl-using-amazon-emr-by-airflow-on-ec2

This is an end-to-end AWS Cloud ETL project. This data pipeline uses an Amazon EMR cluster managed by Apache Airflow that is running on an AWS EC2 instance. It demonstrates how to build orchestration that would perform data transformation using Amazon EMR as well as automatic data ingestion into a Snowflake via Snowpipe. It also features Power BI.

amazon-emr-cluster apache-airflow apache-spark aws-ec2 aws-s3 business-intelligence dags data-visualization etl-pipeline google-colab-notebook orchestration power-bi pyspark redfin snowflake snowpipe sqs-queue

Last synced: 02 May 2026

https://github.com/an4pdm/etl-process-notifier

Projeto de automação de ETL com notificadores para sucesso ou falha no processamento.

airflow dados dags data-engineering database engenharia-de-dados etl etl-automation etl-pipeline pandas pipeline python

Last synced: 06 May 2026

https://github.com/lesiaukr/goit-de-hw-07

Masters degree | Data Engineering | HW | Apache Airflow | goit-de-hw-07

apache-airflow dags docker goit-de-hw-07 mysql python

Last synced: 09 May 2026

https://github.com/till-tietz/causalqueriestools

Bayesian inference with large causal models. CausalQueries companion.

bayes causal dags rcpp rcpparmadillo stan

Last synced: 20 Mar 2025

https://github.com/dmarks84/coursework_project_apache-airflow-kafka-on-toll-booth-data

Project for IBM Data Engineering & Python course on ETL & Big Data -- Read in live toll booth data, wrangles and transformed, and wrote into a SQL database

apache-airflow apache-kafka automation dags data-modeling databases eda elt etl mysql numpy pandas pipelines python sql

Last synced: 11 Apr 2026