Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with airflow

A curated list of projects in awesome lists tagged with airflow .

https://github.com/apache/dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

airflow azkaban cloud-native data-pipelines job-scheduler orchestration powerful-data-pipelines task-scheduler workflow workflow-orchestration workflow-schedule

Last synced: 15 Jan 2026

https://github.com/webankfintech/dataspherestudio

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

airflow atlas azkaban dataworks davinci dolphinscheduler flink governance griffin hadoop hive hue kettle linkis spark supperset tableau visualis workflow zeppelin

Last synced: 14 May 2025

https://github.com/WeBankFinTech/DataSphereStudio

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

airflow atlas azkaban dataworks davinci dolphinscheduler flink governance griffin hadoop hive hue kettle linkis spark supperset tableau visualis workflow zeppelin

Last synced: 14 Mar 2025

https://github.com/teamclairvoyant/airflow-maintenance-dags

A series of DAGs/Workflows to help maintain the operation of Airflow

airflow airflow-maintenance-dags apache-airflow cleanup dag maintenance workflow

Last synced: 15 May 2025

https://github.com/astronomer/dag-factory

Construct Apache Airflow DAGs Declaratively via YAML configuration files

airflow apache-airflow dags python

Last synced: 13 May 2025

https://github.com/astronomer/astronomer-cosmos

Run your dbt Core or dbt Fusion projects as Apache Airflow DAGs and Task Groups with a few lines of code

airflow airflow-operators apache-airflow dbt python workflow

Last synced: 27 Aug 2026

https://github.com/iusztinpaul/energy-forecasting

🌀 𝗧𝗵𝗲 𝗙𝘂𝗹𝗹 𝗦𝘁𝗮𝗰𝗸 𝟳-𝗦𝘁𝗲𝗽𝘀 𝗠𝗟𝗢𝗽𝘀 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 | 𝗟𝗲𝗮𝗿𝗻 𝗠𝗟𝗘 & 𝗠𝗟𝗢𝗽𝘀 for free by designing, building and deploying an end-to-end ML batch system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 2.5 𝘩𝘰𝘶𝘳𝘴 𝘰𝘧 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 & 𝘷𝘪𝘥𝘦𝘰 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴

3-pipeline-design airflow batch-processing cicd data-versioning docker fastapi feature-store gcp github-actions great-expectations hopsworks ml-monitoring mlops model-registry poetry python sktime streamlit weights-and-biases

Last synced: 15 May 2025

https://github.com/raystack/optimus

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

airflow analytics analytics-engineering automation bigquery business-intelligence data-modelling data-pipelines data-transformation data-warehouse dataops elt etl golang workflows

Last synced: 16 May 2025

https://github.com/airflow-helm/charts

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.

airflow chart charts helm helm-chart helm-charts k8s kubernetes

Last synced: 06 Aug 2025

https://github.com/ankurchavda/streamify

A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

airflow data-engineering dbt gcp kafka python spark

Last synced: 19 Jul 2025

https://github.com/josephmachado/beginner_de_project

Beginner data engineering project - batch edition

airflow database docker emr engineering etl python redshift redshift-cluster spark

Last synced: 15 May 2025

https://github.com/andreax79/airflow-code-editor

A plugin for Apache Airflow that allows you to edit DAGs in browser

airflow airflow-plugin apache-airflow python

Last synced: 03 Apr 2026

https://github.com/gocardless/airflow-dbt

Apache Airflow integration for dbt

airflow airflow-dbt dbt

Last synced: 29 Jul 2025

https://github.com/apache/airflow-client-python

Apache Airflow - OpenApi Client for Python

airflow apache apache-airflow apache-airflow-client python

Last synced: 14 May 2025

https://github.com/fieldryand/goflow

Simple but powerful DAG scheduler and dashboard

airflow dashboard directed-acyclic-graph go job-scheduler schedule workflow-engine

Last synced: 14 Mar 2025

https://github.com/cubefs/compass

Compass is a task diagnosis platform for bigdata

airflow bigdata diagnose dolphinscheduler flink hadoop mapreduce scheduler spark sql

Last synced: 15 May 2025

https://github.com/villasv/aws-airflow-stack

Turbine: the bare metals that gets you Airflow

airflow airflow-cluster airflow-cookbook aws aws-cloudformation

Last synced: 14 Mar 2025

https://github.com/astronomer/astro-sdk

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

airflow apache-airflow bigquery dags data-analysis data-science elt etl gcs pandas postgres python s3 snowflake sql sqlite workflows

Last synced: 13 Apr 2025

https://github.com/huseinzol05/Gather-Deployment

Gathers Python deployment, infrastructure and practices.

airflow docker docker-compose kafka pyflink pyspark python tensorflow

Last synced: 06 Apr 2025

https://github.com/feluelle/airflow-diagrams

Auto-generated Diagrams from Airflow DAGs. 🔮 🪄

airflow auto-generated cli diagrams python

Last synced: 08 Oct 2025

https://github.com/teamclairvoyant/airflow-rest-api-plugin

A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces

airflow airflow-plugin airflow-webserver apache-airflow plugin rest-api

Last synced: 13 Jul 2025

https://github.com/astronomer/airflow-chart

A Helm chart to install Apache Airflow on Kubernetes

airflow apache-airflow astro-private-cloud astronomer-software helm-chart kubernetes

Last synced: 21 Jan 2026

https://github.com/epoch8/airflow-exporter

Airflow plugin to export dag and task based metrics to Prometheus.

airflow airflow-exporter prometheus

Last synced: 24 Jan 2026

https://github.com/chandulal/airflow-testing

Airflow Unit Tests and Integration Tests

airflow airflow-dags airflow-testing testing

Last synced: 18 Jul 2025

https://github.com/pipeline-tools/gusty

Making DAG construction easier

airflow data-etl data-pipeline

Last synced: 12 Jan 2026

https://github.com/astronomer/agents

AI agent tooling for data engineering workflows.

agents ai airflow apache-airflow claude cursor data-engineering mcp skills

Last synced: 27 Feb 2026

https://github.com/astronomer/ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer

airflow astronomer llm llm-ops

Last synced: 16 May 2025

https://github.com/kaxil/airflowctl

A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects

airflow apache-airflow cli dags python

Last synced: 08 Apr 2025

https://github.com/GokuMohandas/data-engineering

Construct a modern data stack and orchestration the workflows to create high quality data for analytics and ML applications.

airflow data-engineering data-warehouse dbt etl machine-learning mlops orchestration

Last synced: 11 Sep 2025

https://github.com/apachecn/airflow-doc-zh

:book: [译] Airflow 中文文档

airflow

Last synced: 06 Apr 2025

https://github.com/gokumohandas/data-engineering

Construct a modern data stack and orchestration the workflows to create high quality data for analytics and ML applications.

airflow data-engineering data-warehouse dbt etl machine-learning mlops orchestration

Last synced: 12 May 2025

https://github.com/dataplane-app/dataplane

Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.

airflow data data-analysis data-engineering data-integration data-pipelines data-science dataplane datawarehouse etl finance golang kubernetes pipelines robotics-process-automation rpa scheduler workflow workflow-automation workflows

Last synced: 27 Dec 2025

https://github.com/jgoerner/data-science-stack-cookiecutter

🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)

airflow apistar cookiecutter data-science docker docker-image jupyter minio postgres python superset

Last synced: 29 Mar 2025

https://github.com/andresionek91/airflow-autoscaling-ecs

Airflow Deployment on AWS ECS Fargate Using Cloudformation

airflow airflow-autoscaling-ecs airflow-deployment airflow-ecs data-engineering

Last synced: 24 Dec 2025

https://github.com/godatadriven/whirl

Fast iterative local development and testing of Apache Airflow workflows

airflow docker local-development testing

Last synced: 22 Jan 2026

https://github.com/ris-tlp/audiophile-e2e-pipeline

Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard.

airflow aws data-engineering metabase python terraform

Last synced: 10 Apr 2025

https://github.com/tomasfarias/airflow-dbt-python

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

airflow airflow-hook airflow-operators analytics data-engineering dbt python python3

Last synced: 15 May 2025

https://github.com/apache/airflow-client-go

Apache Airflow - OpenApi Client for Go

airflow apache apache-airflow apache-airflow-client go

Last synced: 06 Apr 2025

https://github.com/googlecloudplatform/cortex-data-foundation

Data Foundation - Google Cloud Cortex Framework

airflow bigquery cloud google googlecloud salesforce sap

Last synced: 16 May 2025

https://github.com/mozilla/telemetry-airflow

Airflow configuration for Telemetry

airflow mozilla-telemetry

Last synced: 16 May 2025

https://github.com/garystafford/tickit-data-lake-demo

Resources for video demonstrations and blog posts related to DataOps on AWS

airflow aws data-lake dataops devops redshift

Last synced: 07 May 2025

https://github.com/dsaidgovsg/airflow-pipeline

An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR

airflow docker hadoop spark

Last synced: 27 Mar 2025

https://github.com/apache/airflow-site

Apache Airflow Website

airflow apache hugo

Last synced: 16 Apr 2026

https://github.com/google/starthinker

Reference framework for building data workflows provided by Google. Accelerates authentication, logging, scheduling, and deployment of solutions using GCP. To borrow a tagline.. "The framework for professionals with deadlines."

airflow app-engine automation bigquery cloud-functions cm360 colab-notebook data-science django dv360 google-ads google-analytics logger python scheduler ui workflows

Last synced: 04 Oct 2025

https://github.com/nicor88/aws-ecs-airflow

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

airflow airflow-ecs aws dag docker ecr ecs etl fargate terraform

Last synced: 13 Apr 2025

https://github.com/jgoerner/beyond-jupyter

🐍💻📊 All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)

airflow apache apistar data-science docker docker-compose jupyter jupyter-notebook minio postgres superset

Last synced: 16 Mar 2025

https://github.com/Tauffer-Consulting/domino

User friendly and open source platform for workflow creation and monitoring

ai airflow containers data gui kubernetes open-source python workflows

Last synced: 22 Apr 2025

https://github.com/apache/incubator-liminal

Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.

ai airflow big-data data-science machine-learning ml workflows

Last synced: 14 Jan 2026

https://github.com/qubole/afctl

afctl helps to manage and deploy Apache Airflow projects faster and smoother.

airflow cli deployment docker management

Last synced: 07 May 2025

https://github.com/astronomer/astronomer-providers

Airflow Providers containing Deferrable Operators & Sensors from Astronomer

airflow airflow-operators airflow-providers apache-airflow python workflow

Last synced: 04 Apr 2025

https://github.com/ninja-van/airflow-boilerplate

A complete development environment setup for working with Airflow

airflow apache-airflow boilerplate pycharm python

Last synced: 12 Jul 2025

https://github.com/iam-mhaseeb/skytrax-data-warehouse

A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.

airflow data-analysis data-analytics data-cleaning data-engineering data-orchestration data-processing data-visualization data-warehouse data-warehousing database docker metabase python python3 redshift s3 s3-bucket sql

Last synced: 12 Aug 2025

https://github.com/datacamp/viewflow

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

airflow apache-airflow data-engineering data-science packages python workflow

Last synced: 20 Aug 2025

https://github.com/bryzgaloff/airflow-clickhouse-plugin

The most popular ClickHouse plugin for Airflow. 🔝 Top-1% downloads on PyPI: https://pypi.org/project/airflow-clickhouse-plugin! Based on mymarilyn/clickhouse-driver.

airflow clickhouse python python3

Last synced: 04 Apr 2025

https://github.com/k3ai/k3ai

A lightweight tool to get an AI Infrastructure Stack up in minutes not days. K3ai will take care of setup K8s for You, deploy the AI tool of your choice and even run your code on it.

airflow artificial-intelligence datascience devops infrastructure-as-code kubeflow machine-learning mlflow mlops notebooks

Last synced: 06 Feb 2026

https://github.com/sungchun12/airflow-toolkit

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) :desktop_computer: >> [ :rocket:, :ship: ]

actions airflow airflow-environments airflow-toolkit cloud cloud-composer composer dbt docker gcp google-cloud hcl kubernetes kubernetes-deployment python python3 shell-script terraform terragrunt terragrunt-deployment

Last synced: 01 Aug 2025

https://github.com/astronomer/ap-airflow

Astronomer Core Docker Images

airflow apache docker kubernetes

Last synced: 15 May 2025

https://github.com/scribd/objinsync

Continuously synchronize directories from remote object store to local filesystem

airflow cplat s3

Last synced: 05 Apr 2025

https://github.com/mikeghen/airflow-tutorial

Use Airflow to move data from multiple MySQL databases to BigQuery

airflow bigquery mysql-database

Last synced: 26 Oct 2025

https://github.com/feluelle/airflint

Enforce Best Practices for all your Airflow DAGs. ⭐

airflow best-practices dag formatter linter

Last synced: 05 Apr 2025

https://github.com/blockchain-etl/polygon-etl

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

airflow bigquery cryptocurrency data-engineering etl gcp matic-network maticnetwork polygon

Last synced: 05 Apr 2025

https://github.com/star3am/hashiqube

HashiQube - The Ultimate Hands on DevOps Lab running All the HashiCorp Products in a Github Codespace or a Docker Container using Vagrant or Docker Compose

airflow ansible ansible-project boundary consul dbt devops gitlab hashicorp hashiqube jenkins minikube nomad packer terraform vagrant vault waypoint

Last synced: 14 Aug 2025

https://github.com/mastak/airflow_multi_dagrun

triggering a DAG run multiple times

airflow airflow-plugin

Last synced: 19 Jan 2026

https://github.com/claude89757/wechat-on-airflow

基于 Apache Airflow 的微信智能应用编排框架，通过可视化工作流驱动 AI 与数据自动化任务。支持 智能客服（多轮对话/知识库）、AI 图文/短视频生成、智能提醒等应用，灵活扩展多模态交互与大模型能力。

airflow chatbot wechat

Last synced: 19 Aug 2026

https://github.com/powerdatahub/terraform-aws-airflow

Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor

airflow apache-airflow aws celery hacktoberfest terraform terraform-module terraform-modules

Last synced: 23 Jul 2025

https://github.com/mendhak/airflow-ms-teams-operator

Airflow operator that can send messages to MS Teams

airflow airflow-plugin ms-teams python

Last synced: 25 Feb 2026

https://github.com/idealista/airflow-role

Ansible role to install Apache Airflow

airflow airflow-role ansible ansible-role apache-airflow debian

Last synced: 25 Oct 2025

https://github.com/bahchis/airflow-cookbook

Airflow workflow management platform chef cookbook.

airflow airflow-cookbook chef-cookbook

Last synced: 14 Jan 2026

https://github.com/domenp/aircal

Visualize Airflow's schedule by exporting future DAG runs as events to Google Calendar.

airflow calendar dag google-calendar schedule

Last synced: 13 Jul 2025