https://github.com/mpolinowski/apache-airflow-intro
Introduction to Apache Airflow
https://github.com/mpolinowski/apache-airflow-intro
apache-airflow dag data-pipelines machine-learning
Last synced: 12 months ago
JSON representation
Introduction to Apache Airflow
- Host: GitHub
- URL: https://github.com/mpolinowski/apache-airflow-intro
- Owner: mpolinowski
- Created: 2023-02-05T15:11:39.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2023-02-06T05:27:37.000Z (over 3 years ago)
- Last Synced: 2025-03-23T13:15:13.375Z (about 1 year ago)
- Topics: apache-airflow, dag, data-pipelines, machine-learning
- Language: Python
- Homepage: https://mpolinowski.github.io/docs/IoT-and-Machine-Learning/AIOps/2023-02-01-apache-airflow-introduction/2023-02-01
- Size: 9.77 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Apache Airflow
Apache Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows. Airflow’s extensible Python framework enables you to build workflows connecting with virtually any technology. A web interface helps manage the state of your workflows. Airflow is deployable in many ways, varying from a single process on your laptop to a distributed setup to support even the biggest workflows.
The main characteristic of Airflow workflows is that all workflows are defined in Python code allowing us to write dynamic _Data Pipelines_ called __DAG__'s — a mathematical abstraction of a pipeline. “Workflows as code” serves several purposes:
* __Dynamic__: Airflow pipelines are configured as Python code, allowing for dynamic pipeline generation.
* __Extensible__: The Airflow framework contains operators to connect with numerous technologies. All Airflow components are extensible to easily adjust to your environment.
* __Flexible__: Workflow parameterization is built-in leveraging the Jinja templating engine.
___
## Articles
* [Apache Airflow Introduction](https://mpolinowski.github.io/docs/IoT-and-Machine-Learning/AIOps/2023-02-01-apache-airflow-introduction/2023-02-01)
* [Apache Airflow Data Pipelines (DAGs)](https://mpolinowski.github.io/docs/IoT-and-Machine-Learning/AIOps/2023-02-04-apache-airflow-data-pipelines/2023-02-04)
* [Apache Airflow DAG Scheduling](https://mpolinowski.github.io/docs/IoT-and-Machine-Learning/AIOps/2023-02-05-apache-airflow-scheduler/2023-02-05)
* [Apache Airflow Dynamic DAGs](https://mpolinowski.github.io/docs/IoT-and-Machine-Learning/AIOps/2023-02-06-apache-airflow-dynamic-dags/2023-02-06)