https://github.com/infra-helpers/induction-airflow
Induction to Apache Airflow
https://github.com/infra-helpers/induction-airflow
airflow apache dag dependency engine graph orchestration orchestrator server workflow
Last synced: 5 months ago
JSON representation
Induction to Apache Airflow
- Host: GitHub
- URL: https://github.com/infra-helpers/induction-airflow
- Owner: infra-helpers
- Created: 2020-06-07T18:06:47.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2020-06-07T23:23:25.000Z (about 6 years ago)
- Last Synced: 2025-02-09T09:25:14.009Z (over 1 year ago)
- Topics: airflow, apache, dag, dependency, engine, graph, orchestration, orchestrator, server, workflow
- Homepage: https://infra-helpers.github.io/induction-airflow/
- Size: 1.15 MB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
Induction to Workflow Engines - Airflow
=======================================
[](https://unsplash.com/photos/Nijc2oQz10I)
# Overview
[That repository](https://github.com/infra-helpers/induction-airflow)
aims at providing end-to-end examples introducing how to setup, run and
monitor workflows, mainly for data processing pipelines.
In those tutorials, Airflow is used. A full end-to-end example is explained
step by step, and actually used for the [Open Travel Data (OPTD)
project](https://github.com/opentraveldata/opentraveldata).
The full details on how to setup an Airflow server on Proxmox LXC containers
are given in the [dedicated `airflow/` sub-folder](airflow/).
Such an Airflow cluster is actually the orchestration engine of the
[Open Travel Data (OPTD)
project](https://github.com/opentraveldata/opentraveldata).
For convenience, most of the Airflow examples are demonstrated both on a local
single-node installation (_e.g._, on a laptop) and on on the above-mentioned
cluster.
## Endpoints
* Airflow:
+ Local server: http://localhost:8080
+ Remote server: https://airflow.example.com
# Table of Content (ToC)
- [Induction to Workflow Engines - Airflow](#induction-to-workflow-engines---airflow)
- [Overview](#overview)
* [Endpoints](#endpoints)
- [Table of Content (ToC)](#table-of-content--toc-)
- [References](#references)
- [Configuration](#configuration)
Table of contents generated with markdown-toc
# References
* [Setup of a Proxmox host](https://github.com/cloud-helpers/kubernetes-hard-way-bare-metal/blob/master/proxmox/README.md)
* [Setup of LXC containers on Proxmox](https://github.com/cloud-helpers/kubernetes-hard-way-bare-metal/blob/master/lxc/README.md)
* [Induction to Monitoring with Elasticsearch (ES), Kibana and Kafka](https://github.com/infra-helpers/induction-monitoring)
# Configuration
* See [`airflow/`](airflow/)