https://github.com/zbrookle/avionix_airflow

Apache Airflow Hosted on a Kubernetes cluster, ready out of the box, with monitoring stack included (Grafana, ElasticSearch, Filebeat)
https://github.com/zbrookle/avionix_airflow

airflow apache-airflow avionix aws aws-eks chart-builder cluster elasticsearch filebeat grafana helm helm-chart kubernetes kubernetes-executor

Last synced: 2 months ago
JSON representation

Apache Airflow Hosted on a Kubernetes cluster, ready out of the box, with monitoring stack included (Grafana, ElasticSearch, Filebeat)

• Host: GitHub
• URL: https://github.com/zbrookle/avionix_airflow
• Owner: zbrookle
• License: bsd-3-clause
• Created: 2020-07-23T22:51:19.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2021-03-23T21:07:04.000Z (almost 4 years ago)
• Last Synced: 2024-08-10T10:09:13.211Z (5 months ago)
• Topics: airflow, apache-airflow, avionix, aws, aws-eks, chart-builder, cluster, elasticsearch, filebeat, grafana, helm, helm-chart, kubernetes, kubernetes-executor
• Language: Python
• Homepage: https://avionix-airflow.readthedocs.io/en/latest/
• Size: 391 KB
• Stars: 7
• Watchers: 1
• Forks: 2
• Open Issues: 4
• Metadata Files:
  • Readme: README.rst
  • License: LICENSE.txt

Awesome Lists containing this project

README

        
Avionix Airflow

===============

.. image:: https://github.com/zbrookle/avionix_airflow/workflows/CI/badge.svg?branch=master

    :target: https://github.com/zbrookle/avionix_airflow/actions?query=workflow

Why are there so many guides on how to set up `Apache Airflow`_ when the code could

just be given to you?!

.. _Apache Airflow: https://airflow.apache.org

 

Welcome to ``Avionix Airflow``, an out of the box solution to installing airflow on

Kubernetes_.

.. _Kubernetes: https://kubernetes.io

 

Currently supported platforms are Minikube_ and `AWS EKS`_ (managed nodegroups only),

pull requests adding support for GKE and AKS are welcome.

.. _Minikube: https://minikube.sigs.k8s.io/docs/

.. _AWS EKS: https://aws.amazon.com/eks/

 

Overview of Avionix Airflow Capabilities

----------------------------------------

``Avionix airflow`` provides the following out of the box solutions for airflow

1. Airflow Webserver and Scheduler with configured RBAC for the KubernetesExecutor

2. Airflow metric collection (statsd -> telegraf -> elasticsearch)

3. Airflow log collection (stdout -> filebeat -> elasticsearch)

4. Metric and log visualization (elasticsearch & postgres -> grafana)

Installation

------------

.. code-block::

    pip install avionix_airflow

Requirements

------------

- kubectl_

- helm_

.. _helm: https://helm.sh

.. _kubectl: https://kubernetes.io/docs/tasks/tools/install-kubectl/

Configuration

-------------

``Avionix airflow`` is configured by using Options objects, there are currently 4

different objects:

 

1. AirflowOptions

2. PostgresOptions

3. MonitoringOptions

4. CloudOptions (Specified using LocalOptions or AwsOptions)

5. RedisOptions (Only for use with CeleryExecutor, ***not recommended***)

These options are then passed into the function *get_chart_builder*, which can be

used to retrieve an avionix_ chart builder object. For instruction on how to use the

builder object, see avionix_

.. _avionix: https://github.com/zbrookle/avionix

FAQ

---

- How do I change the grafana dashboard?

    - You can change the dashboard by setting the MonitoringOptions role to "Admin"

    

Documentation

-------------

Additional documentation can be found `here `__