https://github.com/getindata/kedro-airflow-k8s

Kedro Plugin to support running pipelines on Kubernetes using Airflow.
https://github.com/getindata/kedro-airflow-k8s

airflow k8s kedro kedro-airflow kedro-plugin kuberentes machinelearning mlops

Last synced: 2 months ago
JSON representation

Kedro Plugin to support running pipelines on Kubernetes using Airflow.

• Host: GitHub
• URL: https://github.com/getindata/kedro-airflow-k8s
• Owner: getindata
• License: apache-2.0
• Created: 2021-01-27T09:26:27.000Z (about 4 years ago)
• Default Branch: develop
• Last Pushed: 2023-08-29T14:25:26.000Z (over 1 year ago)
• Last Synced: 2024-10-29T05:34:49.799Z (3 months ago)
• Topics: airflow, k8s, kedro, kedro-airflow, kedro-plugin, kuberentes, machinelearning, mlops
• Language: Python
• Homepage: https://kedro-airflow-k8s.readthedocs.io
• Size: 316 KB
• Stars: 29
• Watchers: 12
• Forks: 11
• Open Issues: 19
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Codeowners: .github/CODEOWNERS

Awesome Lists containing this project

• awesome-kedro - kedro-airflow-k8s - Enables running a Kedro pipeline with Airflow on a Kubernetes cluster. ([Kedro plugins](https://docs.kedro.org/en/stable/extend_kedro/plugins.html))

README

        
# Kedro Airflow K8S Plugin

[![Python Version](https://img.shields.io/badge/python-3.7%20%7C%203.8-blue.svg)](https://github.com/getindata/kedro-airflow-k8s)

[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) 

[![SemVer](https://img.shields.io/badge/semver-2.0.0-green)](https://semver.org/)

[![PyPI version](https://badge.fury.io/py/kedro-airflow-k8s.svg)](https://pypi.org/project/kedro-airflow-k8s/)

[![Downloads](https://img.shields.io/pypi/dm/kedro-airflow-k8s)](https://img.shields.io/pypi/dm/kedro-airflow-k8s) 

[![Maintainability](https://api.codeclimate.com/v1/badges/f2ef65a9be497267c738/maintainability)](https://codeclimate.com/github/getindata/kedro-airflow-k8s/maintainability)

[![Test Coverage](https://api.codeclimate.com/v1/badges/f2ef65a9be497267c738/test_coverage)](https://codeclimate.com/github/getindata/kedro-airflow-k8s/test_coverage)

[![Documentation Status](https://readthedocs.org/projects/kedro-airflow-k8s/badge/?version=latest)](https://kedro-airflow-k8s.readthedocs.io/en/latest/?badge=latest)

## About

The main purpose of this plugin is to enable running kedro pipeline with Airflow on Kubernetes Cluster. In difference to 

[kedro-airflow](https://github.com/quantumblacklabs/kedro-airflow) this plugin does not require additional libraries installed

in airflow runtime, it uses K8S infrastructure instead. It supports translation

from Kedro pipeline DSL to [airflow](https://airflow.apache.org/docs/apache-airflow/stable/python-api-ref.html) (python API)

and generation of airflow [DAGs](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#dags).

The plugin can be used together with `kedro-docker` to simplify preparation of docker image for pipeline execution.   

## Documentation

For detailed documentation refer to https://kedro-airflow-k8s.readthedocs.io/

## Usage guide

```

Usage: kedro airflow-k8s [OPTIONS] COMMAND [ARGS]...

 

Options:

  -e, --env TEXT  Environment to use.

  -p, --pipeline TEXT  Pipeline name to pick.

  -h, --help      Show this message and exit.

Commands:

  compile          Create an Airflow DAG for a project

  init             Initializes configuration for the plugin

  list-pipelines   List pipelines generated by this plugin

  run-once         Uploads pipeline to Airflow and runs once

  schedule         Uploads pipeline to Airflow with given schedule

  ui               Open Apache Airflow UI in new browser tab

  upload-pipeline  Uploads pipeline to Airflow DAG location

```