https://github.com/maxg203/airflow-workflows

Provides a clean, Django-inspired class-based DAG syntax for Apache Airflow.
https://github.com/maxg203/airflow-workflows

Last synced: about 1 month ago
JSON representation

Provides a clean, Django-inspired class-based DAG syntax for Apache Airflow.

• Host: GitHub
• URL: https://github.com/maxg203/airflow-workflows
• Owner: maxg203
• License: apache-2.0
• Created: 2019-05-22T11:20:14.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2019-05-28T14:07:16.000Z (over 5 years ago)
• Last Synced: 2024-08-11T10:49:04.045Z (3 months ago)
• Language: Python
• Homepage: https://pypi.org/project/airflow-workflows/
• Size: 22.5 KB
• Stars: 9
• Watchers: 1
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Workflows

Workflows are a cleaner way of implementing DAGs using a Django-inspired class-based syntax.

## Simple Example

Let's create a single Airflow DAG, whose name is a camelcased version of the class name, and whose operator dependencies are in the order they are defined.

There is an option to override the default [`dependencies`](https://github.com/maxg203/airflow-workflows/blob/master/workflows.py#L165) method implementation to customise the dependency chain for your use case.

```python

import workflows

class ExampleWorkflow(workflows.Workflow):

    class Meta:

        schedule_interval = '0 9 * * *'

    do_something_useful = workflows.PythonOperator(

        python_callable=lambda **kwargs: print('something useful'),

    )

    something_else = workflows.PythonOperator(

        python_callable=lambda **kwargs: print('Something not useful'),

    )

globals()[ExampleWorkflow.DAG.dag_id] = ExampleWorkflow.DAG

```

## Dynamic DAG Example

Let's create (in this case three) DAGs, created dynamically and based on the `ExampleWorkflow` class as implemented above. In other words, they will share the same DAG metadata (so schedule in this case).

```python

import workflows

workflow_names = [

    'Test1',

    'Test2',

    'Test3',

]

for workflow in workflow_names:

    WorkflowClass = workflows.create_workflow(

        workflow,

        base=ExampleWorkflow,

    )

    globals()[WorkflowClass.DAG.dag_id] = WorkflowClass.DAG

```