https://github.com/msumit/airflow-pex

Example deploying Apache Airflow as a pex
https://github.com/msumit/airflow-pex

Last synced: 2 months ago
JSON representation

Example deploying Apache Airflow as a pex

• Host: GitHub
• URL: https://github.com/msumit/airflow-pex
• Owner: msumit
• Created: 2019-09-17T05:44:07.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2020-02-04T06:47:20.000Z (almost 5 years ago)
• Last Synced: 2024-08-03T02:03:29.995Z (5 months ago)
• Language: Python
• Size: 8.79 KB
• Stars: 4
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-apache-airflow - Distribute & deploy Apache Airflow via Python PEX files - Example repo with steps to bundle, distribute, & deploy Apache Airflow as PEX files. (Airflow deployment solutions)

README

        
# Airflow PEX

This repo is highly inspired by another similar work done [here](https://github.com/traviscrawford/airflow-pex-example). 

The only reason I re-attempted it, so that I can get the first hand experience and also resolve some difficulties I 

faced while using the above solution. 

Build an Airflow pex by running:

```$xslt

./pants binary src/python/ariflow:airflow

``` 

This produces `airflow.pex`, a single file that is analogous to a statically-lined binary. It's a self-contained, 

runnable Airflow you can `scp` to another machine and run. However, ideally you would zip this whole folder and `scp` 

to remote machine. 

You can then run Airflow commands, using `airflow_cli` wrapper, which internally calls the `airflow.pex` file. You can 

also use this wrapper file to modify the Airflow config variables. 

```$xslt

./airflow_cli version

[2019-09-16 22:52:23,399] {__init__.py:51} INFO - Using executor SequentialExecutor

  ____________       _____________

 ____    |__( )_________  __/__  /________      __

____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /

___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /

 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/  v1.10.5

```

To start the webserver

```$xslt

./airflow_cli webserver

```

To start the Scheduler

```$xslt

./airflow_cli scheduler

```

## Working Environment

If you've noticed the dags and plugins folder are part of `airflow_home` folder, but ideally they should be part of 

the `airflow.pex` itself. We can easily bundle dags and plugins as well in the `airflow.pex` like we've bundled `libs` 

folder, but unfortunately Airflow can't read them from `airflow.pex/dags` or `airflow.pex/plugins` directly, so  you can 

unzip the final PEX at the server and move them inside `airflow_home` folder.