Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ohld/airflow-dokku
My attempt to run Airflow with Dokku PaaS. Production ready.
https://github.com/ohld/airflow-dokku
Last synced: 18 days ago
JSON representation
My attempt to run Airflow with Dokku PaaS. Production ready.
- Host: GitHub
- URL: https://github.com/ohld/airflow-dokku
- Owner: ohld
- License: mit
- Created: 2021-01-22T10:12:56.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2021-03-27T07:53:00.000Z (almost 4 years ago)
- Last Synced: 2024-07-31T20:32:14.987Z (6 months ago)
- Size: 20.5 KB
- Stars: 9
- Watchers: 5
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- Install-anything-on-Dokku - Airflow
README
# airflow-dokku
My attempt to run Airflow with Dokku PaaS. In production-ready mode (with Postgres, Redis). This code should also work with Heroku (so this can be called `airflow-heroku`) but I have not tested it.
## Production
I really love to book a small droplet on Digital Ocean, install there a Dokku instance and then to try to run everything as Dokku apps.
### Create Dokku app for Airflow
``` bash
dokku apps:create airflow
```
### Plugins for production-ready Airflow
You need to install & attach Postgres and Redis (this is my random choice, you can try to replace these to Mysql & RabbitMQ).
``` bash
dokku postgres:create airflow
dokku postgres:link airflow airflow --alias "AIRFLOW__CORE__SQL_ALCHEMY_CONN"
```
I don't need to scale Airflow's Celery now so I'll skip the Redis installation and attachment (but it is almost the same as for Postgres).
### Setup the env variables
``` bash
dokku config:set airflow \
AIRFLOW__CORE__EXECUTOR=LocalExecutor \
AIRFLOW__CORE__FERNET_KEY='' \
AIRFLOW__CORE__LOAD_EXAMPLES=False \
AIRFLOW__CORE__SQL_ALCHEMY_CONN=
```
### Create Airflow superuser
Open Dokku app's shell:
``` bash
dokku enter airflow web
```
And then call the proper airflow CLI command:
``` bash
airflow users create \
--username admin \
--firstname Peter \
--lastname Parker \
--role Admin \
--email [email protected]
```
## Activate Dokku auto deploys with Github Actions CI
Open the file at `.github/workflows/dokku.yml` and add there the URL to the machine where your Dokku runs. Then open Github repo's Settings tab -> Secrets and add `SHH_PRIVATE_KEY` variable. GitHub Action CI will use this key to connect to your machine via ssh.
## HTTPs with Letsencrypt
If you attach a domain to a Dokku app you can add a HTTPs security to your service. This can be done with [Dokku Letsencrypt](https://github.com/dokku/dokku-letsencrypt) plugin.
```bash
dokku letsencrypt:enable airflow
```
----
[Contact me](https://twitter.com/danokhlopkov) if you have any questions.