Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/digitalearthafrica/deafrica-airflow

A repository containing Airflow DAGs that are used for Digital Earth Africa.
https://github.com/digitalearthafrica/deafrica-airflow

Last synced: 3 months ago
JSON representation

A repository containing Airflow DAGs that are used for Digital Earth Africa.

Awesome Lists containing this project

README

        

# DEAfrica Airflow
A repository containing Airflow DAGs that are used for Digital Earth Africa.

## Development

### Local Editing of DAG's

DAG's can be locally edited and validated. Development can be done in `conda` or `venv` according to developer preference. Grab everything airflow and write DAG's. Use `autopep8` and `pylint` to achieve import validation and consistent formatting as the CI pipeline for this repository matures.

```bash
pip install apache-airflow[aws,kubernetes,postgres,redis,ssh,celery]
pip install pylint pylint-airflow

pylint dags plugins
```

## Pre-commit setup

Install pip modules

```bash
pip install apache-airflow[aws,kubernetes,postgres,redis,ssh,celery]==1.10.11
pip install shapely pyproj
pip install pylint pylint-airflow
```

A [pre-commit](https://pre-commit.com/) config is provided to automatically format
and check your code changes. This allows you to immediately catch and fix
issues before you raise a failing pull request (which run the same checks under
Travis).

If you don't use Conda, install pre-commit from pip:

pip install pre-commit

If you do use Conda, install from conda-forge (*required* because the pip
version uses virtualenvs which are incompatible with Conda's environments)

conda install pre_commit

Now install the pre-commit hook to the current repository:

pre-commit install

Your code will now be formatted and validated before each commit. You can also
invoke it manually by running `pre-commit run --all-files`

## Branch Deployment Layout

| Airflow Instance | Branch |
| --------------------------|:---------------:|
| [Airflow Oregon](https://airflow.digitalearth.africa) | develop |
| [Airflow Cape Town Prod](https://airflow-af.digitalearth.africa)| main |
| [Airflow Cape Town Dev](https://airflow.dev.digitalearth.africa) | develop-af |