Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/digitalearthafrica/deafrica-airflow
A repository containing Airflow DAGs that are used for Digital Earth Africa.
https://github.com/digitalearthafrica/deafrica-airflow
Last synced: 3 months ago
JSON representation
A repository containing Airflow DAGs that are used for Digital Earth Africa.
- Host: GitHub
- URL: https://github.com/digitalearthafrica/deafrica-airflow
- Owner: digitalearthafrica
- License: apache-2.0
- Created: 2020-05-27T00:31:34.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2021-09-20T04:44:48.000Z (about 3 years ago)
- Last Synced: 2024-06-16T20:34:18.295Z (5 months ago)
- Language: Python
- Size: 1.19 MB
- Stars: 2
- Watchers: 12
- Forks: 3
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-apache-airflow - DEAfrica Airflow - Airflow libraries used by [Digital Earth Africa](https://digitalearthafrica.org/), an humanitarian effort to utilize satellite imagery of Africa. (Libraries, Hooks, Utilities)
README
# DEAfrica Airflow
A repository containing Airflow DAGs that are used for Digital Earth Africa.
## Development
### Local Editing of DAG's
DAG's can be locally edited and validated. Development can be done in `conda` or `venv` according to developer preference. Grab everything airflow and write DAG's. Use `autopep8` and `pylint` to achieve import validation and consistent formatting as the CI pipeline for this repository matures.
```bash
pip install apache-airflow[aws,kubernetes,postgres,redis,ssh,celery]
pip install pylint pylint-airflow
pylint dags plugins
```
## Pre-commit setup
Install pip modules
```bash
pip install apache-airflow[aws,kubernetes,postgres,redis,ssh,celery]==1.10.11
pip install shapely pyproj
pip install pylint pylint-airflow
```
A [pre-commit](https://pre-commit.com/) config is provided to automatically format
and check your code changes. This allows you to immediately catch and fix
issues before you raise a failing pull request (which run the same checks under
Travis).
If you don't use Conda, install pre-commit from pip:
pip install pre-commit
If you do use Conda, install from conda-forge (*required* because the pip
version uses virtualenvs which are incompatible with Conda's environments)
conda install pre_commit
Now install the pre-commit hook to the current repository:
pre-commit install
Your code will now be formatted and validated before each commit. You can also
invoke it manually by running `pre-commit run --all-files`
## Branch Deployment Layout
| Airflow Instance | Branch |
| --------------------------|:---------------:|
| [Airflow Oregon](https://airflow.digitalearth.africa) | develop |
| [Airflow Cape Town Prod](https://airflow-af.digitalearth.africa)| main |
| [Airflow Cape Town Dev](https://airflow.dev.digitalearth.africa) | develop-af |