Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/aventer-ug/airflow-provider-mesos
Apache Airflow Provider for Mesos
https://github.com/aventer-ug/airflow-provider-mesos
airflow mesos python
Last synced: 24 days ago
JSON representation
Apache Airflow Provider for Mesos
- Host: GitHub
- URL: https://github.com/aventer-ug/airflow-provider-mesos
- Owner: AVENTER-UG
- Created: 2021-04-28T16:19:07.000Z (almost 4 years ago)
- Default Branch: master
- Last Pushed: 2024-03-20T15:13:45.000Z (10 months ago)
- Last Synced: 2024-05-01T10:52:22.713Z (9 months ago)
- Topics: airflow, mesos, python
- Language: Python
- Homepage: https://aventer-ug.github.io/airflow-provider-mesos/
- Size: 1.31 MB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGES.txt
- Funding: .github/FUNDING.yml
Awesome Lists containing this project
README
# Provider for Apache Airflow 2.x to schedule Apache Mesos
[](https://github.com/m3scluster/airflow-provider-mesos/issues)
[](https://matrix.to/#/#mesos:matrix.aventer.biz?via=matrix.aventer.biz)
[](https://m3scluster.github.io/airflow-provider-mesos/)
This provider for Apache Airflow contain the following features:
- MesosExecuter - A scheduler to run Airflow DAG's on mesos
- MesosOperator - To executer Airflow tasks on mesos. (TODO)
## Issues
To open an issue, please use this place: https://github.com/m3scluster/airflow-provider-mesos/issues
## Requirements
- Airflow 2.x
- Apache Mesos minimum 1.6.x
## How to install and configure
On the Airflow Server, we have to install the mesos provider.
```bash
pip install avmesos_airflow_provider
```
Then we will configure Airflow.
```bash
vim airflow.cfg
executor = avmesos_airflow_provider.executors.mesos_executor.MesosExecutor
[mesos]
mesos_ssl = True
master = leader.mesos:5050
framework_name = Airflow
checkpoint = True
attributes = False
failover_timeout = 604800
command_shell = True
task_cpu = 1
task_memory = 20000
authenticate = True
default_principal =
default_secret =
docker_image_slave =
docker_volume_driver = local
docker_volume_dag_name = airflowdags
docker_volume_dag_container_path = /home/airflow/airflow/dags/
docker_sock = /var/run/docker.sock
docker_volume_logs_name = airflowlogs
docker_volume_logs_container_path = /home/airflow/airflow/logs/
docker_environment = '[{ "name":"", "value":"" }, { ... }]'
api_username =
api_password =
```
## DAG example with mesos executor
```python
from airflow import DAG
from datetime import datetime, timedelta
from airflow.operators.dummy_operator import DummyOperator
from airflow.providers.docker.operators.docker import DockerOperator
from airflow.operators.python import PythonOperator
default_args = {
'owner' : 'airflow',
'description' : 'Use of the DockerOperator',
'depend_on_past' : True,
}
with DAG('docker_dag2', default_args=default_args, schedule_interval="*/10 * * * * ", catchup=True, start_date=datetime.now()) as dag:
t2 = DockerOperator(
task_id='docker_command',
image='centos:latest',
api_version='auto',
auto_remove=False,
command="/bin/sleep 600",
docker_url='unix:///var/run/docker.sock',
executor_config={
"cpus": 2.0,
"mem_limit": 2048
}
)
t2
```
## Development
For development and testing we deliver a nix-shell file to install airflow, our airflow provider and postgresql.
To use it, please follow the following steps:
1. Run mesos-mini:
```bash
docker run --rm --name mesos --privileged=true --shm-size=30gb -it --net host avhost/mesos-mini:1.11.0-0.2.0-1 /lib/systemd/systemd
```
2. Use nix-shell:
```bash
nix-shell
> airflow scheduler
```
3. On the mesos-ui (http://localhost:5050) you will see Airflow as framework.