https://github.com/globocom/auror-core
Azkaban Auror core for flow creation
https://github.com/globocom/auror-core
Last synced: 10 months ago
JSON representation
Azkaban Auror core for flow creation
- Host: GitHub
- URL: https://github.com/globocom/auror-core
- Owner: globocom
- License: apache-2.0
- Created: 2018-11-22T20:33:42.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2020-10-06T13:08:23.000Z (almost 6 years ago)
- Last Synced: 2025-09-25T04:59:52.805Z (11 months ago)
- Language: Python
- Size: 91.8 KB
- Stars: 10
- Watchers: 13
- Forks: 7
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
[](https://travis-ci.com/globocom/auror-core)
# Auror Core
> Simple Flow creation for Azkaban
Photo by Rhii Photography on Unsplash
- [Auror Core](#auror-core)
- [Install](#install)
- [Supported Job Types](#supported-job-types)
- [V1](#v1)
- [V2](#v2)
- [Usage](#usage)
- [Creating a simple Azkaban flow with one command](#creating-a-simple-azkaban-flow-with-one-command)
- [Creating a simple V2 Azkaban flow with one command](#creating-a-simple-v2-azkaban-flow-with-one-command)
- [Creating Flows with dependencies](#creating-flows-with-dependencies)
- [Sharing job attributes](#sharing-job-attributes)
- [Job with extra customization and configuration](#job-with-extra-customization-and-configuration)
- [Integrating with Flow (just for V1)](#integrating-with-flow-just-for-v1)
- [Using Flow Params](#using-flow-params)
- [Using Flow Environment Variables](#using-flow-environment-variables)
- [Using Flow Environment Variables and Params](#using-flow-environment-variables-and-params)
- [Join multiple variables in one](#join-multiple-variables-in-one)
- [Load jobs from YAML File (just for V2)](#load-jobs-from-yaml-file-just-for-v2)
- [Dump memory flows to a Python File (just for V2)](#dump-memory-flows-to-a-python-file-just-for-v2)
- [Plugins](#plugins)
- [Contribute](#contribute)
## Install
```bash
pip install auror_core
```
## Supported Job Types
### V1
* **Command**
* **Flow**
### V2
* **Command**
## Usage
### Creating a simple Azkaban flow with one command
You just need to import job type and project
```python
from auror_core.v1.job import Job, Command
from auror_core import Project
com1 = Job()\
.as_type(Command)\
.with_name("commands job")\
.with_command("bash echo 1")\
.with_another_command("bash echo 2")
Project("folder_to_generate_files", com1).write()
```
### Creating a simple V2 Azkaban flow with one command
V2 flow is implemented under v2 subfolder with same job types
```python
from auror_core.v2.job import Job, Command
from auror_core import Project
com1 = Job()\
.as_type(Command)\
.with_name("commands job")\
.with_command("bash echo 1")\
.with_another_command("bash echo 2")
Project("folder_to_generate_files", com1).is_v2().write()
```
### Creating Flows with dependencies
```python
from auror_core.v2.job import Job, Command
from auror_core import Project
com1 = Job()\
.as_type(Command)\
.with_name("commands job")\
.with_command("bash echo 1")\
.with_another_command("bash echo 2")
com2 = Command()\
.with_name("sub command job")\
.with_command("bash echo 1")\
.with_dependencies(com1)
Project("folder_to_generate_files", com1, com2).is_v2().write()
```
### Sharing job attributes
Organize jobs with same configuration
```python
from auror_core.v2.job import Command
from auror_core import Project
com = Command()\
.with_command("bash echo 1")
com1 = com.with_name("commands job")\
.with_another_command("bash echo 2")
com2 = com.with_name("sub command job")\
.with_dependencies(com1)
Project("folder_to_generate_files", com1, com2).is_v2().write()
```
### Job with extra customization and configuration
Simulating a Command with base Job (NOT RECOMMENDED)
```python
from auror_core.v1.job import Job
from auror_core import Project
com1 = Job()\
.with_name("commands job")\
.with_(command="bash echo 1")
com1._type = "command"
Project("folder_to_generate_files", com1).write()
```
### Integrating with Flow (just for V1)
V2 already have flow included
```python
from auror_core.v1.job import Command, Flow, Job
from auror_core import Project
com1 = Command()\
.with_name("commands job")\
.with_command("bash echo 1")
flow = Job()\
.as_type(Flow)\
.with_name("flow")\
.with_dependencies(com1)
Project("folder_to_generate_files", com1, flow).write()
```
### Using Flow Params
```python
from auror_core.v2.job import Command
from auror_core.v2.params import Params
from auror_core import Project
params = Params(
teste1="my test",
variable="my variable"
)
com = Command()\
.with_command("bash echo ${variable}")
Project("folder_to_generate_files", com)\
.is_v2()\
.with_params(params)\
.write()
```
### Using Flow Environment Variables
```python
from auror_core.v2.job import Command
from auror_core.v2.params import Env
from auror_core import Project
env = Env(
TESTE="my test",
VARIABLE="my variable"
)
com = Command()\
.with_command("bash echo $VARIABLE")
Project("folder_to_generate_files", com)\
.is_v2()\
.with_params(env)\
.write()
```
### Using Flow Environment Variables and Params
```python
from auror_core.v2.job import Command
from auror_core.v2.params import Env, Params
from auror_core import Project
env = Env(
TESTE="my test",
VARIABLE="my variable"
)
params = Params(
teste1="my test",
variable="my variable"
)
com = Command()\
.with_command("bash echo $VARIABLE ${variable}")
Project("folder_to_generate_files", com)\
.is_v2()\
.with_params(params, env)\
.write()
```
### Join multiple variables in one
```python
from auror_core.v2.job import Command
from auror_core.v2.params import Env
from auror_core import Project
env = Env(
TESTE="env test",
VARIABLE="env variable"
)
params = Params(
teste1="my test",
variable="my variable"
)
one_param = ParamsJoin("params_strange_name", ",") ## param name and separator
com = Command()\
.with_command("bash echo ${params_strange_name}")
## it will print: my test,my variable,env test,env variable
## THERE IS NO ORDER GUARANTEE, JUST Python 3.6 >
Project("folder_to_generate_files", com)\
.is_v2()\
.with_params(one_param(params, env))\
.write()
```
### Load jobs from YAML File (just for V2)
**You can find some YAML File examples on [Azkaban Flow Documentation](https://github.com/azkaban/azkaban/wiki/Azkaban-Flow-2.0-Design#flow-yaml-file)**
```python
from auror_core.v2.loader import Loader
loader = Loader('/path/to/file/flow.yaml')
jobs = loader.as_job_objects()
```
Or you can export these jobs as a Python File
```python
from auror_core.v2.loader import Loader
loader = Loader('/path/to/file/flow.yaml')
jobs = loader.as_python_file('/path/to/desired/directory') # will be dumped with 'flow.py' name
```
### Dump memory flows to a Python File (just for V2)
```python
from auror_core.v2.dumper import Dumper
com1 = Job() \
.with_name("commands job 1") \
.with_(command="bash echo 1")
com2 = Job()\
.with_name("commands job 2")\
.with_(command="bash echo 2")
dumper = Dumper('/path/to/desired/directory') # will be dumped with 'flow.py' name
dumper.dump_jobs(com1, com2)
```
## Plugins
Plugins are just extensions from auror_core
There is a cookiecutter for new azkaban jobtypes with Auror template too: https://github.com/globocom/azkaban-jobtype-cookiecutter
We already have email plugin: https://github.com/globocom/azkaban-jobtype-email
## Contribute
For development and contributing, please follow [Contributing Guide](https://github.com/globocom/auror-core/blob/master/CONTRIBUTING.md) and ALWAYS respect the [Code of Conduct](https://github.com/globocom/auror-core/blob/master/CODE_OF_CONDUCT.md)