Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/bahchis/airflow-cookbook
Airflow workflow management platform chef cookbook.
https://github.com/bahchis/airflow-cookbook
airflow airflow-cookbook chef-cookbook
Last synced: 9 days ago
JSON representation
Airflow workflow management platform chef cookbook.
- Host: GitHub
- URL: https://github.com/bahchis/airflow-cookbook
- Owner: bahchis
- License: apache-2.0
- Created: 2015-10-12T07:31:38.000Z (about 9 years ago)
- Default Branch: master
- Last Pushed: 2019-07-11T08:48:40.000Z (over 5 years ago)
- Last Synced: 2024-08-01T18:24:49.852Z (3 months ago)
- Topics: airflow, airflow-cookbook, chef-cookbook
- Language: Ruby
- Size: 84 KB
- Stars: 68
- Watchers: 6
- Forks: 49
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
- awesome-apache-airflow-br - airflow-cookbook - Receitas de como fazer o deploy do Airflow (Soluções de deployment do Airflow)
- awesome-apache-airflow - airflow-cookbook - Chef cookbook for deploying Airflow. (Airflow deployment solutions)
README
# Airflow Chef Cookbook
Installs and configures Airflow workflow management platform. More information about Airflow can be found here: https://github.com/airbnb/airflow
## Supported Platforms
Ubuntu (Tested on 14.04, 16.04).
CentOS (Tested on 7.2).
## Limitations
The Airflow **all** and **oracle** packages are not supported, this is due the Oracle package having dependencies which cannot be automatically installed. I will look how to solve this and add support for those packages at later stage.
## Contributing
Please follow instructions in the [contributing doc](CONTRIBUTING.md).
## Usage
- Use the relevant cookbooks to install and configure Airflow.
- Use environment variable in /etc/default/airflow (for Ubuntu) or /etc/sysconfig/airflow (for CentOS) to configure Airflow during the startup process. (More info about Airflow environment variables at: [Setting Configuration Options](https://pythonhosted.org/airflow/configuration.html#setting-configuration-options))
- Make sure to run **airflow initdb** as part of your startup script.
## Recipes
- default - Executes other recipes.
- directories - Creates required directories.
- user - Creates OS user and group.
- packages - Installs OS and pip packages.
- config - Handles airflow.cfg
- services - Creates services env file.
- webserver - Configures service for webserver.
- scheduler - Configures service for scheduler.
- worker - Configures service for worker.
- flower - Configures service for flower.
- kerberos - Configures service for kerberos.
- packages - Installs Airflow and supporting packages.
## Attributes
##### User config
- ["airflow"]["airflow_package"] - Airflow package name, defaults to 'apache-airflow'. Use 'airflow' for installing version 1.8.0 or lower.
- ["airflow"]["version"] - The version of airflow to install, defaults to latest (nil).
- ["airflow"]["user"] - The user Airflow is executed with and owner of all related folders.
- ["airflow"]["group"] - Airflow user group.
- ["airflow"]["user_uid"] - Airflow user uid
- ["airflow"]["group_gid"] - Airflow group gid
- ["airflow"]["user_home_directory"] - Airflow user home directory.
- ["airflow"]["shell"] - Airflow user shell.
##### General config
- ["airflow"]["directories_mode"] - The permissions airflow and user directories are created.
- ["airflow"]["config_file_mode"] - The permissions airflow.cfg is created.
- ["airflow"]["bin_path"] - Path to the bin folder, default is based on platform.
- ["airflow"]["run_path"] - Pid files base directory
- ["airflow"]["is_upstart"] - Should upstart be used for services, determined automatiaclly.
- ["airflow"]["init_system"] - The init system to use when configuring services, only upstart or systemd are supported and defaults based on ["airflow"]["is_upstart"] value.
- ["airflow"]["env_path"] - The path to services env file, determined automatiaclly.
##### Python config
- ["airflow"]["python_runtime"] = Python runtime as used by [poise-python cookbook](https://github.com/poise/poise-python#quick-start).
- ["airflow"]["python_version"] = Python version to install as used by poise-python cookbook.
- ["airflow"]["pip_version"] = Pip version to install (true - installs latest) as used by poise-python cookbook.
##### Package config
- default['airflow']['packages'] - The Python packages to install for Airflow.
- default['airflow']['dependencies'] - The dependencies of the packages listed in default['airflow']['packages']. These are OS packages, not Python packages.
##### airflow.cfg
This cookbook enables to configure any airflow.cfg paramters dynamically by using attributes structure like (see the attributes file for [airflow.cfg examples](attributes/default.rb)):
["airflow"]["config"]["CONFIG_SECTION"]["CONFIG_ENTRY"]
## License
Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0)
## Author
[Sergey Bahchissaraitsev](http://www.bahchis.com/about/)