https://github.com/rssanders3/airflow-zip-operator-plugin
A plugin to Apache Airflow to allow you to run Zip and UnZip commands as an Operator
https://github.com/rssanders3/airflow-zip-operator-plugin
airflow operator unzip zip
Last synced: over 1 year ago
JSON representation
A plugin to Apache Airflow to allow you to run Zip and UnZip commands as an Operator
- Host: GitHub
- URL: https://github.com/rssanders3/airflow-zip-operator-plugin
- Owner: rssanders3
- License: apache-2.0
- Created: 2017-03-22T20:58:24.000Z (about 9 years ago)
- Default Branch: master
- Last Pushed: 2023-07-26T13:00:16.000Z (almost 3 years ago)
- Last Synced: 2024-10-24T16:50:54.423Z (over 1 year ago)
- Topics: airflow, operator, unzip, zip
- Language: Python
- Size: 8.79 KB
- Stars: 12
- Watchers: 1
- Forks: 10
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# airflow-zip-operator-plugin
## Description
A plugin to Apache Airflow (Documentation: https://pythonhosted.org/airflow/, Source Code: https://github.com/apache/incubator-airflow) to allow you to run Zip and UnZip commands as an Operator from Workflows
## TODO List
* Print out metrics about zip file (compression, size, etc)
* Test extensively
## How do Deploy
1. Copy the zip_operator_plugin.py file into the Airflow Plugins directory
* The Airflow Plugins Directory is defined in the airflow.cfg file as the variable "plugins_folder"
* The Airflow Plugins Directory is, by default, ${AIRFLOW_HOME}/plugins
* You may have to create the Airflow Plugins Directory folder as it is not created by default
* quick way of doing this:
$ cd {AIRFLOW_PLUGINS_FOLDER}
$ wget https://raw.githubusercontent.com/rssanders3/airflow-zip-operator-plugin/master/zip_operator_plugin.py
2. Restart the Airflow Services
3. Create or Deploy DAGs which utilize the Operator
4. Your done!
## ZipOperator
### Operator Definition
class **airflow.operators.ZipOperator**(input_file_path, output_file_path, *args, **kwargs)
Bases: **airflow.operators.BaseOperator**
An operator which takes in a path to a file and zips the contents to a location you define.
Parameters:
* **path_to_file_to_zip** (string) - Full path to the file you want to Zip
* **path_to_save_zip** (string) - Full path to where you want to save the Zip file
### Example
```
from airflow.operators import ZipOperator
zip_task = ZipOperator(
task_id='zip_task',
path_to_file_to_zip="/path/to/file",
path_to_save_zip="/path/to/file.zip",
dag=dag)
```
## UnzipOperator
### Operator Definition
class **airflow.operators.UnzipOperator**(input_file_path, output_file_path, *args, **kwargs)
Bases: **airflow.operators.BaseOperator**
An operator which takes in a path to a zip file and unzips the contents to a location you define.
Parameters:
* **path_to_zip_file** (string) - Full path to the zip file you want to Unzip
* **path_to_unzip_contents** (string) - Full path to where you want to save the contents of the Zip file you're Unzipping
### Example
```
from airflow.operators import UnzipOperator
unzip_task = UnzipOperator(
task_id='unzip_task',
path_to_zip_file="/path/to/file.zip",
path_to_unzip_contents="/path/to/unzip/to/",
dag=dag)
```