https://github.com/powerdatahub/airflow_redshift_plugin
Move data from Amazon Redshift to other sources like Amazon S3, Apache Druid and more
https://github.com/powerdatahub/airflow_redshift_plugin
airflow airflow-plugin apache-airflow apache-druid hacktoberfest redshift
Last synced: 4 months ago
JSON representation
Move data from Amazon Redshift to other sources like Amazon S3, Apache Druid and more
- Host: GitHub
- URL: https://github.com/powerdatahub/airflow_redshift_plugin
- Owner: PowerDataHub
- License: apache-2.0
- Created: 2019-07-14T19:20:16.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2019-07-15T01:30:43.000Z (almost 6 years ago)
- Last Synced: 2025-02-26T23:55:44.100Z (4 months ago)
- Topics: airflow, airflow-plugin, apache-airflow, apache-druid, hacktoberfest, redshift
- Language: Python
- Homepage:
- Size: 23.4 KB
- Stars: 2
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://powerdatahub.com/?ref=repo_aws_airflow) [](https://github.com/apache/airflow/)
# Airflow Plugin - Redshift
Move data from [Amazon Redshift](https://aws.amazon.com/pt/redshift/) to other sources like Amazon S3, Apache Druid and more
## Operators
### RedshiftToDruidOperator
Executes an UNLOAD command to s3 and load into Apache Druid
```
:param s3_bucket: reference to a specific S3 bucket
:type s3_bucket: str
:param s3_key: reference to a specific S3 key
:type s3_key: str
:param schema: reference to a specific schema in redshift database
:type schema: str
:param table: reference to a specific table in redshift database
:type table: str
:param sql: Custom SQL statement to execute
:type sql: str
:param druid_ingest_spec: druid ingestion json spec
:type druid_ingest_spec: json
:param unload_options: reference to a list of UNLOAD options
:type unload_options: list
:param include_header: Should include headers in the final file?
:type include_header: bool
:param autocommit: if True perform autocommit
:type autocommit: bool
:param aws_conn_id: reference to a specific S3 connection
:type aws_conn_id: str
:param redshift_conn_id: reference to a specific redshift database
:type redshift_conn_id: str
:param druid_conn_id: reference to a specific Druid overlord connection
:type druid_conn_id: str
```
### RedshiftToS3CustomOperador
Executes an UNLOAD command to s3 as a CSV with headers
```
:param schema: reference to a specific schema in redshift database
:type schema: str
:param table: reference to a specific table in redshift database
:type table: str
:param s3_bucket: reference to a specific S3 bucket
:type s3_bucket: str
:param s3_key: reference to a specific S3 key
:type s3_key: str
:param sql: Custom SQL statement to execute
:type sql: str
:param redshift_conn_id: reference to a specific redshift database
:type redshift_conn_id: str
:param aws_conn_id: reference to a specific S3 connection
:type aws_conn_id: str
:param verify: Whether or not to verify SSL certificates for S3 connection.
By default SSL certificates are verified.
You can provide the following values:
- ``False``: do not validate SSL certificates. SSL will still be used
(unless use_ssl is False), but SSL certificates will not be
verified.
- ``path/to/cert/bundle.pem``: A filename of the CA cert bundle to uses.
You can specify this argument if you want to use a different
CA cert bundle than the one used by botocore.
:type verify: bool or str
:param unload_options: reference to a list of UNLOAD options
:type unload_options: list
```
### S3ToRedshiftOperator
Executes an COPY command to load a S3 file into Redshift
---
[](https://forthebadge.com) [](https://forthebadge.com) [](https://forthebadge.com)