{"id":16176141,"url":"https://github.com/piotr-kalanski/airflow-aws-plugins","last_synced_at":"2025-03-19T01:30:35.209Z","repository":{"id":117456699,"uuid":"144408854","full_name":"piotr-kalanski/airflow-aws-plugins","owner":"piotr-kalanski","description":"Airflow plugin with AWS operators","archived":false,"fork":false,"pushed_at":"2020-04-03T21:13:29.000Z","size":30,"stargazers_count":1,"open_issues_count":5,"forks_count":5,"subscribers_count":1,"default_branch":"development","last_synced_at":"2025-03-17T01:21:45.565Z","etag":null,"topics":["airflow","airflow-plugin","aws","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/piotr-kalanski.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-11T18:23:23.000Z","updated_at":"2020-04-15T05:05:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"6424fd6b-5406-4f6a-aa99-fe599d2f89df","html_url":"https://github.com/piotr-kalanski/airflow-aws-plugins","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/piotr-kalanski%2Fairflow-aws-plugins","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/piotr-kalanski%2Fairflow-aws-plugins/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/piotr-kalanski%2Fairflow-aws-plugins/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/piotr-kalanski%2Fairflow-aws-plugins/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/piotr-kalanski","download_url":"https://codeload.github.com/piotr-kalanski/airflow-aws-plugins/tar.gz/refs/heads/development","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244336087,"owners_count":20436765,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-plugin","aws","python"],"created_at":"2024-10-10T04:47:51.749Z","updated_at":"2025-03-19T01:30:35.203Z","avatar_url":"https://github.com/piotr-kalanski.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# airflow-aws-plugin\n\nAirflow plugin with AWS operators\n\n# Table of contents\n\n- [Installation](#installation)\n- [Operators](#operators)\n    - [AWS Lambda](#aws-lambda)\n        - [ExecuteLambdaOperator](#executelambdaoperator)\n    - [AWS Redshift](#aws-redshift)    \n        - [ExecuteRedshiftQueryOperator](#executeredshiftqueryoperator)\n        - [ExecuteCopyToRedshiftOperator](#executecopytoredshiftoperator)            \n\n# Installation\n\nCopy [aws_operators](/aws_operators) directory to *plugins* directory in airflow (default AIRFLOW_HOME/plugins/).\n\n# Operators\n\nList of operators by AWS service:\n\n## AWS Lambda\n\n### ExecuteLambdaOperator\n\nOperator responsible for triggering AWS Lambda function.\n\n*Example:*\n\n```python\nExecuteLambdaOperator(\n    task_id='task_with_execute_lambda_operator',\n    airflow_context_to_lambda_payload=lambda c: {\"date\": c[\"execution_date\"].strftime('%Y-%m-%d')   },\n    additional_payload={\"param1\": \"value1\", \"param2\": 21},\n    lambda_function_name=\"LambdaFunctionName\"\n)\n```\n\nAbove task executes AWS Lambda function `LambdaFunctionName` with payload:\n\n```json\n{\n  \"date\": \"2018-08-01\",\n  \"param1\": \"value1\",\n  \"param2\": 21\n}\n```\nwhere `date` is equal to `execution_date` of airflow dag. This is extracted by `airflow_context_to_lambda_payload` function from airflow context dictionary.\n\n## AWS Redshift\n\n### ExecuteRedshiftQueryOperator\n\nExecute Redshift query.\n\n*Example:*\n\nDROP Redshift table:\n\n```python\nExecuteRedshiftQueryOperator(\n    task_id='drop_table',\n    redshift_conn_id='redshift_dev',\n    query='DROP TABLE IF EXISTS TEST_TABLE'\n)\n```\n\n#### Query depending on execution date\n\nQuery can be constructed based on Airflow context, especially execution date.\n\nExample:\n```python\nExecuteRedshiftQueryOperator(\n    task_id='delete_from_table',\n    redshift_conn_id='redshift_dev',\n    query=lambda c: \"DELETE FROM TABLE TEST_TABLE WHERE MONTH = '{y}-{m}'\".format(y=c[\"execution_date\"].year, m=c[\"execution_date\"].strftime(\"%m\"))\n)\n```\n\n### ExecuteCopyToRedshiftOperator\n\nExecute Redshift COPY command.\n\n*Example 1 - append data:*\n\n```python\nExecuteCopyToRedshiftOperator(\n    task_id='redshift_copy_append',\n    redshift_conn_id='redshift_dev',\n    s3_bucket='bucket',\n    s3_key='key',\n    redshift_schema='public',\n    table='table',\n    iam_role='iam_role',\n    mode='append'\n)\n```\n\n*Example 2 - overwrite table:*\n\n```python\nExecuteCopyToRedshiftOperator(\n    task_id='redshift_copy_overwrite',\n    redshift_conn_id='redshift_dev',\n    s3_bucket='bucket',\n    s3_key='key',\n    redshift_schema='public',\n    table='table',\n    iam_role='iam_role',\n    mode='overwrite',\n    copy_params=['CSV']\n)\n```\n\n### ExecuteUnloadFromRedshiftOperator\n\nExecute Redshift UNLOAD command.\n\n```python\nExecuteUnloadFromRedshiftOperator(\n    task_id='redshift_unload',\n    redshift_conn_id='redshift_dev',\n    select_statement='SELECT * FROM TABLE',\n    s3_bucket='bucket',\n    s3_key='key',\n    iam_role='iam_role',\n    unload_params=[\"DELIMITER AS ';'\", \"GZIP\"]\n)\n```\n\n#### S3 key dependent on airflow context\n\nSource S3 key can be constructed using custom Python function based on airflow context.\n\nExample:\n\n```python\nExecuteCopyToRedshiftOperator(\n    s3_key=lambda context: \"year={y}/month={m}/day={d}/\".format(y=context[\"execution_date\"].year, m=context[\"execution_date\"].strftime(\"%m\"), d=context[\"execution_date\"].strftime(\"%d\"))\n)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpiotr-kalanski%2Fairflow-aws-plugins","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpiotr-kalanski%2Fairflow-aws-plugins","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpiotr-kalanski%2Fairflow-aws-plugins/lists"}