{"id":19607544,"url":"https://github.com/andreax79/airflow-aws-cost-explorer","last_synced_at":"2025-04-27T20:32:26.225Z","repository":{"id":57409281,"uuid":"213654074","full_name":"andreax79/airflow-aws-cost-explorer","owner":"andreax79","description":"Apache Airflow Operator exporting AWS Cost Explorer data to local file or S3","archived":false,"fork":false,"pushed_at":"2019-10-23T09:18:09.000Z","size":39,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-15T12:06:49.925Z","etag":null,"topics":["airflow","airflow-operator","apache-airflow","aws","aws-","aws-cost-explorer","python"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andreax79.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.txt","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-08T13:47:05.000Z","updated_at":"2022-01-31T21:09:19.000Z","dependencies_parsed_at":"2022-09-04T22:02:18.063Z","dependency_job_id":null,"html_url":"https://github.com/andreax79/airflow-aws-cost-explorer","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreax79%2Fairflow-aws-cost-explorer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreax79%2Fairflow-aws-cost-explorer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreax79%2Fairflow-aws-cost-explorer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreax79%2Fairflow-aws-cost-explorer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andreax79","download_url":"https://codeload.github.com/andreax79/airflow-aws-cost-explorer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251204548,"owners_count":21552239,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-operator","apache-airflow","aws","aws-","aws-cost-explorer","python"],"created_at":"2024-11-11T10:11:18.361Z","updated_at":"2025-04-27T20:32:25.765Z","avatar_url":"https://github.com/andreax79.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Airflow AWS Cost Explorer Plugin\nA plugin for [Apache Airflow](https://github.com/apache/airflow) that allows\nyou to export [AWS Cost Explorer](https://aws.amazon.com/aws-cost-management/aws-cost-explorer/)\nas [S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html) metrics to\nlocal file or S3 in Parquet, JSON, or CSV format.\n\n### System Requirements\n\n* Airflow Versions\n    * 1.10.3 or newer\n* pyarrow or fastparquet (optional, for writing Parquet files)\n\n### Deployment Instructions\n\n1. Install the plugin\n\n    pip install airflow-aws-cost-explorer\n\n2. Optional for writing Parquet files - Install pyarrow or fastparquet\n\n    pip install pyarrow\n\n    or\n\n    pip install fastparquet\n\n2. Restart the Airflow Web Server\n\n3. Configure the AWS connection (Conn type = 'aws')\n\n4. Optional for S3 - Configure the S3 connection (Conn type = 's3')\n\n## Operators\n\n### AWSCostExplorerToS3Operator\n```AWS Cost Explorer to S3 Operator\n\n    :param day:             Date to be exported as string in YYYY-MM-DD format or date/datetime instance (default: yesterday)\n    :type day:              str, date or datetime\n    :param aws_conn_id:     Cost Explorer AWS connection id (default: aws_default)\n    :type aws_conn_id:      str\n    :param region_name:     Cost Explorer AWS Region\n    :type region_name:      str\n    :param s3_conn_id:      Destination S3 connection id (default: s3_default)\n    :type s3_conn_id:       str\n    :param s3_bucket:       Destination S3 bucket\n    :type s3_bucket:        str\n    :param s3_key:          Destination S3 key\n    :type s3_key:           str\n    :param file_format:     Destination file format (parquet, json or csv default: parquet)\n    :type file_format:      str or FileFormat\n    :param metrics:         Metrics (default: UnblendedCost, BlendedCost)\n    :type metrics:          list\n\n```\n\n### AWSCostExplorerToLocalFileOperator\n```AWS Cost Explorer to local file Operator\n\n    :param day:             Date to be exported as string in YYYY-MM-DD format or date/datetime instance (default: yesterday)\n    :type day:              str, date or datetime\n    :param aws_conn_id:     Cost Explorer AWS connection id (default: aws_default)\n    :type aws_conn_id:      str\n    :param region_name:     Cost Explorer AWS Region\n    :type region_name:      str\n    :param destination:     Destination file complete path\n    :type destination:      str\n    :param file_format:     Destination file format (parquet, json or csv default: parquet)\n    :type file_format:      str or FileFormat\n    :param metrics:         Metrics (default: UnblendedCost, BlendedCost)\n    :type metrics:          list\n\n```\n\n### AWSBucketSizeToS3Operator\n```AWS Bucket Size to S3 Operator\n\n    :param day:             Date to be exported as string in YYYY-MM-DD format or date/datetime instance (default: yesterday)\n    :type day:              str, date or datetime\n    :param aws_conn_id:     Cost Explorer AWS connection id (default: aws_default)\n    :type aws_conn_id:      str\n    :param region_name:     Cost Explorer AWS Region\n    :type region_name:      str\n    :param s3_conn_id:      Destination S3 connection id (default: s3_default)\n    :type s3_conn_id:       str\n    :param s3_bucket:       Destination S3 bucket\n    :type s3_bucket:        str\n    :param s3_key:          Destination S3 key\n    :type s3_key:           str\n    :param file_format:     Destination file format (parquet, json or csv default: parquet)\n    :type file_format:      str or FileFormat\n    :param metrics:         Metrics (default: bucket_size, number_of_objects)\n    :type metrics:          list\n\n```\n\n### AWSBucketSizeToLocalFileOperator\n```AWS Bucket Size to local file Operator\n\n    :param day:             Date to be exported as string in YYYY-MM-DD format or date/datetime instance (default: yesterday)\n    :type day:              str, date or datetime\n    :param aws_conn_id:     Cost Explorer AWS connection id (default: aws_default)\n    :type aws_conn_id:      str\n    :param region_name:     Cost Explorer AWS Region\n    :type region_name:      str\n    :param destination:     Destination file complete path\n    :type destination:      str\n    :param file_format:     Destination file format (parquet, json or csv default: parquet)\n    :type file_format:      str or FileFormat\n    :param metrics:         Metrics (default: bucket_size, number_of_objects)\n    :type metrics:          list\n\n```\n\n### Example\n\n```\n    #!/usr/bin/env python\n    import airflow\n    from airflow import DAG\n    from airflow_aws_cost_explorer import AWSCostExplorerToLocalFileOperator\n    from datetime import timedelta\n\n    default_args = {\n        'owner': 'airflow',\n        'depends_on_past': False,\n        'start_date': airflow.utils.dates.days_ago(1),\n        'email': ['airflow@example.com'],\n        'email_on_failure': False,\n        'email_on_retry': False,\n        'retries': 1,\n        'retry_delay': timedelta(minutes=30)\n    }\n\n    dag = DAG('cost_explorer',\n        default_args=default_args,\n        schedule_interval=None,\n        concurrency=1,\n        max_active_runs=1,\n        catchup=False\n    )\n\n    aws_cost_explorer_to_file = AWSCostExplorerToLocalFileOperator(\n        task_id='aws_cost_explorer_to_file',\n        day='{{ yesterday_ds }}',\n        destination='/tmp/{{ yesterday_ds }}.parquet',\n        file_format='parquet',\n        dag=dag)\n\n    if __name__ == \"__main__\":\n        dag.cli()\n```\n\n### Links\n\n* Apache Airflow - https://github.com/apache/airflow\n* Apache Arrow - https://github.com/apache/arrow\n* fastparquet - https://github.com/dask/fastparquet\n* AWS Cost Explorer - https://aws.amazon.com/aws-cost-management/aws-cost-explorer/ [API Reference](https://docs.aws.amazon.com/aws-cost-management/latest/APIReference/API_GetCostAndUsage.html)\n* S3 CloudWatch Metrics - https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreax79%2Fairflow-aws-cost-explorer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandreax79%2Fairflow-aws-cost-explorer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreax79%2Fairflow-aws-cost-explorer/lists"}