{"id":13626405,"url":"https://github.com/aws-samples/aws-usage-queries","last_synced_at":"2025-04-16T14:33:30.028Z","repository":{"id":38685020,"uuid":"362393068","full_name":"aws-samples/aws-usage-queries","owner":"aws-samples","description":"This application bootstraps everything needed to query the AWS Cost and Usage reports through Amazon Athena. It also includes reference data and preconfigured SQL queries.","archived":false,"fork":false,"pushed_at":"2024-06-17T06:27:05.000Z","size":991,"stargazers_count":47,"open_issues_count":1,"forks_count":5,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-08-01T22:09:31.889Z","etag":null,"topics":["amazon-athena","cdk","cost","cur-data"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit-0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aws-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-04-28T08:24:47.000Z","updated_at":"2024-07-09T08:20:29.000Z","dependencies_parsed_at":"2024-01-14T08:50:11.175Z","dependency_job_id":null,"html_url":"https://github.com/aws-samples/aws-usage-queries","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Faws-usage-queries","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Faws-usage-queries/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Faws-usage-queries/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Faws-usage-queries/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aws-samples","download_url":"https://codeload.github.com/aws-samples/aws-usage-queries/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223716510,"owners_count":17191057,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-athena","cdk","cost","cur-data"],"created_at":"2024-08-01T21:02:17.508Z","updated_at":"2024-11-08T16:30:37.589Z","avatar_url":"https://github.com/aws-samples.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"# AWS usage queries\n\nIf you want to identify the top AWS usage in your AWS accounts the [AWS Cost and Usage reports](https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html) (CUR) have much of the data required. You can [query the CUR data through Amazon Athena](https://docs.aws.amazon.com/cur/latest/userguide/cur-query-athena.html).\n\nThis repository is an AWS Cloud Development Kit (CDK) template that bootstraps:\n\n- AWS Glue catalog metadata in your account to query the CUR\n- additional reference data you can use to enrich your queries - such as instance type information\n- common queries as AWS Glue views, e.g. EC2 instance vCPU hours by account and month\n\nYou can modify the template to add your own data and queries.\n\n![Amazon Athena view in the AWS Management Console](athena_view.png)\n\n## Architectural Overview\n\n![Architectural Overview](architectural_overview.png)\n\n## Get Started\n\nThis application is available in the AWS Serverless Application Repository. You can deploy it to your account from there:\n\n[![cloudformation-launch-button](https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png)](https://serverlessrepo.aws.amazon.com/#/applications/arn:aws:serverlessrepo:us-east-1:387304072572:applications~aws-usage-queries)\n\n### Compile the CDK template\n\nThe CDK template is written in TypeScript. TypeScript sources must be compiled to JavaScript initially and after each modification. Open a new terminal and keep this terminal open in the background if you like to change the source files. Change the directory to the one where `cdk.json` is and execute:\n\n```bash\nnpm install\nnpm run watch\n```\n\nRead the [CDK developer guide](https://docs.aws.amazon.com/cdk/latest/guide/home.html) for more information.\n\n### Deployment using CDK\n\nCreate or update the application with `cdk deploy`. You must provide parameters used when [creating cost and usage reports](https://docs.aws.amazon.com/cur/latest/userguide/cur-create.html). The CUR data must be stored in Apache Parquet format.\n\n```bash\ncdk deploy \\\n    --parameters CurBucketName=\u003cbucket name\u003e \\\n    --parameters ReportPathPrefix=\u003cpath without leading or trailing slash\u003e \\\n    --parameters ReportName=\u003creport name\u003e \\\n    --databaseName=\u003coptional databasename override, default: aws_usage_queries_database\u003e\n```\n\n### Access the queries\n\nYou can find the queries deployed via the Amazon Athena console. You can choose **Preview** next to each view or run a query like:\n\n```\nSELECT * FROM \u003csee databaseName parameter above\u003e.\"monthly_s3_storage_by_account\" limit 10;\n```\n\n## Cleaning Up\n\nTo avoid incurring charges, delete the AWS CloudFormation stack when you are finished experimenting via `cdk destroy` in the directory where `cdk.json` is:\n\n```bash\ncdk destroy\n```\n\nOr delete the AWS CloudFormation stack manually:\n\n* Sign in to the AWS CloudFormation console and choose your stack.\n* Choose **Delete** to delete all resources, including the bucket that stores the reference data.\n\n## FAQs\n\n### Q: Can I use CUR information stored in an other AWS account?\n\nYes. You can use a bucket policy for [cross-account access in Athena to Amazon S3 buckets](https://docs.aws.amazon.com/athena/latest/ug/cross-account-permissions.html) or you [setup cross-account replication of the report bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-walkthrough-2.html).\n\n### Q: Why is my CUR data not being delivered to the report bucket?\n\nMake sure your Amazon S3 bucket is granting the [permissions for Cost and Usage Reports](https://docs.aws.amazon.com/cur/latest/userguide/cur-s3.html) to PUT the CUR data into the bucket.\n\n### Q: How much do resources in this template cost?\n\nStandard AWS charges apply to the resources you deploy with this template. For the query cost read the [Amazon Athena pricing](https://aws.amazon.com/athena/pricing/) and [AWS Glue pricing](https://aws.amazon.com/glue/pricing/) for details. For the storage and requests to CUR data in your Amazon S3 bucket read the [Amazon S3 Pricing](https://aws.amazon.com/s3/pricing/). When creating the CUR report select *Enable report data integration for Athena*. That way the report data is written with the Apache Parquet columnar format which usually results in better performance and cost optimization for both Amazon S3 and Amazon Athena.\n\n### Q: How can I refresh the instance types\n\nThe instance types come from the EC2 API. Query and write it to a file:\n\n```{bash}\necho \"instance_family,instance_type,clock_speed_in_ghz,vcpu_count\" \\\n    \u003e referenceData/instanceTypes/data.csv\naws ec2 describe-instance-types \\\n    --output text \\\n    --query 'InstanceTypes[*].[InstanceType, ProcessorInfo.SustainedClockSpeedInGhz, VCpuInfo.DefaultVCpus]' \\\n    | sed -E 's/[[:blank:]]+/,/g' \\\n    | sed -E 's/^(([^.]+).*)/\\2,\\1/' \\\n    | sort \u003e\u003e referenceData/instanceTypes/data.csv\n```\n\n### Q: How can I add a new question to this list?\n\nIf you found yourself wishing this set of frequently asked questions had an answer for a particular problem, please [submit a pull request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/). The chances are good that others will also benefit from having the answer listed here.\n\n### Q: How can I contribute?\n\nSee the [Contributing Guidelines](CONTRIBUTING.md) for details.\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis library is licensed under the MIT-0 License. See the LICENSE file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws-samples%2Faws-usage-queries","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faws-samples%2Faws-usage-queries","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws-samples%2Faws-usage-queries/lists"}