{"id":13819991,"url":"https://github.com/andyfase/CURdashboard","last_synced_at":"2025-05-16T07:32:43.232Z","repository":{"id":57746942,"uuid":"109806864","full_name":"andyfase/CURdashboard","owner":"andyfase","description":"CURdashboard is an automated, extendable and configurable AWS customer usage report analyzer (CUR)","archived":false,"fork":false,"pushed_at":"2020-01-03T22:22:54.000Z","size":23661,"stargazers_count":22,"open_issues_count":3,"forks_count":18,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-19T19:43:47.083Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andyfase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-07T08:24:36.000Z","updated_at":"2024-09-09T21:47:07.000Z","dependencies_parsed_at":"2022-09-02T23:20:46.818Z","dependency_job_id":null,"html_url":"https://github.com/andyfase/CURdashboard","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyfase%2FCURdashboard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyfase%2FCURdashboard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyfase%2FCURdashboard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyfase%2FCURdashboard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andyfase","download_url":"https://codeload.github.com/andyfase/CURdashboard/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254488445,"owners_count":22079429,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T08:00:56.530Z","updated_at":"2025-05-16T07:32:40.232Z","avatar_url":"https://github.com/andyfase.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# CURdashboard\r\r\rCURdashboard is an automated, extendable and configurable AWS customer usage report analyzer (CUR). \r\rUse it to:\r\r* Visualize your AWS costs via Cloudwatch dashboards \r* Alert and react to spend changes within a few hours\r* Build automation and alarming based on changes or cost thresholds.\r* Gain insights into your AWS costs using simple SQL\n\n![DBRdashboard Screenshot](https://raw.githubusercontent.com/andyfase/awsDBRanalysis/master/dbr_dashboard.png)\n\r\rCURdashboard automatically converts and queries the CUR. It produces cost metrics which are piped to AWS Cloudwatch to allow for dashboards to be produced and alarms/automation to be developed.\r\rMetrics are generated by providing SQL queries which are executed using AWS Athena. A standard set of queries are provided, however the solution is designed to easly allow futher queries to be configured, so that metrics can be produced based on your needs and requirements.\r\rCURDashboard also maintains a set of AWS Athena tables for you - to query your CUR as you wish. A table per month is created and kept upto date as new billing data is produced\r\r## How does this work?\r\rAWS publishes [customer usage records](http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-reports.html#other-reports) periodically during the day. These reports contain very detailed line-by-line billing details for every AWS charge related to your account.\r\rCURdashboard sets up a AWS Autoscale Group configured to spin up a EC2 instance every N hours. This instance then bootstrap itself, converts the CSV based CUR reports into [parquet format](https://parquet.apache.org/) and re-uploads these converted files to S3. It then utilizes AWS Athena and standard SQL to create CUR tables within Athena and query specific billing metrics within them. The results of the queries are then reported to AWS Cloudwatch as custom metrics. \r\rOnce the metrics are in Cloudwatch, it is then very easy to:\r\r* Graph metrics and create a billing Cloudwatch dashboard customized to your exact requirements. \r* Produce alarms based on CUR metrics, which can then trigger alerting or automation via SNS. \r\rIn addition to querying the customer usage records. CURdashboard also queries any [reserved instances](https://aws.amazon.com/ec2/pricing/reserved-instances/) on the account and then correlates them against actual usage to generate RI utilization metrics. \r\r## How much will this cost?\r\rCURdashboard utilizes a number of cost-saving measures to minimize its cost. \r\rFor compute, EC2 spot instances are utilized. The instance will self-terminate after a few minutes of processing - hence will only be charged for the total time of processing, due to EC2 per-second billing.\r\rAWS Athena charges on a per query and per data scanned basis. Parquet files greatly reduce the quantity of data that AWS Athena need to process, hence the costs of each query is reduced.\r\rEach query/metric can be enabled or disabled so that only the metrics you need are ingested into Cloudwatch. \r\rOverall costs will vary depending on the size of the Customer Usage Reports, the type of EC2 instance required and the number of metrics ingested into Cloudwatch. \r\rIt is expected compute, storage and query costs will be less that $1 per month. \r\r[Insert Cloudwatch costs here]\r\r## Setup\r\rSetup of CURdashboard should take ~15 minutes.\r\r### Step 1\r\rIf you have not already, turn on customer usage records in your AWS account, follow the instructions [here](http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-reports.html#turnonreports)\r\rPlease allow upto 24 hours for the CUR report to be generated and pushed into the configured S3 bucket before processing to step #2.\r\r### Step 2\r\rFork this GIT repo on Github.\r\rThe EC2 instance itself will bootstrap itself by cloning a configurable GIT repo and then run scripts and custom binaries to generate and upload the custom metrics. \r\rThis custom binary utilizes a configuration file which will need to be edited to enable/disable certain functionality and customize the metrics and queries that are run.\r\rTherefore, forking this repo allow you to commit configuration modifications which will automatically come into effect the next time the EC2 instance spins up.\r\r### Step 3\r\rReview file `analyzeCUR.config` to ensure the Metrics that are configured are applicable to your use-case. See below for the details on the configuration file format.\r\rNote: Each metric can be configured to send either hourly, daily (or both) dimensions. Thus, dashboards can be generated showing costs per hour or per day. \r\r### Step 4\r\rUsing cloudformation bring up both stacks that are within the `cf` directory in your newly forked repo.\r\rThe network stack creates exports which are then referenced by the app stack. To be able to keep multiple stacks operational a `ResourcePrefix` is used which is pre-pended to the exports. The `ResourcePrefix` can be any text string **but must be identical across both stacks**.\r\r`cur_network.yaml` is a CF template that will setup the VPC and general networking required. It is recomended to use a small CIDR block, as CURdownload will only ever spin-up a single EC2 instance.\r\r`cur_app.yaml` is a CF template that sets up the required IAM EC2 role autoscale group with a configured time-based scale up policy. It is recommended to configure the schedule parameter to ~ 4-6 hours, as this is roughly how often the CUR report is updated by AWS.\r\rEnsure you specify the GIT clone URL for your own repo. This will allow you to push configuration (or code) changes which will then be automatically picked up for the next time the EC2 instance spins up.\r\r### Step 5\r\rOnce setup you will need to wait for the first auto-scale spin-up to occur before metrics will be pushed into Cloudwatch.\r\rTo accelerate this up, you could also manually set the `desired` capacity of the ASG to `1` so that an instance will immediately spins up. It’s safe to leave this set to `1` as the instance will update the desired value back to zero (hence self-terminate) after it has finished processing.\r\rOnce the instance has spun up it will bootstrap itself and run the code to generate the custom metrics. These should start to appear in Cloudwatch, typically these appear within 15 minutes of instance start\r\rUsing the custom metrics, generate graphs that you would like and start creating your very own cost dashboard. Instructions on creating a Cloudwatch dashboard can be viewed [here](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html)\r\r\r\r## Configuration\r\rConfiguration for CURdashboard is performed by editing the configuration file `analyzeCUR.config`.\r\rThe configuration file is in the [TOML](www.toml.org?) format and has a number of sections which are described below.\r\r\r### General Configuration options\r\rThese options are held within the `[general]` TOML section\r\rOption Name     | Description                                   | Default Value\r--------------- | --------------------------------------------- | -------------\r`namespace`     | The Cloudwatch namespace used for all metrics | `CUR`\r\r### Athena Configuration options\r\rThese options are held within the `[athena]` TOML section\r\rption Name     | Description                                   | Default Value\r--------------- | --------------------------------------------- | -------------\r`database_name`     | The database to create within Athena  | `CUR`\r`table_prefix`     | The prefix used when creating the monthly CUR Athena tables. The current date will be appended to this in the format `_MMYYYY`  | `autocur`\r\r### RI Configuration options\r\rTBD\r\r### Metric Configuration\r\rEach metric is held within a `TOML` array in the configuration file. This array is iterated over to query Athena and then send the results as metrics to Cloudwatch.\r\rTo add new metrics simply copy-and-paste an existing `[[metric]]` entry and then modify the various attributes, which are\r\rMetric Attribute  |  Description\r----------------- | ------------\r`enabled`         | Enables / Disables the metric\r`hourly`          | Enables / Disables hourly metric reporting\r`daily`           | Enables / Disables daily metric reporting\r`type`            | Reserved for future use. value of `dimension-per-row` is only accepted value currently\r`cwName` | The metric name that will be sent to Cloudwatch\r`cwDimension` | The dimension name that will be sent to Cloudwatch (the value of the dimension will be taken from the \"dimension\" row value (see below)\r`cwType` | The cloudwatch metric type that will be sent to cloudwatch\r`sql` | The SQL that will be executed on the Athena CUR table to fetch the metric information (see below)\r\r### Athena Metric SQL\r\rEach metric that you wish to display on the dashboard is obtained by querying the CUR Athena table. Each row that is returned is considered a new metric value. The `date` column is used as the time-series \"divider\" and is converted to a timestamp which is sent for this row. \r\rDefault useful metrics are pre-configured within the original configuration file. These can be disabled if required or even completely removed. New metrics can be added as described above. \r\rCURdashboard uses a substitution parameter for the date column `**INTERVAL**`, this is used so that the same query can retrieve costs split by hour and day. It is recommended that the date column SQL always be:\r\r `substr(\"lineitem/usagestartdate\",1,**INTERVAL**) as date`\r\rEach row in the query results **MUST** contain the following aliased columns\r\rColumn Name | Description\r----------- | -----------\r`date`      | the timeperiod for the metric. Typically the hour (`format YYYY-MM-DD HH`) or day \r`value`     | The metric value for this time period (normally a count(*) in SQL\r`dimension` | The dimension value that will be sent for this row. \r\rFor example, if a query returns a row with \r\r`date` | `dimension` | `value`\r------ | ----------- | ------\r2017-02-01 17 | m3.xlarge | 50\r\rThen a custom metric (named using the `cwName` parameter) will be sent to Cloudwatch as follows:\r\r* The **timestamp** will be set to `2017-02-01 17:00:00`\r* The **dimension name** will be set to the parameter value `cwDimension`\r* The **dimension value** will be set to `m3.xlarge`\r* The **value** will be set to `50`\r\rEvery row returned will send a metric using `put-metric-data` \r\rNote. Athena uses Presto under-the-hood. Hence all Presto SQL functions are available for you to utilize. These can be found [here](https://prestodb.io/docs/current/functions.html).\r\r### Limitations\r\r* Only RI's under the \"running\" account are fetched and used to generate % RI utilization. If RI's exist under linked accounts they are not currently included and will cause incorrect results. Temporary work around for this is to move all RI's into the payer account if possible.\r\r\r\r\r\r","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandyfase%2FCURdashboard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandyfase%2FCURdashboard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandyfase%2FCURdashboard/lists"}