{"id":22621978,"url":"https://github.com/servian/aws-redshift-smart-pause-and-resume","last_synced_at":"2025-08-28T08:17:49.595Z","repository":{"id":40839740,"uuid":"253357778","full_name":"servian/aws-redshift-smart-pause-and-resume","owner":"servian","description":"Open source application to automatically resume and pause Redshift using AWS Lambda, CloudWatch Metrics and Events, Forecast and Step Functions.","archived":false,"fork":false,"pushed_at":"2022-08-31T00:54:31.000Z","size":610,"stargazers_count":4,"open_issues_count":2,"forks_count":3,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-11T16:59:54.326Z","etag":null,"topics":["aws","forecast","machine-learning","redshift","serverless"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/servian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-06T00:10:37.000Z","updated_at":"2023-11-17T07:41:15.000Z","dependencies_parsed_at":"2023-01-17T01:45:43.139Z","dependency_job_id":null,"html_url":"https://github.com/servian/aws-redshift-smart-pause-and-resume","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/servian/aws-redshift-smart-pause-and-resume","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/servian%2Faws-redshift-smart-pause-and-resume","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/servian%2Faws-redshift-smart-pause-and-resume/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/servian%2Faws-redshift-smart-pause-and-resume/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/servian%2Faws-redshift-smart-pause-and-resume/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/servian","download_url":"https://codeload.github.com/servian/aws-redshift-smart-pause-and-resume/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/servian%2Faws-redshift-smart-pause-and-resume/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272468463,"owners_count":24939728,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","forecast","machine-learning","redshift","serverless"],"created_at":"2024-12-08T23:12:40.937Z","updated_at":"2025-08-28T08:17:49.552Z","avatar_url":"https://github.com/servian.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Redshift Smart Pause and Resume\n\n![Language](https://img.shields.io/github/languages/top/servian/aws-auto-remediate.svg) [![serverless](http://public.serverless.com/badges/v3.svg)](http://www.serverless.com)\n\nOpen source tool to automatically pause and resume Redshift (single and multi-node) clusters using AWS Lambda, CloudWatch Metrics and Events, Amazon Forecast and Step Functions.\n\n## About\n\n### Resuming and Pausing Redshift using Cluster CPU utilisation Metrics from CloudWatch\n\nCPU utilisation data from an existing Redshift data warehouse is scraped from Cloudwatch metrics. In particular, the metric is the average CPU utilisation at 15 minute intervals by default (value is configurable: recommended values are 5, 15, 30, 60 minute intervals). The data scraped is then used to train an Amazon forecast model, and the resulting forecast predictions are used to determine when to pausea and resume a Redshift data warehouse cluster. \n\nA threshold value is set and used determine when to pause and resume a Redshift cluster. To illustrate, if a threshold value of 5% (i.e., 5% CPU utilisation) is set, the Reshift cluster will be scheduled to resume around a timestamp when CPU utilisation is forecasted to be over the threshold value. On the other hand, given the same threshold value, a Reshift cluster will be scheduled pause around a timestamp when CPU utilisation is forecasted to be under the threshold value. \n\nAn example of this is showcased below. For this example, given the forecasted CPU utilisation values:\n\n* The Redshift cluster will be scheduled to resume on 7:45 \n* The Redshift cluster will be scheduled to pause on 21:15 \n* A buffer of 30 minutes is subtracted and added from the actual timestamps observed to give ample time for the Redshift cluster to resume and pause, respectively.\n\n![schedule-example](images/schedule-example.svg)\n\n### AWS Serverless Architecture \n\nThis tool in a nutshell consists to 2 step functions: (1) the [Train Forecast Model Step Function](#train-forecast-model-step-function) and (2) the [Generate Forecasts Step Function](#generate-forecasts-step-function). Both of these step functions are executed using Lambda functions, and these Lambda functions are triggered with scheduled Cloudwatch Events. Events are scheduled based on the timezone specified when deploying the tool. \n\n![serverless-architecture](images/serverless-architecture.svg)\n\n### Train Forecast Model Step Function\n\nThe following step function consists of a number of steps aimed to produce an Amazon forecast model predicting Redshift CPU utilisation. This step function can be scheduled to run more frequently depending on how often Redshift utilisation activity pattern changes. If AutoML is enabled the most appropriate forecast model will be fitted to the provided dataset. By default this step function is scheduled to run on the first day of each month at 9:00 (Time based on the provided timezone). \n\n### Generate Forecasts Step Function\n\nThe following step function consists of a number of steps aimed to produce Amazon forecast predictions using the resulting model trained from the [Train Forecast Model Step Function](#train-forecast-model-step-function). This step function is scheduled to run daily. In particular, Redshift metrics data from the previous day will be scraped and used alongside existing data to generate Redshift CPU utilisation forecasts for the following (current) day. Forecasts are then used to determine when to pause and resume the Redshift cluster. By default this step function is scheduled to run everyday at 12:05 midnight (Time is based on the provided timezone). \n\n## Setup\n \n### Deploy \n\n1. Install Serverless Framework\n\n```bash\nnpm install serverless\n```\n\n2. Install AWS CLI \n\n```bash\npip3 install awscli \n```\n\nConfigure AWS CLI following instructions found [here](#https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration). Ensure that the user configured has the appropriate IAM permissions to create Lambda Functions, S3 Buckets, IAM Roles, Step functions, Amazon Forecast resources and CloudFormation Stacks.\n\n3. Install Redshift Smart Pause and Resume\n\n```bash\nserverless create --template-url https://github.com/servian/aws-redshift-smart-pause-and-resume --path aws-redshift-smart-pause-and-resume\n```\n\n4. Change to Redshift Smart Pause and Resume directory\n\n```bash\ncd aws-redshift-smart-pause-and-resume \n```\n\n5. Install Serverless Plugins\n\n```bash\nserverless plugin install --name serverless-python-requirements\nserverless plugin install --name serverless-iam-roles-per-function\nserverless plugin install --name serverless-pseudo-parameters\nserverless plugin install --name serverless-local-schedule\n```\n\n6. Deploy service to AWS Account. The option `redshiftclusterid` is required and need to be specified upon deploying the tool. (See [Deployment Options](#deployment-options) below for more details on the other options provided).\n\n* **`NOTE:`** If deploying the tool to schedule another Redshift cluster, ensure that a different value is set for the option `servicename`. The value for this option is set to `smart-sched` by default. Possible value is `smart-sched-01`.\n\n```bash\nserverless deploy \\\n [--region \u003cAWS region\u003e] \\\n [--aws-profile \u003cAWS CLI profile\u003e] \\\n [--redshiftclusterid \u003cAWS redshift cluster id\u003e]\n [--stage \u003cdeployment environment\u003e] \\\n [--servicename \u003ctool service/stack name\u003e] # default value is smart-sched\n```\n\n7. After deploying the tool, run the following command to initially scrape for data and train the forecast model. (See [Scraping and Training Forecast Model After Deployment](#Scraping-and-Training-Forecast-Model-After-Deployment) below for more details on the options provided).\n\n* **`NOTE:`** When deploying the tool, if a value for the option `servicename` was specified and if this value is different from the default value `smart-sched`, be sure to provide the exact same value for `cfnstackname`. \n\n```bash\npython3 local_scrape_and_train.py \n[--awsprofile \u003cvalue\u003e] \\\n[--numdaystoscrape \u003cvalue\u003e] \\\n[--cfnstackname  \u003cvalue\u003e] \\ # value must be consitent with servicename from Deploy Step 6\n[--stage  \u003cvalue\u003e] # value must be consistent with stage from Deploy Step 6\n```\n\n### Update\n\n1. Remove existing library\n\n2. Install/recreate Redshift Smart Pause and Resume\n\n```bash\nserverless create --template-url https://github.com/servian/aws-redshift-smart-pause-and-resume --path aws-redshift-smart-pause-and-resume\n```\n\n3. Change to Redshift Smart Pause and Resume directory\n\n```bash\ncd aws-redshift-smart-pause-and-resume \n```\n\n3. Redeploy service to AWS Account. The option `redshiftclusterid` is required and need to be specified upon deploying the tool. (See [Deployment Options](#deployment-options) below for more details on the other options provided).\n\n* **`NOTE:`** If updating the tool to schedule another Redshift cluster, ensure that the value set for the option `servicename` is consistent to the value when the tool was first deployed. The value for this option is set to `smart-sched` by default. Possible value is `smart-sched-01`, if this is the value used when deploying the tool to another Redshift cluster. \n\n```bash\nserverless deploy \\\n [--region \u003cAWS region\u003e] \\\n [--aws-profile \u003cAWS CLI profile\u003e] \\\n [--redshiftclusterid \u003cAWS redshift cluster id\u003e]\n [--stage \u003cdeployment environment\u003e] \\\n [--servicename \u003ctool service/stack name\u003e] # default value is smart-sched\n```\n\n### Remove\n\n1. Change to Redshift Smart Pause and Resume directory\n\n```bash\ncd aws-redshift-smart-pause-and-resume \n```\n\n2. Remove Service from AWS Account\n\n```bash\nserverless remove \\\n [--region \u003cAWS region\u003e] \\\n [--aws-profile \u003cAWS CLI profile\u003e] \\\n [--redshiftclusterid \u003cAWS redshift cluster id\u003e]\n [--stage \u003cdeployment environment\u003e] \\\n [--servicename \u003ctool service/stack name\u003e] # default value is smart-sched\n```\n\n## Deployment Options\n\n```\nserverless deploy \\\n[--aws-profile \u003cvalue\u003e] \\\n[--region \u003cvalue\u003e] \\\n[--stage \u003cvalue\u003e] \\\n[--redshiftclusterid \u003cvalue\u003e] \\\n[--servicename \u003cvalue\u003e] \\\n[--enableautoml \u003cvalue\u003e] \\\n[--algorithmarn \u003cvalue\u003e] \\\n[--timezone \u003cvalue\u003e] \\\n```\n\n`--aws-profile` (string)\n\nAWS Profile to deploy resources (default value: `default`)\n\n`--region` (string)\n\nAWS Region to deploy resources (default value: `ap-southeast-2`: Sydney Region)\n\n`--stage` (string)\n\ndeployment environment suffix (default value: `dev`)\n\n`--redshiftclusterid` (string: REQUIRED)\n\nunique identifier of the redshift cluster to enable smart scheduling\n\n`--servicename` (string)\n\nunique identifier for the tool stack (default value: `smart-sched`)\n\n`--enableautoml` (string)\n\npossible values `ENABLED` or `DISABLED` (default: `DISABLED`)\n\n`--algorithmarn` (string)\n\n[Possible values](https://docs.aws.amazon.com/forecast/latest/dg/aws-forecast-choosing-recipes.html) (default: `arn:aws:forecast:::algorithm/ARIMA`)\n\n`--timezone` (string)\n\n[Possible values](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)  (default: `Australia/Melbourne`)\n\n`--intervalminutes` (int)\n\ngranularity of average Redshift CPU utilisation to use throughout stack (default: `15`)\n\n## Scraping Data and Training Amazon Forecast Model After Deployment\n\nThe following script scrapes Redshift CPU utilisation data, and uses it to initially train an Amazon Forecast Model after deploying the stack by executing the Train Forecast Model Step Function. \n\n```bash\npython3 local_scrape_and_train.py \\\n[--awsprofile \u003cvalue\u003e] \\\n[--numdaystoscrape \u003cvalue\u003e] \\\n[--cfnstackname  \u003cvalue\u003e] \\ # value must be consitent with servicename from Deploy Step 6\n[--stage  \u003cvalue\u003e] # value must be consistent with stage from Deploy Step 6\n```\n\n`--awsprofile` (string)\n\nAWS Profile to deploy resources (default value: `default`)\n\n`--numdaystoscrape` (string)\n\nnumber of days (from previous day) worth of Redshift CPU utilisation data to scrape (default value: `14`)\n\n`--cfnstackname` (string)\n\ncloudformation stack name (default value: `smart-sched`. Which is `service-name` in serverless.yml template)\n\n`--stage` (string)\n\nenvironment suffix (default value: `dev`) \n\n## References\n\n[Automating your Amazon Forecast Workflow with Lambda, Step Functions and Cloudwatch Events Rule](https://aws.amazon.com/blogs/machine-learning/automating-your-amazon-forecast-workflow-with-lambda-step-functions-and-cloudwatch-events-rule/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fservian%2Faws-redshift-smart-pause-and-resume","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fservian%2Faws-redshift-smart-pause-and-resume","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fservian%2Faws-redshift-smart-pause-and-resume/lists"}