{"id":14070520,"url":"https://github.com/aws-samples/rds-db-schema-migrations-cicd","last_synced_at":"2025-07-30T07:33:40.356Z","repository":{"id":40925457,"uuid":"419798681","full_name":"aws-samples/rds-db-schema-migrations-cicd","owner":"aws-samples","description":"Integrating RDS databases' schema migrations in the CI/CD pipeline. ","archived":false,"fork":false,"pushed_at":"2023-05-26T18:17:29.000Z","size":467,"stargazers_count":24,"open_issues_count":0,"forks_count":13,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-08-13T07:18:05.122Z","etag":null,"topics":["aws-codepipeline","aws-lambda","cdk","knexjs","postgresql","rds-aurora"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit-0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aws-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-21T16:30:03.000Z","updated_at":"2024-03-29T13:41:22.000Z","dependencies_parsed_at":"2024-08-13T07:17:21.579Z","dependency_job_id":"7326e7d6-a3d3-4826-b2e0-9056a8a3448e","html_url":"https://github.com/aws-samples/rds-db-schema-migrations-cicd","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Frds-db-schema-migrations-cicd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Frds-db-schema-migrations-cicd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Frds-db-schema-migrations-cicd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Frds-db-schema-migrations-cicd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aws-samples","download_url":"https://codeload.github.com/aws-samples/rds-db-schema-migrations-cicd/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228110429,"owners_count":17871181,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-codepipeline","aws-lambda","cdk","knexjs","postgresql","rds-aurora"],"created_at":"2024-08-13T07:07:49.621Z","updated_at":"2024-12-04T12:31:19.512Z","avatar_url":"https://github.com/aws-samples.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"# Integrating Database Schema changes in the CI/CD pipeline. \nContinous delivery strategies are commonly used to increase speed of development and to manage application code changes. However, database changes are often left out of this process and maintained separately, which can become a bottleneck to automated validations as step within CI/CD Pipelines. This project provides an example of how to automate schema changes using AWS Lambda and integrate this step in a CI/CD Pipeline. The solution contains the AWS CDK code for building the infrastructure and the cross-account CI/CD Pipeline.\n\n\u003e In this example we're deploying an Amazon Aurora PostgreSQL-compatible Serverless edition database. However, the same concept can be applied to other supported RDS Engines. \n​\n## Solution overview\nThe CI/CD pipeline is built using the [CDK Pipelines module](https://docs.aws.amazon.com/cdk/api/latest/docs/pipelines-readme.html). The pipeline's source is retrieved from an AWS CodeCommit repository and is triggered upon a push to the default branch. The pipeline contains two stages, the development stage and the production stage. The development stage is configured to deploy resources in the account where the pipeline is deployed. The production stage deployed the resources in another AWS Account. After the resources are deployed / updated, an AWS CodeBuild project is configured to trigger the AWS Lambda function, which runs the schema migrations.\n​\n​\n![CI/CD Diagram](./images/rds-db-schema-migrations.png)\n​\n## Prerequisites \n​\n* Two AWS accounts.\n* [AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html#getting_started_install) installed.\n* [Docker](https://www.docker.com/products/docker-desktop) installed for the initial deploy.\n​\n## Setup\n[Create a new AWS CodeCommit repository](https://docs.aws.amazon.com/codecommit/latest/userguide/how-to-create-repository.html) in the AWS Account and region where you want to deploy the pipeline and upload the source code from this repository. Then, install the npm dependencies: \n\n```\nnpm install\n```\n\nBootstrap the development and production accounts:\n\n```\nchmod +x cdk-boostrap.sh\n./cdk-boostrap.sh \u003cDEV ACCOUNT ID\u003e \u003cPROD ACCOUNT ID\u003e \u003cREGION\u003e \u003cREPOSITORY NAME in your account\u003e \u003cAWS PROD PROFILE\u003e \u003cAWS DEV PROFILE\u003e\n```\n​\n```\nnpx cdk bootstrap  \\ \n    --profile \u003cDEV PROFILE\u003e  \\ \n    --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess  \\\n    aws://\u003cDEV ACCOUNT ID\u003e/\u003cREGION\u003e\n```\n​\nBootstrap the production environment: \n```\nnpx cdk bootstrap \\                \n    --profile \u003cPROD PROFILE\u003e \\\n    --trust \u003cDEV ACCOUNT ID\u003e \\\n    --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess \\\n    aws://\u003cPROD ACCOUNT ID\u003e/\u003cREGION\u003e\n```\n​\nDeploy the CI/CD Pipeline: \n​\n```\nsh ./cdk-deploy-to.sh \u003cDEV ACCOUNT ID\u003e \u003cPROD ACCOUNT ID\u003e \u003cREGION\u003e \u003cREPOSITORY_NAME\u003e\n```\n​\nAfter this initial deployment, the CI/CD Pipeline will deploy the infrastructure resources. Go to AWS Management Console, CodePipeline service to track the pipeline progress.\n​\n## Schema migrations with Knex.js\n​\n[Knex.js](https://knexjs.org/) is a SQL Query Builder library for Node.js compatible with Postgres databases. In addition, Knex offers an easy way to manage\nschema migrations by providing CLI commands to easily create and apply migrations using timestamp based migration files. You can learn more about Knex migrations in [their documentaion](https://knexjs.org/#Migrations). \n​\n### Creating migration files\nThe migration file is created inside the *lib/lambda/migrations* directory. The name of the file is composed of the timestamp and the migration name you specify in the command, for example: `20211004160632_CreateInitialDemoTable.ts`. Customize this file with the SQL Queries necessary for the migration. Take a look at the provided file as a reference.\n\n```\nnpm run migration:make CreateInitialTable\n```\n​\nBelow is an example of a migration file. If you wish to only use RDS Native Backup and snapshots, you can leave out the *down* (rollback) specification.\n\n```\nconst Knex = require(\"knex\");\n\nexport async function up(knex: typeof Knex): Promise\u003cvoid\u003e {\n  return await knex.schema.createTable('demo_table', (table: typeof Knex.TableBuilder) =\u003e {\n    table.increments('id')\n  })\n}\n\nexport async function down(knex: typeof Knex): Promise\u003cvoid\u003e {\n  await knex.schema.dropTable('demo_table')\n}\n```\n​\n### Applying migrations \nMigrations are applied using the AWS Lambda functions and are run as part of the CI/CD pipeline after the resources are deployed for both the Development and Production environments. You can view the AWS Lambda function logs and the AWS CodeBuild project logs to check the status of the schema migration.\n\n## Connecting to the RDS Database in the private VPC from a local machine \n​\nA Bastion Host is deployed to connect to the development PostgreSQL database from your local machine.\nThe host is configured to use AWS Session Manager to connect to the instance and than forward the traffic to the database.  \n​\n1. Install Session Manager plugin for AWS CLI following [these instructions](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html).\n2. Configure SSH in your local machine by modifying the *~/.ssh/config* file to include the following code:\n​\n```\n# SSH over Session Manager\nhost i-* mi-*\n    ProxyCommand sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'\"\n```\n​\n3. Add your public ssh key to the instance's authorized keys: \n​\n```\n# grap your public key, typically found in: ~/.ssh/id_rsa.pub\ncat ~/.ssh/id_rsa.pub\n​\n# Connect to the Instance using Session Manager in AWS Console and add the authorized key from above:\nsudo su\nnano /home/ec2-user/.ssh/authorized_keys\n```\n​\n4. You can now connect to the instance using port forwarding from your local machine:\n​\n```\nssh -N -L \u003cLOCAL PORT\u003e:\u003cDATABASE HOST NAME\u003e:5432 ec2-user@\u003cBASTION HOST INSTANCE ID\u003e\n```\n\nYou can now use your preferred SQL Client to connect to the database by using *127.0.0.1* as a hostname and \u003cLOCAL PORT\u003e as port. The database username and password can be retrieved by the AWS Secret Manager secret. \n\n## Cleanup\nDestroy the CDK Pipeline: \n\n```\nsh ./cdk-destroy.sh \u003cDEV ACCOUNT ID\u003e \u003cPROD ACCOUNT ID\u003e \u003cREGION\u003e \u003cREPOSITORY_NAME\u003e\n```\n\nThe resources created by the pipeline are not destroyed with the above step. You need to go to AWS CloudFormation and delete these stacks manually. For example, below is the stack created by the pipeline for the development environment, named *Development-PrimaryStack*: \n\n![CI/CD Diagram](./images/cfn.png)\n\nAlternatively, you can use AWS CLI to delete the infrastructure stacks.\nClean the development account resources created by the CI/CD Pipeline: \n\n```\naws cloudformation delete-stack --stack-name Development-PrimaryStack\n```\n\nClean the production account resources created by the CI/CD Pipeline: \n\n```\naws cloudformation delete-stack --stack-name Production-PrimaryStack\n```\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis library is licensed under the MIT-0 License. See the LICENSE file.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws-samples%2Frds-db-schema-migrations-cicd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faws-samples%2Frds-db-schema-migrations-cicd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws-samples%2Frds-db-schema-migrations-cicd/lists"}