{"id":14969026,"url":"https://github.com/localstack-samples/sample-dms-cdc-rds-to-kinesis","last_synced_at":"2025-10-26T06:31:02.009Z","repository":{"id":241686494,"uuid":"804925068","full_name":"localstack-samples/sample-dms-kinesis-rds-mariadb","owner":"localstack-samples","description":"Sample Application showcasing how to use DMS to create CDC and full load tasks using the CDK in Python.","archived":false,"fork":false,"pushed_at":"2024-05-29T04:34:43.000Z","size":89,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-31T16:11:42.341Z","etag":null,"topics":["cdk","cdk-python","database-migration-service","dms","kinesis","localstack","mariadb","rds"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/localstack-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-23T14:35:23.000Z","updated_at":"2024-09-14T02:16:43.000Z","dependencies_parsed_at":"2024-05-29T17:53:26.745Z","dependency_job_id":null,"html_url":"https://github.com/localstack-samples/sample-dms-kinesis-rds-mariadb","commit_stats":null,"previous_names":["localstack-samples/sample-dms-kinesis-rds-mariadb"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-kinesis-rds-mariadb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-kinesis-rds-mariadb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-kinesis-rds-mariadb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-kinesis-rds-mariadb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/localstack-samples","download_url":"https://codeload.github.com/localstack-samples/sample-dms-kinesis-rds-mariadb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238275997,"owners_count":19445324,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdk","cdk-python","database-migration-service","dms","kinesis","localstack","mariadb","rds"],"created_at":"2024-09-24T13:40:58.072Z","updated_at":"2025-10-26T06:31:02.004Z","avatar_url":"https://github.com/localstack-samples.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Real-time Database Replication with DMS and Kinesis on LocalStack\n\n| Key          | Value                                                                                    |\n| ------------ | ---------------------------------------------------------------------------------------- |\n| Environment  | LocalStack, AWS                                                                          |\n| Services     | DMS, RDS, Kinesis, VPC, Secrets Manager                                                  |\n| Integrations | AWS CDK, Docker Compose, AWS SDK for Python                                              |\n| Categories   | Database Migration, Change Data Capture, Streaming                                       |\n| Level        | Intermediate                                                                             |\n| Use Case     | Database Migration, Real-time Data Replication, CDC Implementation                       |\n| GitHub       | [Repository link](https://github.com/localstack-samples/sample-dms-cdc-rds-to-kinesis)          |\n\n## Introduction\n\nThis sample demonstrates how to use AWS Database Migration Service (DMS) to create change data capture (CDC) and full load replication tasks using the AWS Cloud Development Kit in Python. The application showcases real-time data replication from MariaDB databases to Kinesis streams, enabling you to capture and stream database changes as they occur. It is a self-contained setup that will create a local VPC to host 2 databases, a Kinesis stream, and 4 replication tasks. To test this application sample, we will demonstrate how you use LocalStack to deploy the complete DMS infrastructure on your developer machine and validate the data replication workflow locally. This provides a cost-effective way to develop and test database migration patterns before deploying to production AWS environments.\n\n## Architecture\n\nThe following diagram shows the architecture that this sample application builds and deploys:\n\n![dms-mariadb-to-kinesis](./images/architecture.png)\n\n- [VPC](https://docs.localstack.cloud/aws/services/ec2/#vpc) with custom networking to host database resources\n- [RDS MariaDB instance](https://docs.localstack.cloud/aws/services/rds/) as the target database for CDC replication\n- External MariaDB container as the source database for full load replication  \n- [DMS Replication Instance](https://docs.localstack.cloud/aws/services/dms/) to execute migration tasks\n- [DMS Source/Target Endpoints](https://docs.localstack.cloud/aws/services/dms/) connecting to both MariaDB instances\n- [DMS Replication Tasks](https://docs.localstack.cloud/aws/services/dms/) for full load and CDC operations\n- [Kinesis Data Stream](https://docs.localstack.cloud/aws/services/kinesis/) as the target for replicated data\n- [Secrets Manager](https://docs.localstack.cloud/aws/services/secretsmanager/) for secure database credential storage\n\n## Prerequisites\n\n- [`LOCALSTACK_AUTH_TOKEN`](https://docs.localstack.cloud/getting-started/auth-token/)\n- [Python 3.10+](https://www.python.org/downloads/) \u0026 `pip`\n- [Docker Compose](https://docs.docker.com/compose/install/) \n- [CDK](https://docs.localstack.cloud/user-guide/integrations/aws-cdk/) with the [`cdklocal`](https://github.com/localstack/aws-cdk-local) wrapper\n- [`make`](https://www.gnu.org/software/make/) (**optional**, but recommended for running the sample application)\n\n## Installation\n\nTo run the sample application, you need to install the required dependencies.\n\nFirst, clone the repository:\n\n```shell\ngit clone https://github.com/localstack/sample-dms-cdc-rds-to-kinesis.git\n```\n\nThen, navigate to the project directory:\n\n```shell\ncd sample-dms-cdc-rds-to-kinesis\n```\n\nInstall all the dependencies by running the following command:\n\n```shell\nmake install\n```\n\nThis will create a virtual environment and install the required Python packages including AWS CDK dependencies.\n\n## Deployment\n\nStart LocalStack Pro with the `LOCALSTACK_AUTH_TOKEN` pre-configured:\n\n```shell\nexport LOCALSTACK_AUTH_TOKEN=\u003cyour-auth-token\u003e\nmake start\n```\n\nThe Docker Compose file will start LocalStack Pro container and a MariaDB container that will serve as the external source database.\n\nTo deploy the sample application infrastructure, run the following command:\n\n```shell\nmake deploy\n```\n\nAfter successful deployment, you will see the following output:\n\n```shell\nOutputs:\nDMsSampleSetupStack.cdcTask1 = arn:aws:dms:us-east-1:000000000000:task:A001NYMR4Z0NK45ZBJT6954RNMGEKL2PQ9XQYR4\nDMsSampleSetupStack.cdcTask2 = arn:aws:dms:us-east-1:000000000000:task:GO5RC4J6CKZWSJKF4CGB6ZV3ZEMGI38DFPJF2ZU\nDMsSampleSetupStack.cdcTaskSecret = arn:aws:secretsmanager:us-east-1:000000000000:secret:DMsSampleSetupStack-rdsinstanceSecret07FEB42-907ed0cf-RSPkZq\nDMsSampleSetupStack.fullTask1 = arn:aws:dms:us-east-1:000000000000:task:BCZLANJP9WFXKNTYBEWTAQ1YHIVJ5C2ZUIHDPB2\nDMsSampleSetupStack.fullTask2 = arn:aws:dms:us-east-1:000000000000:task:ZO7WPZTTAKOA1CONK2Y3Y0H6FXLAFWUYX1OPGPM\nDMsSampleSetupStack.fullTaskSecret = arn:aws:secretsmanager:us-east-1:000000000000:secret:DMsSampleSetupStack-mariadbaccesssecret40AD7-611fcbcd-IKWDDh\nDMsSampleSetupStack.kinesisStream = arn:aws:kinesis:us-east-1:000000000000:stream/DMsSampleSetupStack-TargetStream3B4B2880-02dd0371\nStack ARN:\narn:aws:cloudformation:us-east-1:000000000000:stack/DMsSampleSetupStack/b8298866\n\n✨  Total time: 49.33s\n```\n\n## Testing\n\nYou can run the replication tasks and validate the data pipeline by executing the following command:\n\n```shell\nmake run\n```\n\nThis will execute the complete test scenario including:\n\n- Creating test tables and inserting initial data\n- Starting full load replication tasks\n- Monitoring Kinesis stream for replicated events\n- Starting CDC replication tasks  \n- Performing additional data changes to trigger CDC\n- Logging table statistics and replication progress\n\nThe test validates both full load and CDC replication patterns, demonstrating how DMS captures and streams database changes to Kinesis in real-time.\n\n## Use Cases\n\n### Full Load Replication\n\nThis sample demonstrates full load replication tasks against an external MariaDB database running in Docker. The full load scenario showcases initial data migration and bulk data transfer patterns.\n\nThe full load replication workflow includes:\n\n- Creating three tables: `authors`, `accounts`, `novels` with sample data\n- Starting full load task 1 targeting tables starting with 'a' (`a%` table mapping)\n- Starting full load task 2 targeting the `novels` table (specific table mapping)\n- Capturing Kinesis events for table operations: drop tables, create tables, and data inserts\n- Monitoring table statistics and replication progress for both tasks\n- Demonstrating selective table replication using different mapping rules\n\nThis pattern is ideal for initial database migrations where you need to transfer existing data from on-premises or external databases to AWS-managed services.\n\n### Change Data Capture (CDC) \n\nThe CDC replication tasks demonstrate real-time change capture from a MariaDB RDS instance, streaming ongoing database changes to Kinesis as they occur.\n\nThe CDC replication workflow includes:\n\n- Creating three tables: `authors`, `accounts`, `novels` in the RDS database\n- Starting CDC task 1 targeting tables starting with 'a' (`a%` table mapping)  \n- Starting CDC task 2 targeting the `novels` table (specific table mapping)\n- Capturing real-time changes: INSERT, UPDATE, and DELETE operations\n- Performing table alterations and schema changes during active replication\n- Streaming all changes to the same Kinesis Data Stream for downstream processing\n- Monitoring replication lag and table statistics for ongoing operations\n\nThis pattern enables building event-driven architectures and real-time analytics pipelines that respond to database changes as they happen.\n\n## Summary\n\nThis sample application demonstrates how to build, deploy, and test a complete database migration and replication pipeline using AWS DMS and related services. It showcases the following patterns:\n\n- Deploying DMS infrastructure using AWS CDK with Python\n- Configuring full load and CDC replication tasks for different migration scenarios\n- Integrating multiple database sources (RDS and external MariaDB) with streaming targets\n- Using Secrets Manager for secure credential management in DMS workflows\n- Monitoring data replication through Kinesis stream events and DMS table statistics\n- Leveraging LocalStack Pro for cost-effective development and testing of DMS workflows\n\nThe application provides a foundation for understanding enterprise database migration patterns and real-time data replication architectures.\n\n## Learn More\n\n- [LocalStack DMS Documentation](https://docs.localstack.cloud/aws/services/dms/)\n- [AWS DMS Best Practices](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_BestPractices.html)\n- [Change Data Capture Patterns](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html)\n- [Using AWS CDK with LocalStack](https://docs.localstack.cloud/user-guide/integrations/aws-cdk/)\n- [Kinesis Data Streams for Real-time Processing](https://docs.localstack.cloud/aws/services/kinesis/)\n- [Database Migration Strategies with DMS](https://aws.amazon.com/dms/resources/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flocalstack-samples%2Fsample-dms-cdc-rds-to-kinesis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flocalstack-samples%2Fsample-dms-cdc-rds-to-kinesis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flocalstack-samples%2Fsample-dms-cdc-rds-to-kinesis/lists"}