Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/localstack-samples/sample-dms-kinesis-rds-mariadb
Sample Application showcasing how to use DMS to create CDC and full load tasks using the CDK in Python.
https://github.com/localstack-samples/sample-dms-kinesis-rds-mariadb
cdk cdk-python database-migration-service dms kinesis localstack mariadb rds
Last synced: 8 days ago
JSON representation
Sample Application showcasing how to use DMS to create CDC and full load tasks using the CDK in Python.
- Host: GitHub
- URL: https://github.com/localstack-samples/sample-dms-kinesis-rds-mariadb
- Owner: localstack-samples
- License: apache-2.0
- Created: 2024-05-23T14:35:23.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2024-05-29T04:34:43.000Z (9 months ago)
- Last Synced: 2025-01-31T16:11:42.341Z (19 days ago)
- Topics: cdk, cdk-python, database-migration-service, dms, kinesis, localstack, mariadb, rds
- Language: Python
- Homepage:
- Size: 86.9 KB
- Stars: 3
- Watchers: 3
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Sample Application showcasing how to use DMS to create CDC
## Introduction
This scenario demonstrates how to use Database Migration Service (DMS) to create change data capture (CDC) and full load tasks using the Cloud Development Kit in Python. It is a self-contained setup that will create a VPC to host 2 databases, a Kinesis stream, and 4 replication tasks.

## Pre-requisites
- [LocalStack Auth Token](https://docs.localstack.cloud/getting-started/auth-token/)
- [Python 3.10](https://www.python.org/downloads/) & `pip`
- [Docker Compose](https://docs.docker.com/compose/install/)
- [CDK](https://docs.localstack.cloud/user-guide/integrations/aws-cdk/) with the [`cdklocal`](https://github.com/localstack/aws-cdk-local) wrapper.
Start LocalStack Pro with the `LOCALSTACK_AUTH_TOKEN` pre-configured:```bash
export LOCALSTACK_AUTH_TOKEN=
docker-compose up
```The Docker Compose file will start LocalStack Pro container and a MariaDB container. The MariaDB container will be used to showcase how to reach a database external to LocalStack.
## Instructions
### Install the dependencies
Install all the dependencies by running the following command:
```bash
make install
```### Creating the infrastructure
To deploy the infrastructure, you can run the following command:
```bash
make deploy
```After successful deployment, you will see the following output:
```bash
Outputs:
DMsSampleSetupStack.cdcTask1 = arn:aws:dms:us-east-1:000000000000:task:A001NYMR4Z0NK45ZBJT6954RNMGEKL2PQ9XQYR4
DMsSampleSetupStack.cdcTask2 = arn:aws:dms:us-east-1:000000000000:task:GO5RC4J6CKZWSJKF4CGB6ZV3ZEMGI38DFPJF2ZU
DMsSampleSetupStack.cdcTaskSecret = arn:aws:secretsmanager:us-east-1:000000000000:secret:DMsSampleSetupStack-rdsinstanceSecret07FEB42-907ed0cf-RSPkZq
DMsSampleSetupStack.fullTask1 = arn:aws:dms:us-east-1:000000000000:task:BCZLANJP9WFXKNTYBEWTAQ1YHIVJ5C2ZUIHDPB2
DMsSampleSetupStack.fullTask2 = arn:aws:dms:us-east-1:000000000000:task:ZO7WPZTTAKOA1CONK2Y3Y0H6FXLAFWUYX1OPGPM
DMsSampleSetupStack.fullTaskSecret = arn:aws:secretsmanager:us-east-1:000000000000:secret:DMsSampleSetupStack-mariadbaccesssecret40AD7-611fcbcd-IKWDDh
DMsSampleSetupStack.kinesisStream = arn:aws:kinesis:us-east-1:000000000000:stream/DMsSampleSetupStack-TargetStream3B4B2880-02dd0371
Stack ARN:
arn:aws:cloudformation:us-east-1:000000000000:stack/DMsSampleSetupStack/b8298866✨ Total time: 49.33s
```### Running the tasks
You can run the tasks by executing the following command:
```bash
make run
```## Developer Notes
Four tasks are deployed with the stack, split into two parts.
First, a full load replication task runs against the external DB:
- Creates three tables: `authors`, `accounts`, `novels`
- Makes four inserts
- Starts full load task 1 targeting tables starting with 'a' (`a%` table mapping)
- Captures and logs six Kinesis events: 2 drop tables, 2 create tables, 2 inserts
- Starts full load task 2 targeting the `novels` table (`novels` table mapping)
- Captures and logs four Kinesis events: 1 drop table, 1 create table, 2 inserts
- Logs `table_statistics` for both tasksNext, a CDC replication task runs against the RDS database:
- Creates three tables: `authors`, `accounts`, `novels`
- Starts CDC task 1 targeting tables starting with 'a' (`a%` table mapping)
- Starts CDC task 2 targeting the `novels` table (`novels` table mapping)
- Captures and logs five Kinesis events: 2 for `awsdms_apply_exceptions` table, 3 for our tables
- Makes four inserts
- Captures and logs four Kinesis events: 2 for tables in task 1, 2 for table in task 2
- Makes three table alterations, one per table
- Captures and logs three Kinesis events
- Logs `table_statistics` for both tasksTwo tasks perform full load replication on Dockerized MariaDB. The other two perform CDC replication on a MariaDB RDS database.
All tasks target the same Kinesis Stream.
## Deploying on AWS
You can deploy and run the stack on AWS by running the following commands:
```bash
make deploy-aws
make run-aws
```## License
This project is licensed under the Apache 2.0 License.