{"id":28069957,"url":"https://github.com/localstack-samples/sample-dms-postgresql-kinesis","last_synced_at":"2025-09-03T07:34:33.364Z","repository":{"id":279387475,"uuid":"936501627","full_name":"localstack-samples/sample-dms-postgresql-kinesis","owner":"localstack-samples","description":"Sample Application showcasing how to use Aurora PostgreSQL as a source and kinesis as a target in DMS to create a cdc task using the CDK in Python","archived":false,"fork":false,"pushed_at":"2025-02-26T20:01:26.000Z","size":90,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-05-12T19:42:17.054Z","etag":null,"topics":["aurora-postgresql","cdc","cdk","database-migration-service","dms","kinesis","localstack","postgresql","rds","replication"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/localstack-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-21T07:40:23.000Z","updated_at":"2025-02-28T09:55:40.000Z","dependencies_parsed_at":"2025-02-25T10:28:29.322Z","dependency_job_id":"caaab4e9-11b9-4284-b06f-7fa0ac6381e7","html_url":"https://github.com/localstack-samples/sample-dms-postgresql-kinesis","commit_stats":null,"previous_names":["localstack-samples/sample-dms-postgresql-kinesis"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/localstack-samples/sample-dms-postgresql-kinesis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-postgresql-kinesis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-postgresql-kinesis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-postgresql-kinesis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-postgresql-kinesis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/localstack-samples","download_url":"https://codeload.github.com/localstack-samples/sample-dms-postgresql-kinesis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-postgresql-kinesis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273409438,"owners_count":25100444,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-03T02:00:09.631Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aurora-postgresql","cdc","cdk","database-migration-service","dms","kinesis","localstack","postgresql","rds","replication"],"created_at":"2025-05-12T19:33:22.052Z","updated_at":"2025-09-03T07:34:33.335Z","avatar_url":"https://github.com/localstack-samples.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sample Application showcasing how to use DMS to create CDC with Aurora PostgreSQL source and kinesis target\n\n## Introduction\n\nThis scenario demonstrates how to use Database Migration Service (DMS) to create change data capture (CDC) tasks using the Cloud Development Kit in Python. It is a self-contained setup that will create a VPC to host the DMS replication instance, a database, a Kinesis stream, and a replication task.\n\n![dms-postgres-to-kinesis](./dms-postgres-to-kinesis.jpg)\n\n## Pre-requisites\n\n-   [LocalStack Auth Token](https://docs.localstack.cloud/getting-started/auth-token/)\n-   [Python 3.10](https://www.python.org/downloads/) \u0026 `pip`\n-   [Docker Compose](https://docs.docker.com/compose/install/)\n-   [CDK](https://docs.localstack.cloud/user-guide/integrations/aws-cdk/)  with the  [`cdklocal`](https://github.com/localstack/aws-cdk-local) wrapper.\n\n  \nStart LocalStack Pro with the `LOCALSTACK_AUTH_TOKEN`  pre-configured:\n\n```bash\nexport LOCALSTACK_AUTH_TOKEN=\u003cyour-auth-token\u003e\ndocker-compose up\n```\n\nThe Docker Compose file will start LocalStack Pro container and a Postgres container. The Postgres container will be used to showcase how to reach a database external to LocalStack.\n\n## Instructions\n\n### Install the dependencies\n\nInstall all the dependencies by running the following command:\n\n```bash\nmake install\n```\n\n### Creating the infrastructure\n\nTo deploy the infrastructure, you can run the following command:\n\n```bash\nmake deploy\n```\n\n\u003e NOTE: By default we create the kinesis target endpoint with default settings.\n\u003e For non-default settings we enable capturing events related to DDL operations and \n\u003e include `NULL` and empty column values from the events.\n\u003e In order to create the target with non-default values set the environment to `KINESIS_TARGET=non-default`. \n\u003e To know more about these settings, checkout the [official AWS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Kinesis.html#:~:text=Kinesis%20Data%20Streams%20endpoint%20settings).\n\nAfter successful deployment, you will see the following output:\n\n```bash\nOutputs:\nDMsSampleSetupStack.cdcTask = arn:aws:dms:us-east-1:000000000000:task:F6V3I917K2919C2HGVXCKE8O8AY19SX7M4TZH2U\nDMsSampleSetupStack.dbSecret = arn:aws:secretsmanager:us-east-1:000000000000:secret:DMsSampleSetupStack-postgressecret-cb6c3bd1-vgGron\nDMsSampleSetupStack.kinesisStream = arn:aws:kinesis:us-east-1:000000000000:stream/DMsSampleSetupStack-TargetStream3B4B2880-1d69ef19\nStack ARN:\narn:aws:cloudformation:us-east-1:000000000000:stack/DMsSampleSetupStack/8f4fb494\n\n✨  Total time: 45.06s\n```\n\n### Running the tasks\n\nYou can run the tasks by executing the following command:\n\n```bash\nmake run\n```\n\n## Developer Notes\n\nA replication task gets deployed with the stack:\n\nA CDC replication task runs against the RDS database:\n\n-   Creates three tables: `authors`, `accounts`, `books`\n-   Starts CDC replication task\n-   Captures and logs 4 Kinesis events: 1 for `awsdms_apply_exceptions` table, 3 for our tables\n-   Makes 3 inserts\n-   Captures and logs 3 Kinesis events\n-   Makes 3 table alterations, 1 per table\n-   Captures and logs 3 Kinesis events for non-default settings else 0\n-   Logs `table_statistics` for the task\n\n## Deploying on AWS\n\nYou can deploy and run the stack on AWS by running the following commands:\n\n```bash\nmake deploy-aws\nmake run-aws\n```\n\n## License\n\nThis project is licensed under the Apache 2.0 License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flocalstack-samples%2Fsample-dms-postgresql-kinesis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flocalstack-samples%2Fsample-dms-postgresql-kinesis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flocalstack-samples%2Fsample-dms-postgresql-kinesis/lists"}