{"id":18573144,"url":"https://github.com/localstack-samples/sample-dms-s3-kinesis","last_synced_at":"2025-05-15T23:12:46.450Z","repository":{"id":249792673,"uuid":"830874002","full_name":"localstack-samples/sample-dms-s3-kinesis","owner":"localstack-samples","description":"Sample Application showcasing how to use s3 as a source and kinesis as a target to create full-load and cdc tasks using the CDK in Python","archived":false,"fork":false,"pushed_at":"2024-07-24T10:12:22.000Z","size":84,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-12-26T14:28:59.657Z","etag":null,"topics":["cdk","cdk-python","csv-source","data-migration-service","dms","kinesis","localstack","s3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/localstack-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-19T07:17:48.000Z","updated_at":"2024-07-24T10:23:06.000Z","dependencies_parsed_at":"2024-12-26T14:37:54.611Z","dependency_job_id":null,"html_url":"https://github.com/localstack-samples/sample-dms-s3-kinesis","commit_stats":null,"previous_names":["localstack-samples/sample-dms-s3-kinesis"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-s3-kinesis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-s3-kinesis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-s3-kinesis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-dms-s3-kinesis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/localstack-samples","download_url":"https://codeload.github.com/localstack-samples/sample-dms-s3-kinesis/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239309239,"owners_count":19617848,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdk","cdk-python","csv-source","data-migration-service","dms","kinesis","localstack","s3"],"created_at":"2024-11-06T23:08:09.974Z","updated_at":"2025-02-17T14:40:48.894Z","avatar_url":"https://github.com/localstack-samples.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sample Application showcasing how to use DMS with s3 source and kinesis target\n\n## Introduction\n\nThis scenario demonstrates how to use Database Migration Service (DMS) to create change data capture (CDC) and full load tasks using the Cloud Development Kit in Python. It is a self-contained setup that will create a VPC to host the DMS replication instance, an S3 bucket as the data source, a Kinesis stream as the target, and the replication tasks.\n\n![dms-s3-kinesis](./dms-s3-kinesis.jpg)\n\n## Pre-requisites\n\n-   [LocalStack Auth Token](https://docs.localstack.cloud/getting-started/auth-token/)\n-   [Python 3.10](https://www.python.org/downloads/) \u0026 `pip`\n-   [Docker Compose](https://docs.docker.com/compose/install/)\n-   [CDK](https://docs.localstack.cloud/user-guide/integrations/aws-cdk/)  with the  [`cdklocal`](https://github.com/localstack/aws-cdk-local) wrapper.\n\n  \nStart LocalStack Pro with the `LOCALSTACK_AUTH_TOKEN`  pre-configured:\n\n```bash\nexport LOCALSTACK_AUTH_TOKEN=\u003cyour-auth-token\u003e\ndocker-compose up\n```\n\nThe Docker Compose file will start LocalStack Pro container and a MariaDB container. The MariaDB container will be used to showcase how to reach a database external to LocalStack.\n\n## Instructions\n\n### Install the dependencies\n\nInstall all the dependencies by running the following command:\n\n```bash\nmake install\n```\n\n### Creating the infrastructure\n\nTo deploy the infrastructure, you can run the following command:\n\n```bash\nmake deploy\n```\n\nAfter successful deployment, you will see the following output:\n\n```bash\nOutputs:\nDmsS3ToKinesisStack.cdcTask = arn:aws:dms:us-east-1:000000000000:task:WLVQO3DQTCC31K5OGLN9ZQI1J6X88GF9T4PKLY8\nDmsS3ToKinesisStack.fullLoadTask = arn:aws:dms:us-east-1:000000000000:task:AJGNKCENBYN0UZFYCLIU62WEH9NF62713YTKJZU\nDmsS3ToKinesisStack.kinesisStream = arn:aws:kinesis:us-east-1:000000000000:stream/DmsS3ToKinesisStack-TargetStream3B4B2880-6e4a4e31\nDmsS3ToKinesisStack.s3Secret = arn:aws:secretsmanager:us-east-1:000000000000:secret:DmsS3ToKinesisStack-s3accesssecret988D0EB7-79f34adc-scxavJ\nDmsS3ToKinesisStack.sourceBucket = source-bucket-s3-kinesis-dms\nStack ARN:\narn:aws:cloudformation:us-east-1:000000000000:stack/DmsS3ToKinesisStack/edecb02a\n\n✨  Total time: 36.43s\n```\n\n### Running the tasks\n\nYou can run the tasks by executing the following command:\n\n```bash\nmake run\n```\n\n## Developer Notes\n\nTwo tasks are deployed with the stack:\n\nFirst, a full load replication task runs against the S3 source:\n\n-   Creates three tables: `employee`, `department`, `project`\n-   Uploads sample CSV data for each table to the S3 bucket\n-   Starts the full load task\n-   Captures and logs Kinesis events for the uploaded data: 3 create table, 3 drop table, 10 inserts\n-   Logs `table_statistics` for the task\n\nNext, a CDC replication task runs against the S3 :\n\n-   Creates three tables: `employee`, `department`, `project`\n-   Uploads sample CSV data for each table to the S3 bucket\n-   Captures and logs Kinesis events for the uploaded data: 3 create table, 4 inserts, 3 updates, 4 deletes, 1 table for `awsdms_apply_exceptions`\n-   Logs `table_statistics` for the task\n\nBoth the tasks target the same Kinesis Stream.\n\n## Deploying on AWS\n\nYou can deploy and run the stack on AWS by running the following commands:\n\n```bash\nmake deploy-aws\nmake run-aws\n```\n\n## License\n\nThis project is licensed under the Apache 2.0 License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flocalstack-samples%2Fsample-dms-s3-kinesis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flocalstack-samples%2Fsample-dms-s3-kinesis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flocalstack-samples%2Fsample-dms-s3-kinesis/lists"}