Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/kaviara-14/aws-rds-migration-with-dynamodb-integration

Migrating an On-premise MySQL to RDS MySQL and also inserting the migrated records into AWS DynamoDB using Lambda Function
https://github.com/kaviara-14/aws-rds-migration-with-dynamodb-integration

aws awsdynamodb awslambda awsrds

Last synced: 4 days ago
JSON representation

Migrating an On-premise MySQL to RDS MySQL and also inserting the migrated records into AWS DynamoDB using Lambda Function

Awesome Lists containing this project

README 

        
# MySQL to AWS RDS Migration with DynamoDB Integration



Migrating an on-premises MySQL database to Amazon RDS MySQL by using AWS Database Migration Services (DMS).And Configured a CloudWatch Subscription filter to look for specific patterns related to new records . If the pattern matches, we are triggering a Lambda function. That Lambda function checks and retrieves the newly added records from the RDS MySQL Database and inserts them into the AWS DynamoDB Database.






### AWS Services : AWS RDS, AWS DMS, Lambda Function, AWS DynamoDB, CloudWatch, VPC



# Project Description : 



### 1. Set Up On-Premises MySQL Database

Ensure that MySQL is installed and running in your on-premises environment. Create the necessary databases and tables that you plan to migrate to Amazon RDS. Populate these tables with some initial records to serve as the source data for the migration.



### 2. Create Amazon RDS MySQL Instance

In the AWS Management Console, create an RDS MySQL instance named **awsemployeedb**. Configure the instance type, storage, and network settings. Set up a Security Group named mysql-sg with port 3306 open in the inbound rules and attach it to the instance.






### 3. Create Replication Instance

Navigate to the AWS DMS Console and create a new replication instance named **dbreplica**. This instance serves as a bridge between your on-premises MySQL database and the RDS MySQL database. Select an instance size and settings that align with your workload requirements.






### 4. Create Source and Target endpoints

Set up the source endpoint to connect to your on-premises MySQL database by providing the necessary connection details such as server name, port, database name, and credentials. Similarly, set up the target endpoint to connect to your RDS MySQL instance, and provide the connection details for the RDS instance.






### 5. Create Migration Task

In the AWS DMS Console, create a new migration task to transfer data from your on-premises MySQL database to the RDS MySQL instance. Choose the appropriate migration type, specify the replication instance, and configure the source and target endpoints you created earlier. Additionally, set up table mappings to include the tables you wish to migrate. Start the migration task and monitor its status in the DMS console.






### 6. Configure CloudWatch Monitoring

Enable CloudWatch Logs for your RDS MySQL instance and configure a subscription filter to monitor logs for patterns indicating the addition of new records. In CloudWatch, create a subscription filter to match specific patterns, such as SQL INSERT statements.



### 7. Configure Lambda Function

In the AWS Lambda console, create a new function that is triggered by CloudWatch Logs. This function should query and fetch new records from the RDS MySQL database and insert these records into DynamoDB. Additionally, attach the necessary permissions to the Lambda function to allow access to DynamoDB.

   

### 8. Create DynamoDB Table

In the DynamoDB console, create a new table to store the migrated records. Specify the partition key schema and any required sort indexes.Monitor the table to confirm that records are correctly added and use the DynamoDB console