https://github.com/idrsdev/dynamo-opensearch-sync
Sync DynamoDB with OpenSearch in near real-time using DynamoDB(Single Table Design) Streams (CDC) and AWS Lambda for seamless data updates and searchability
https://github.com/idrsdev/dynamo-opensearch-sync
dynamodb dynamodb-streams opensearch single-table-design typescript
Last synced: 11 months ago
JSON representation
Sync DynamoDB with OpenSearch in near real-time using DynamoDB(Single Table Design) Streams (CDC) and AWS Lambda for seamless data updates and searchability
- Host: GitHub
- URL: https://github.com/idrsdev/dynamo-opensearch-sync
- Owner: idrsdev
- Created: 2024-11-10T14:24:19.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-16T07:32:08.000Z (over 1 year ago)
- Last Synced: 2025-03-03T15:47:39.437Z (over 1 year ago)
- Topics: dynamodb, dynamodb-streams, opensearch, single-table-design, typescript
- Language: TypeScript
- Homepage:
- Size: 83 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: ReadMe.md
Awesome Lists containing this project
README
## Table of Contents
1. [Stream Architecture](#stream-architecture)
2. [Local Testing](#local-testing)
# Stream Architecture
This repository contains the DynamoDB stream processing logic for entities like user, order etc, as well as integration with services like OpenSearch. Below is the breakdown of the folder structure.
## 📦 Folder Structure
### `handlers`
Contains Lambda handlers and entry points.
- **StreamBatchHandler.ts**: Lambda handler responsible for processing a batch of records.
### `processors`
Contains the core processing logic.
- **BaseEntityProcessor.ts**: Base class for all entity processors, providing shared functionality for processing records.
- **BatchProcessor.ts**: Logic for batching records for efficient processing.
- **EntityProcessorFactory.ts**: Factory pattern for creating and delegating to appropriate entity processors based on record types.
- **EventProcessor.ts**: Handles processing specific to events.
#### `entity-processors`
Processors based on the partition key (PK). These are high-level processors responsible for deciding which specific entity processor to delegate to.
- **OrderEntityProcessor.ts**: Processor for handling order-related entities.
- **UserEntityProcessor.ts**: Processor for handling user-related entities.
- _The same structure applies for future entities to be added._
#### `order-processors`
Processors specific to service requests based on the sort key (SK). These processors are categorized by specific features or types within service requests.
- _And any future processors to be added._
### `services`
Contains services related to external integrations, such as OpenSearch.
- **OpenSearchBulkService.ts**: Service handling bulk operations for OpenSearch.
### Root files
- **event.json**: A sample event file used for testing.
- **index.ts**: The main entry point of the application.
# Local Testing
`npm i --registry=https://registry.npmjs.org/`
`cp parameters.example.json parameters.json`
provide the required values in the parameters.json
Run the following command to invoke this function using the test event:
sam build LogisticsStreamCallbackFunction && sam local invoke LogisticsStreamCallbackFunction --event "event.json" --env-vars parameters.json
> **Note**: Use this repository **at your own discretion**, especially for production use. This repository is provided as-is, and caution should be exercised when adapting it for live or production environments. Misconfigurations or improper usage could lead to data inconsistencies or other issues.