https://github.com/thesmartmonkey/aws-cdk-fanout
CDK contruct for a Fanout pattern with Api Gateway > SNS > SQS > lambda
https://github.com/thesmartmonkey/aws-cdk-fanout
Last synced: 3 months ago
JSON representation
CDK contruct for a Fanout pattern with Api Gateway > SNS > SQS > lambda
- Host: GitHub
- URL: https://github.com/thesmartmonkey/aws-cdk-fanout
- Owner: TheSmartMonkey
- License: mit
- Created: 2025-05-05T20:01:16.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-05-11T14:11:52.000Z (about 1 year ago)
- Last Synced: 2025-10-30T00:30:59.083Z (8 months ago)
- Language: TypeScript
- Size: 271 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# AWS CDK Fanout Construct
CDK contruct for a Fanout pattern with Api Gateway > SNS > SQS > lambda
[](https://badge.fury.io/js/aws-cdk-fanout)
[](https://opensource.org/licenses/MIT)
## Overview
The AWS CDK Fanout construct simplifies creating webhook fanout architectures in AWS. It provides a streamlined way to:
1. Receive webhook events via API Gateway
2. Publish events to an SNS topic
3. Route messages to multiple SQS queues based on filtering criteria
4. Process messages with Lambda functions
## Installation
```bash
npm install aws-cdk-fanout
```
## Getting Started
### Basic Usage
```typescript
import { FanoutConstruct, FanoutConstructPropsEntity, SqsToLambdaPropsEntity } from 'aws-cdk-fanout';
import { Duration, Stack } from 'aws-cdk-lib';
import { FilterOrPolicy } from 'aws-cdk-lib/aws-sns';
import { Construct } from 'constructs';
export class MyStack extends Stack {
constructor(scope: Construct, id: string) {
super(scope, id);
// Define the SQS to Lambda mapping
const orderProcessingQueue = new SqsToLambdaPropsEntity({
lambdaName: 'process-orders',
handlerPath: 'src/lambdas/process-orders.ts',
envVars: {
ENVIRONMENT: 'dev',
},
snsFilter: {
eventType: FilterOrPolicy.stringFilter({
allowlist: ['order.created', 'order.updated']
})
},
sqsMaxBatchSize: 10,
sqsMaxBatchingWindow: Duration.seconds(30),
sqsVisibilityTimeout: Duration.seconds(120)
});
// Create the fanout construct
new FanoutConstruct(this, 'WebhooksFanout', new FanoutConstructPropsEntity({
stage: 'dev',
region: 'us-east-1',
sqsToLambda: [orderProcessingQueue]
}));
}
}
```
### Lambda Handler Example
Here's an example of a Lambda handler that processes messages from the SQS queue:
```typescript
import { SQSEvent } from 'aws-lambda';
export const handler = async (event: SQSEvent): Promise => {
for (const record of event.Records) {
console.log('Processing message:', record.body);
// Parse the message body
const message = JSON.parse(record.body);
const snsMessage = JSON.parse(message.Message);
// Process the message based on your business logic
console.log('Event data:', snsMessage);
}
};
```
## Features
- **API Gateway Integration**: Automatically creates an API Gateway endpoint to receive webhook events
- **Message Filtering**: Filter messages using SNS message filtering attributes
- **Dead Letter Queues**: Automatic DLQ setup for failed message processing
- **Batch Processing**: Configure batch size and batching window for Lambda processing
- **Secure by Default**: Includes API key authentication for the API Gateway endpoint
## Advanced Configuration
### Multiple Consumers with Different Filters
```typescript
// Order processing queue
const orderProcessingQueue = new SqsToLambdaPropsEntity({
lambdaName: 'process-orders',
handlerPath: 'src/lambdas/process-orders.ts',
envVars: { /* ... */ },
snsFilter: {
eventType: FilterOrPolicy.stringFilter({
allowlist: ['order.created', 'order.updated']
})
},
sqsMaxBatchSize: 10,
sqsMaxBatchingWindow: Duration.seconds(30),
sqsVisibilityTimeout: Duration.seconds(120)
});
// User processing queue
const userProcessingQueue = new SqsToLambdaPropsEntity({
lambdaName: 'process-users',
handlerPath: 'src/lambdas/process-users.ts',
envVars: { /* ... */ },
snsFilter: {
eventType: FilterOrPolicy.stringFilter({
allowlist: ['user.created', 'user.updated']
})
},
sqsMaxBatchSize: 5,
sqsMaxBatchingWindow: Duration.seconds(60),
sqsVisibilityTimeout: Duration.seconds(180)
});
// Create the fanout construct with multiple consumers
new FanoutConstruct(this, 'WebhooksFanout', new FanoutConstructPropsEntity({
stage: 'dev',
region: 'us-east-1',
sqsToLambda: [orderProcessingQueue, userProcessingQueue]
}));
```
### Customizing Queue Options
```typescript
const queue = new SqsToLambdaPropsEntity({
// ... other properties
queueOptions: {
retentionPeriod: Duration.days(7),
dataKeyReuse: Duration.hours(1),
encryption: sqs.QueueEncryption.KMS_MANAGED
}
});
```
### Customizing Lambda Options
```typescript
const queue = new SqsToLambdaPropsEntity({
// ... other properties
lambdaOptions: {
memorySize: 512,
timeout: Duration.seconds(30),
runtime: Runtime.NODEJS_18_X
}
});
```
### Turning Off Components
```typescript
new FanoutConstruct(this, 'WebhooksFanout', new FanoutConstructPropsEntity({
stage: 'dev',
region: 'us-east-1',
sqsToLambda: [...],
removeApiGateway: true, // Don't create an API Gateway
removeApiGatewayKeyAuth: true, // Don't use API key authentication
removeLambda: true // Don't create Lambda functions
}));
```
## License
This project is licensed under the MIT License - see the [LICENSE](https://github.com/TheSmartMonkey/aws-cdk-fanout/blob/main/LICENSE) file for details.
## Testing
### Testing the Fanout Filtering
The construct supports filtering messages based on the `eventType` field. You can test this functionality using the provided test:
```typescript
import { LocalStackClient } from './tests/localstack';
// Initialize LocalStack
const localStackClient = await LocalStackClient.getInstance();
await localStackClient.initStack();
// Get the filter configurations
const sendEventFilterConfig = localStackClient.getFilterConfig('send-event');
const receiveEventFilterConfig = localStackClient.getFilterConfig('receive-event');
// Verify that messages are routed correctly based on eventType
const message = {
eventType: 'send',
data: { /* your payload */ }
};
// In a real scenario, you would send this message to the API Gateway endpoint:
// POST /send-event with the message as the body
```
When a message with `eventType: "send"` is sent to the API Gateway, it will be published to the SNS topic, and then filtered to the appropriate SQS queue that triggers the Lambda with the `send-event` name.
### Running E2E Tests
To run the end-to-end tests with LocalStack:
```bash
npm test tests/e2e/api-gateway.e2e.test.ts
```
This test verifies that messages with `eventType: "send"` are correctly routed to the Lambda function named `send-event`.