https://github.com/subhamay-cloudworks/0063-carnation-cft
This is a sample project to use map state in a state machine to loop through a list of events and invoke a Lambda to process them and store the erros events in a SQS queue. In the subsequest run the Lambda receives the error events from the SQS to retry the processing. The entire stack is created using AWS CloudFormation.
https://github.com/subhamay-cloudworks/0063-carnation-cft
aws-cloudformation aws-cloudformation-stack aws-dynamodb aws-python-lambda aws-sqs aws-step-function
Last synced: 2 months ago
JSON representation
This is a sample project to use map state in a state machine to loop through a list of events and invoke a Lambda to process them and store the erros events in a SQS queue. In the subsequest run the Lambda receives the error events from the SQS to retry the processing. The entire stack is created using AWS CloudFormation.
- Host: GitHub
- URL: https://github.com/subhamay-cloudworks/0063-carnation-cft
- Owner: subhamay-cloudworks
- License: mit
- Created: 2023-05-09T12:30:07.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2023-07-24T01:02:21.000Z (almost 2 years ago)
- Last Synced: 2025-01-18T05:43:40.807Z (4 months ago)
- Topics: aws-cloudformation, aws-cloudformation-stack, aws-dynamodb, aws-python-lambda, aws-sqs, aws-step-function
- Language: Python
- Homepage:
- Size: 43.9 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Project Carnation: A Demo Step Function with Map State and Error Retry
This is a sample project to use map state in a state machine to loop through a list of events and invoke a Lambda to process tham and store the erros events in a SQS queue. In the subsequest run the Lambda receives the error events from the SQS to retry the processing.
## Description
This sample state machine consists of two Lambdas, one SQS, one SNS and CloudWatch alarms for Lambda. The first Lambda generates some sample data with sequence number, random strings and generated Unix time stamp. The map state loops through the list generated by the first Lambda and processes them by inserting the records into a DynamoDB table. If any insertion fails then they are collected in a list and pushed into a SQS queue and the state machine fails. If all the generated data gets processed successfully the state machine ends in a success state. In the subsequent run of the state machine, the first lambda pulls the error events from the SQS (if any) and reprocesses them. If for any reason the processing fails even after third retry of the process, they are inserted into a dead letter queue.The entire stack is created using CloudFormation.
## Getting Started
### Dependencies
* Create a Customer Managed KMS Key in the region where you want to create the stack.
* Modify the KMS Key Policy to let the IAM user encrypt / decrypt using any resource using the created KMS Key.
### Installing
* Clone the repository.
* Create a S3 bucket to store the CFT and python code zip file.
* Create the folders - carnation/cft/nested-stacks, carnation/code/state-machine, carnation/code/python
* Upload the following YAML templates to carnation/cft/nested-stacks/
* cloudwatch-stack.yaml
* dynamodb-stack.yaml
* lambda-function-stack.yaml
* sqs-stack.yaml
* iam-role-stack.yaml
* sns-stack.yaml
* Upload the following YAML templates to carnation/cft/
* carnation-root-stack.yaml
* Upload state machine ASL to the folder cft/state-machine/
* Zip and Upload the Python files to carnation/code/python
* Create the entire using by using the root stack template carnation-root-stack.yaml by providing the required parameters.
### Executing program
* To execute the state machine use the following input:
```
{
"start_seq": 1,
"end_seq":
}
```
The above input will create a payload consisting of (end_seq - start_seq) elements.
In the subsequent run use the following input to just reprocess the events from the SQS queue:
```
{
"start_seq": 0,
"end_seq": 0
}
```
## Help
Post message in my blog (https://blog.subhamay.com)
## Authors
Contributors names and contact info
Subhamay Bhattacharyya - [[email protected]](https://subhamay.blog)
## Version History
* 0.1
* Initial Release
## License
This project is licensed under Subhamay Bhattacharyya. All Rights Reserved.
## Acknowledgments
* AWS [Dynamically process data with a Map state] (https://docs.aws.amazon.com/step-functions/latest/dg/sample-map-state.html)