https://github.com/gakas14/event-driven-architecture
Event driven Architecture with S3, SNS, SQS, and Lambda. Uses Events to decouple an application's components.
https://github.com/gakas14/event-driven-architecture
aws-lambda aws-s3 sns-topic sqs-queue
Last synced: 3 months ago
JSON representation
Event driven Architecture with S3, SNS, SQS, and Lambda. Uses Events to decouple an application's components.
- Host: GitHub
- URL: https://github.com/gakas14/event-driven-architecture
- Owner: gakas14
- Created: 2023-12-06T07:14:35.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2023-12-06T14:43:09.000Z (almost 2 years ago)
- Last Synced: 2024-12-31T12:34:05.819Z (9 months ago)
- Topics: aws-lambda, aws-s3, sns-topic, sqs-queue
- Homepage:
- Size: 4.88 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Event-Driven-Architecture
Event driven Architecture with S3, SNS, SQS, and Lambda. Uses Events to decouple an application's components.
- loosely coupled applications
- add new features without changing existing applications
- Scale and fail components independently.
I. Create the s3 bucket
II. Create an SNS topic
- create the topic and change the access policy: copy your account number, s3 bucket name, and SQS queues.
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__default_statement_ID",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:::",
"Condition": {
"StringEquals": {
"aws:SourceAccount": ""
},
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:"
}
}
},
{
"Sid": "sqs_statement",
"Effect": "Allow",
"Principal": {
"Service": "sqs.amazonaws.com"
},
"Action": "sns:Subscribe",
"Resource": "arn:aws:sns:::",
"Condition": {
"ArnEquals": {
"aws:SourceArn": [
"arn:aws:sqs:::",
"arn:aws:sqs:::"
]
}
}
}
]
}
III. Create SQS Quere
- event stream: create the queue and change the access policy:
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "Stmt1234",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": [
"sqs:ReceiveMessage",
"sqs:sendMessage"
],
"Resource": "arn:aws:sqs:::",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:lambda:::"
}
}
},
{
"Sid": "Stmt12345",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:::",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:sns:::"
}
}
}
]
IV. Create the Lambda Functions
- create two lambda policies to allow access to SQS and CloudWatch
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"sqs:DeleteMessage",
"logs:CreateLogStream",
"sqs:ReceiveMessage",
"sqs:GetQueueAttributes",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:sqs:::q1",
"arn:aws:logs:::log-group:/aws/lambda/:*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"sqs:ReceiveMessage",
"logs:CreateLogGroup"
],
"Resource": [
"arn:aws:logs:::*",
"arn:aws:sqs:::"
]
}
]
}
- Create two roles for each lambda function and then attach each policy to the corresponding lambda role.
- Create the two lambda functions, and let's print the data
V. Connect the four components
- connect the s3 bucket to the SNS by creating an event notification at the s3 bucket level: select "All object create events" as the event type and the SNS topic as a destination
- connect SQS and the SNS by creating a subscription at the SNS level (with a filter policy). NB: create two subscriptions for each queue.
The filter policy: we filter the message based on the event name:
The put event will be forwarded to queue 1
{
"Records": {
"eventName": [
"ObjectCreated:Put"
]
}
}
The copy event will be forwarded to queue 2
{
"Records": {
"eventName": [
"ObjectCreated:Copy"
]
}
}
- Finally, connect SQS queues to the lambda functions at the SQS level.
VI. Test Setup
- put an object in the s3 bucket, and let's check the Cloudwatch log into the lambda function 1
- 
- copy the same file to check the second lambda log
