https://github.com/jayesh-tanna/aws-sqs-lambda-s3-example
Download html content of URL.
https://github.com/jayesh-tanna/aws-sqs-lambda-s3-example
aws-lambda aws-s3 aws-sdk aws-sqs cloudwatch-logs dotnet-core json web-api
Last synced: 7 months ago
JSON representation
Download html content of URL.
- Host: GitHub
- URL: https://github.com/jayesh-tanna/aws-sqs-lambda-s3-example
- Owner: jayesh-tanna
- Created: 2019-09-15T10:22:36.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2022-12-08T06:06:14.000Z (almost 3 years ago)
- Last Synced: 2025-01-14T09:55:55.032Z (9 months ago)
- Topics: aws-lambda, aws-s3, aws-sdk, aws-sqs, cloudwatch-logs, dotnet-core, json, web-api
- Language: C#
- Homepage: https://factorycode.wordpress.com
- Size: 15.6 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# aws-sqs-lambda-s3-example
Problem:
We want to save html content of entered url. E.g. Google.com then html content of this page should be saved. Solution should be scalable. Should handle huge traffic.
Solution projects / components:
1. Api: It will receive request from consumer.
2. Sqs: Api will send JSON seriallized message to AWS sqs.
3. Lambda (.net core): Lambda function will be executed as soon as new message arrives in sqs.
4. Tester: It will make call to Api.
Api will send message to sqs. Lambda wil be triggered as soon as message arrives in sqs. Lambda function will deseriallize message and download the content from the entered url. Same lambda function will upload the content to S3 bucket. Logs are written in CloudWatch from lambda function.
Prerequisite:
VS 2017/19, AWS toolkit, AWS account
Notes:
Need to create AWS account. Proper IAM user along with enough role to execute SQS, Lambda and S3 service.
AWS services:
Lambda function,
S3 bucket,
SQS,
Cloudwatch,
IAM roles
Nuget packages:
AWSSDK.SQS,
Newtonsoft.Json,
AWSSDK.S3,
Microsoft.AspNet.WebApi.Client