https://github.com/tejeshkumarreddyc/log-processing-of-webserver-logs-aws

Logs of a web server such as apache are logged in cloudwatch and processed using subscription filter & ultimately error logs are stored in s3 via lambda for further analysis
https://github.com/tejeshkumarreddyc/log-processing-of-webserver-logs-aws

aws cloudwatch-logs ec2-instance event-driven httpd-server iam lambda-functions logprocess rest-api s3-storage sdk-python shell-script sns

Last synced: 3 months ago
JSON representation

Logs of a web server such as apache are logged in cloudwatch and processed using subscription filter & ultimately error logs are stored in s3 via lambda for further analysis

• Host: GitHub
• URL: https://github.com/tejeshkumarreddyc/log-processing-of-webserver-logs-aws
• Owner: TejeshKumarReddyC
• Created: 2025-01-25T18:33:59.000Z (8 months ago)
• Default Branch: master
• Last Pushed: 2025-01-31T18:44:47.000Z (8 months ago)
• Last Synced: 2025-03-28T20:15:47.783Z (6 months ago)
• Topics: aws, cloudwatch-logs, ec2-instance, event-driven, httpd-server, iam, lambda-functions, logprocess, rest-api, s3-storage, sdk-python, shell-script, sns
• Language: Python
• Homepage:
• Size: 7.81 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          

**Workflow:-**

Webserver_logs(EC2) ---> Log_group(Cloudwatch) ---> Lambda ---> S3_Bucket & Notification (SNS)

-->**Introduction**

In this project we're gonna perform log processing of a web server using cloudwatch logs, lambda, s3 and subscription filters.

-->**Installation of a simple http server in EC2**

sudo yum update -y

sudo yum install httpd -y

systemctl start httpd

systemctl enable httpd

-->**AWS CLI installation**

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

unzip awscliv2.zip

sudo ./aws/install

aws configure

-->**Cloudwatch agent installation in EC2**

yum install amazon-cloudwatch-agent -y

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

systemctl enable amazon-cloudwatch-agent

systemctl start amazon-cloudwatch-agent

-->**Create an S3 BUCKET**

-->**Create a subscription filter using the below pattern**

filter pattern 

    403 ( Filters the 403 errors of logs of the web server from the cloudwatch ).

destination

    Lambda function ( It'll be invoked by the cloudwatch and those error messages of web server are stored in S3 for further analysis).

-->**Create a rest API and link it to the Lambda**

  User can access those logs from S3 via API gateway using the API endpoint.