https://github.com/dmdhrumilmistry/elb-log-analyzer
Project for analyzing logs from AWS ELB
https://github.com/dmdhrumilmistry/elb-log-analyzer
amazon aws aws-s3 aws-security cybersecurity dmdhrumilmistry elb-logs python python3 security
Last synced: about 1 year ago
JSON representation
Project for analyzing logs from AWS ELB
- Host: GitHub
- URL: https://github.com/dmdhrumilmistry/elb-log-analyzer
- Owner: dmdhrumilmistry
- License: mit
- Created: 2023-02-08T18:45:25.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-11-11T14:41:24.000Z (over 2 years ago)
- Last Synced: 2025-02-08T07:08:56.148Z (over 1 year ago)
- Topics: amazon, aws, aws-s3, aws-security, cybersecurity, dmdhrumilmistry, elb-logs, python, python3, security
- Language: Python
- Homepage:
- Size: 280 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# ELB Log Analyzer
Tool for analyzing ELB logs for automating steps to retreive details of ip's user agent, total request count, to which urls requests were made along with their total count, and http methods in json format.
## S3 Bucket Log Downloader
Downloads S3 bucket objects that we created in specified time window.
## Installation
- Using Pip
```bash
python3 -m pip install elb-log-analyzer
```
### AWS configuration
- Create IAM policy with below configuration
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "S3ListSpecificDirectory",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::alb-log-bucket-name"
},
{
"Sid": "S3GetSpecificDirectory",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/*"
}
]
}
```
> **Note**: above policy will allow user to list all contents in the bucket but download objects only from `s3://alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/*`
- Create AWS access keys
- Use aws cli to configure access key for boto3
```bash
aws configure
```
### S3 Bucket Log Downloader Usage
- Print Help Menu.
```bash
python3 -m elb_log_analyzer.s3_log -h
```
- Download all log files generated in 10 hours from now.
```bash
python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' -H 10
```
- Download all log files generated in 40 mins from now.
```bash
python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' -m 40
```
- Download all log files generated in 20 secs from now.
```bash
python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' -s 20
```
- Download all log files generated in 10 hours, 40 mins and 20 secs from now and store in a directory.
```bash
python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' --hours 10 --minutes 40 --seconds 20 -o './logs/downloads'
```
## Analyzer
Analyzes downloaded log files.
### Analyzer Usage
- Print Help Menu
```bash
python3 -m elb_log_analyzer -h
```
- Print json data on console
```bash
python3 -m elb_log_analyzer -i [INPUT_LOG_FILE_PATH]
```
- Store json data in a file
```bash
python3 -m elb_log_analyzer -i [INPUT_LOG_FILE_PATH] -o [OUTPUT_FILE_PATH]
```
> **Note**: **INPUT_LOG_FILE_PATH** can be log file or a directory containing all log files ending with `.log` extension
- Get IP details from IPAbuseDB
```bash
python3 -m elb_log_analyzer -i [LOG_FILE_PATH] -t [REQUESTS_THRESHOLD_VALUE] -k [IP_ABUSE_DB_API_KEY] -o [OUTPUT_FILE_PATH]
```
## Alerts
Send alert to slack channel with abusive ip details.
### Usage
- Send alert from analyzed file
```bash
python elb_log_analyzer.alerts -w [SLACK_WEBHOOK] -f [ANALYZED_LOG_FILE_LOCATION]
```
## Dashboard
Dashboard to visualize data.
### Dashboard Installation
- Install requirements
```bash
python3 -m pip install dashboard/requirements.txt
```
### Usage
- Start App
```bash
streamlit run dashboard/app.py
```
- Enter Log File/Directory Path
## Publish package to pypi
- Using poetry
```bash
python3 -m poetry publish --build --username [PYPI_USERNAME] --password [PYPI_PASSWORD]
```
## Usage Summary
- Download log files
```bash
python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' -H [HOURS] -o logs
```
- Analyze Log Files
```bash
python3 -m elb_log_analyzer -i logs -o log.json -t [REQUEST_THRESHOLD] -k [IP_ABUSE_API_KEY]
```
- Send Alert to slack with client ips having total number of requests greater than threshold requests
```bash
python -m elb_log_analyzer.alerts -w [SLACK_WEBHOOK] -f [ANALYZED_LOG_FILE_LOCATION]
```
- Visualize Analyzed Logs using Dashboard
```bash
streamlit run dashboard/app.py
```
## Docker WorkFlow
- Pull image
```bash
docker pull dmdhrumilmistry/elb-log-analyzer
```
- Create an `.env` file
```bash
# bucket configuration
BUCKET_NAME='elb-logs-bucket-name'
BUCKET_PREFIX='AWSLogs/XXXXXXXX/elasticloadbalancing/eu-west-2/'
# SECRETS conf
REQUESTS_THRESHOLD=400
IP_ABUSE_DB_API_KEY='UPDATE_HERE'
SLACK_WEBHOOK='UPDATE_HERE'
# consts
DATE_SUFFIX="$(date '+%Y/%m/%d')"
LOG_ANALYSIS_INTERVAL=5
```
- Start Container
```bash
docker run --env-file .env --rm dmdhrumilmistry/elb-log-analyzer
```