{"id":23010107,"url":"https://github.com/dmdhrumilmistry/elb-log-analyzer","last_synced_at":"2025-04-02T16:15:23.942Z","repository":{"id":65774205,"uuid":"599244609","full_name":"dmdhrumilmistry/elb-log-analyzer","owner":"dmdhrumilmistry","description":"Project for analyzing logs from AWS ELB","archived":false,"fork":false,"pushed_at":"2023-11-11T14:41:24.000Z","size":287,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-08T07:08:56.148Z","etag":null,"topics":["amazon","aws","aws-s3","aws-security","cybersecurity","dmdhrumilmistry","elb-logs","python","python3","security"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmdhrumilmistry.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-02-08T18:45:25.000Z","updated_at":"2024-12-05T21:11:29.000Z","dependencies_parsed_at":"2023-11-11T15:30:37.065Z","dependency_job_id":"fe6061f5-69c2-4273-8d79-026beebb75ca","html_url":"https://github.com/dmdhrumilmistry/elb-log-analyzer","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmdhrumilmistry%2Felb-log-analyzer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmdhrumilmistry%2Felb-log-analyzer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmdhrumilmistry%2Felb-log-analyzer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmdhrumilmistry%2Felb-log-analyzer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmdhrumilmistry","download_url":"https://codeload.github.com/dmdhrumilmistry/elb-log-analyzer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246847137,"owners_count":20843444,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon","aws","aws-s3","aws-security","cybersecurity","dmdhrumilmistry","elb-logs","python","python3","security"],"created_at":"2024-12-15T09:16:57.550Z","updated_at":"2025-04-02T16:15:23.901Z","avatar_url":"https://github.com/dmdhrumilmistry.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ELB Log Analyzer\n\nTool for analyzing ELB logs for automating steps to retreive details of ip's user agent, total request count, to which urls requests were made along with their total count, and http methods in json format.\n\n## S3 Bucket Log Downloader\n\nDownloads S3 bucket objects that we created in specified time window.\n\n## Installation\n\n- Using Pip\n\n    ```bash\n    python3 -m pip install elb-log-analyzer\n    ```\n\n### AWS configuration\n\n- Create IAM policy with below configuration\n\n    ```json\n    {\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"S3ListSpecificDirectory\",\n            \"Effect\": \"Allow\",\n            \"Action\": \"s3:ListBucket\",\n            \"Resource\": \"arn:aws:s3:::alb-log-bucket-name\"\n        },\n        {\n            \"Sid\": \"S3GetSpecificDirectory\",\n            \"Effect\": \"Allow\",\n            \"Action\": \"s3:GetObject\",\n            \"Resource\": \"arn:aws:s3:::alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/*\"\n        }\n    ]\n    }\n    ```\n\n    \u003e **Note**: above policy will allow user to list all contents in the bucket but download objects only from `s3://alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/*`\n\n- Create AWS access keys\n\n- Use aws cli to configure access key for boto3\n\n    ```bash\n    aws configure\n    ```\n\n### S3 Bucket Log Downloader Usage\n\n- Print Help Menu.\n\n    ```bash\n    python3 -m elb_log_analyzer.s3_log -h\n    ```\n\n- Download all log files generated in 10 hours from now.\n\n    ```bash\n    python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' -H 10\n    ```\n\n- Download all log files generated in 40 mins from now.\n\n    ```bash\n    python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' -m 40\n    ```\n\n- Download all log files generated in 20 secs from now.\n\n    ```bash\n    python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' -s 20\n    ```\n\n- Download all log files generated in 10 hours, 40 mins and 20 secs from now and store in a directory.\n\n    ```bash\n    python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' --hours 10 --minutes 40 --seconds 20 -o './logs/downloads'\n    ```\n\n## Analyzer\n\nAnalyzes downloaded log files.\n\n### Analyzer Usage\n\n- Print Help Menu\n\n    ```bash\n    python3 -m elb_log_analyzer -h\n    ```\n\n- Print json data on console\n\n    ```bash\n    python3 -m elb_log_analyzer -i [INPUT_LOG_FILE_PATH]\n    ```\n\n- Store json data in a file\n\n    ```bash\n    python3 -m elb_log_analyzer -i [INPUT_LOG_FILE_PATH] -o [OUTPUT_FILE_PATH]\n    ```\n\n    \u003e **Note**: **INPUT_LOG_FILE_PATH** can be log file or a directory containing all log files ending with `.log` extension\n\n- Get IP details from IPAbuseDB\n\n    ```bash\n    python3 -m elb_log_analyzer -i [LOG_FILE_PATH] -t [REQUESTS_THRESHOLD_VALUE] -k [IP_ABUSE_DB_API_KEY] -o [OUTPUT_FILE_PATH]\n    ```\n\n## Alerts\n\nSend alert to slack channel with abusive ip details.\n\n### Usage\n\n- Send alert from analyzed file\n\n    ```bash\n    python elb_log_analyzer.alerts -w [SLACK_WEBHOOK] -f [ANALYZED_LOG_FILE_LOCATION]\n    ```\n\n## Dashboard\n\nDashboard to visualize data.\n\n### Dashboard Installation\n\n- Install requirements\n\n    ```bash\n    python3 -m pip install dashboard/requirements.txt\n    ```\n\n### Usage\n\n- Start App\n\n    ```bash\n    streamlit run dashboard/app.py\n    ```\n\n- Enter Log File/Directory Path\n\n## Publish package to pypi\n\n- Using poetry\n\n    ```bash\n    python3 -m poetry publish --build --username [PYPI_USERNAME] --password [PYPI_PASSWORD]\n    ```\n\n## Usage Summary\n\n- Download log files\n\n    ```bash\n    python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' -H [HOURS] -o logs\n    ```\n\n- Analyze Log Files\n\n    ```bash\n    python3 -m elb_log_analyzer -i logs -o log.json -t [REQUEST_THRESHOLD] -k [IP_ABUSE_API_KEY] \n    ```\n\n- Send Alert to slack with client ips having total number of requests greater than threshold requests\n\n    ```bash\n    python -m elb_log_analyzer.alerts -w [SLACK_WEBHOOK] -f [ANALYZED_LOG_FILE_LOCATION]\n    ```\n\n- Visualize Analyzed Logs using Dashboard\n\n    ```bash\n    streamlit run dashboard/app.py\n    ```\n\n## Docker WorkFlow\n\n- Pull image\n\n    ```bash\n    docker pull dmdhrumilmistry/elb-log-analyzer\n    ```\n\n- Create an `.env` file\n\n    ```bash\n    # bucket configuration\n    BUCKET_NAME='elb-logs-bucket-name'\n    BUCKET_PREFIX='AWSLogs/XXXXXXXX/elasticloadbalancing/eu-west-2/'\n\n    # SECRETS conf\n    REQUESTS_THRESHOLD=400\n    IP_ABUSE_DB_API_KEY='UPDATE_HERE'\n    SLACK_WEBHOOK='UPDATE_HERE'\n\n    # consts\n    DATE_SUFFIX=\"$(date '+%Y/%m/%d')\"\n    LOG_ANALYSIS_INTERVAL=5\n    ```\n\n- Start Container\n\n    ```bash\n    docker run --env-file .env --rm dmdhrumilmistry/elb-log-analyzer\n    ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmdhrumilmistry%2Felb-log-analyzer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmdhrumilmistry%2Felb-log-analyzer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmdhrumilmistry%2Felb-log-analyzer/lists"}