{"id":17767266,"url":"https://github.com/salaah01/almost-free-elk-analytics","last_synced_at":"2026-04-19T03:03:21.675Z","repository":{"id":258780006,"uuid":"875669452","full_name":"Salaah01/almost-free-elk-analytics","owner":"Salaah01","description":"A cost-effective way to get logs over to ELK for analytics. Works by syncing nginx logs from a remote server to an S3 bucket and then locally loading those logs into ELK.","archived":false,"fork":false,"pushed_at":"2024-10-21T01:28:47.000Z","size":11,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-14T11:47:29.722Z","etag":null,"topics":["analytics","aws-s3","elk","elk-stack","nginx"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Salaah01.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-20T15:33:32.000Z","updated_at":"2024-10-21T01:28:51.000Z","dependencies_parsed_at":"2024-10-26T21:14:18.819Z","dependency_job_id":"57c5ef37-2890-40ad-92db-a31eadb11c67","html_url":"https://github.com/Salaah01/almost-free-elk-analytics","commit_stats":null,"previous_names":["salaah01/almost-free-elk-analytics"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Salaah01/almost-free-elk-analytics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2Falmost-free-elk-analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2Falmost-free-elk-analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2Falmost-free-elk-analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2Falmost-free-elk-analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Salaah01","download_url":"https://codeload.github.com/Salaah01/almost-free-elk-analytics/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2Falmost-free-elk-analytics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31992822,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T20:23:30.271Z","status":"online","status_checked_at":"2026-04-19T02:00:07.110Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","aws-s3","elk","elk-stack","nginx"],"created_at":"2024-10-26T20:44:43.534Z","updated_at":"2026-04-19T03:03:21.653Z","avatar_url":"https://github.com/Salaah01.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Almost Free ELK Analytics\n\nThis repository provides a simple, cost-effective ELK stack setup for analysing NGINX logs from a remote server on your local machine. By keeping everything local, there's no need for expensive third-party services or provisioning cloud servers to run ELK.\n\nAt present, this setup only works with NGINX logs, giving you the ability to analyse and visualise them directly in Kibana.\n\n## Contents\n- [Almost Free ELK Analytics](#almost-free-elk-analytics)\n  - [Contents](#contents)\n  - [Why?](#why)\n  - [How?](#how)\n  - [Limitations](#limitations)\n  - [Setup](#setup)\n    - [AWS](#aws)\n    - [Remote Server](#remote-server)\n      - [Prerequisites](#prerequisites)\n    - [Local Machine](#local-machine)\n      - [Prerequisites](#prerequisites-1)\n  - [Adhoc Runs](#adhoc-runs)\n  - [Further Development](#further-development)\n  - [Issues](#issues)\n\n\n\n## Why?\n\nI wanted to gain deeper insights into my NGINX logs to understand traffic patterns and identify errors, especially when things go wrong. However, I didn’t want to rely on expensive third-party log analysis services, nor did I want to deploy a full ELK stack on my server.\n\nThis project aims to provide a cost-effective solution for log analysis that runs entirely on your local machine, but processes logs from a remote server.\n\n## How?\n\n\nThe server generates NGINX logs, which are periodically uploaded to an S3 bucket via a cron job. When you want to analyse these logs, a script is run locally to download the logs from S3 and process them using Logstash. The processed logs are then stored in Elasticsearch and visualized in Kibana.\n\n\n```mermaid\nflowchart LR\n    subgraph remote_server[Remote Server]\n        nginx[Nginx] --\u003e web_app[Web App]\n        nginx --\u003e remote_logs[Logs]\n    end\n\n    remote_logs -. Push .-\u003e s3[AWS S3] -. Pull (Sync) .-\u003e local_logs_vol[Logs]\n\n    subgraph local_machine[Local Machine]\n        local_logs_vol --\u003e docker\n        subgraph docker[Docker]\n            logstach[Logstach] --\u003e elasticsearch[Elasticsearch] --\u003e kibana[Kibana]\n        end\n    end\n```\n\n## Limitations\n\nCurrently, this setup is designed to work only with NGINX logs.\nThe logs must be stored as files on the remote server.\nOnly logs stored in an S3 bucket are supported.\nThe setup assumes the default NGINX log format, so any customisations may require changes to the Logstash configuration.\n\n## Setup\n\n### AWS\n\n1. Create an S3 bucket to store the logs.\n2. Configure the remote server with permissions to write to the S3 bucket. Example policy:\n\n```json\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"\u003c\u003c Some identifier \u003e\u003e\",\n            \"Effect\": \"Allow\",\n            \"Action\": \"s3:PutObject\",\n            \"Resource\": \"arn:aws:s3:::\u003c\u003c bucket_name \u003e\u003e/*\"\n        },\n        {\n            \"Sid\": \"\u003c\u003c Some indentifier \u003e\u003e\",\n            \"Effect\": \"Allow\",\n            \"Action\": \"s3:ListBucket\",\n            \"Resource\": \"arn:aws:s3:::\u003c\u003c bucket_name \u003e\u003e\"\n        }\n    ]\n}\n```\n\n3. Grant your local machine the necessary permissions to read from the S3 bucket:\n\n```json\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"\u003c\u003c Some identifier \u003e\u003e\",\n            \"Effect\": \"Allow\",\n            \"Action\": \"s3:GetObject\",\n            \"Resource\": \"arn:aws:s3:::\u003c\u003c bucket_name \u003e\u003e/*\"\n        },\n        {\n            \"Sid\": \"\u003c\u003c Some identifier \u003e\u003e\",\n            \"Effect\": \"Allow\",\n            \"Action\": \"s3:ListBucket\",\n            \"Resource\": \"arn:aws:s3:::\u003c\u003c bucket_name \u003e\u003e\"\n        }\n    ]\n}\n```\n\n### Remote Server\n\n#### Prerequisites\n\n* NGINX installed and configured to log to a file (default location is /var/log/nginx/access.log).\n* AWS CLI installed and configured for S3 access.\n\n1. Configure NGINX to log to a file. If the log location differs from the default, update the `push_logs_to_s3.sh` script accordingly.\n2. Copy the `push_logs_to_s3.sh` script to your remote server.\n3. Create a cron job to run the `push_logs_to_s3.sh` script. Below is an example cron job that runs every 3 hours:\n\n```bash\n0 */3 * * * /path/to/push_logs_to_s3.sh \u003cbucket_name\u003e\n```\n\n### Local Machine\n\n#### Prerequisites\n\n* Docker\n* Docker Compose\n* AWS CLI\n\n1. Clone this repository.\n2. Create a cron job to run the `pull_logs_from_s3.sh` script. Below is an example cron job that runs every 3 hours:\n\n```bash\n0 */3 * * * /path/to/pull_logs_from_s3.sh \u003cbucket_name\u003e\n```\n\n3. Start the ELK stack by running:\n\n```bash\ndocker-compose up\n```\n4. Navigate to `http://localhost:5601` to access Kibana.\n5. Click on the `Discover` tab and create an index pattern. Use `nginx-logs*` as the index pattern.\n6. You should now be able to visualise the NGINX logs in Kibana.\n\n## Adhoc Runs\n\nTo run the process manually, you can execute the `push_logs_to_s3.sh` script on your remote server and the `pull_logs_from_s3.sh` script on your local machine.\n\n## Further Development\n\nCurrently, this project is a basic setup that meets my needs. Future enhancements could include:\n\n* Supporting logs from other web servers beyond NGINX.\n* Adding support for additional log storage formats (besides S3).\n* Further automation of the log processing pipeline.\n\nIf you have ideas or suggestions, feel free to submit a pull request or open an issue. Contributions are welcome!\n\n## Issues\n\nIf you encounter any problems, please create an issue. This setup is a quick solution I developed over a few hours, so it may not be production-ready. I'll do my best to help, but bear in mind it's not designed for large-scale production environments.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalaah01%2Falmost-free-elk-analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalaah01%2Falmost-free-elk-analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalaah01%2Falmost-free-elk-analytics/lists"}