{"id":19056279,"url":"https://github.com/akarce/elk-stack-mastery","last_synced_at":"2025-07-31T18:31:35.657Z","repository":{"id":261146004,"uuid":"873914985","full_name":"akarce/elk-stack-mastery","owner":"akarce","description":"A comprehensive project focusing on setting up and configuring the Elastic Stack (Elasticsearch, Logstash, and Kibana) for efficient log management and analytics. This project includes Elasticsearch configurations, Logstash pipelines, and Kibana visualizations, with detailed step-by-step documentation.","archived":false,"fork":false,"pushed_at":"2024-11-04T23:14:49.000Z","size":467,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-02T11:29:32.444Z","etag":null,"topics":["dataanalytics","datapipeline","devops","elasticsearch","elasticstack","elkstack","kibana","logging","logmanagement","logstash","monitoring","opensource","systemmonitoring","virtualbox","visualization"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akarce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-17T00:08:25.000Z","updated_at":"2024-12-23T12:34:48.000Z","dependencies_parsed_at":"2024-11-05T00:20:42.218Z","dependency_job_id":"775582ea-e789-4a84-90ed-2486b430f089","html_url":"https://github.com/akarce/elk-stack-mastery","commit_stats":null,"previous_names":["akarce/elk-stack-mastery"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akarce%2Felk-stack-mastery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akarce%2Felk-stack-mastery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akarce%2Felk-stack-mastery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akarce%2Felk-stack-mastery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akarce","download_url":"https://codeload.github.com/akarce/elk-stack-mastery/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240112349,"owners_count":19749632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataanalytics","datapipeline","devops","elasticsearch","elasticstack","elkstack","kibana","logging","logmanagement","logstash","monitoring","opensource","systemmonitoring","virtualbox","visualization"],"created_at":"2024-11-08T23:48:58.499Z","updated_at":"2025-02-22T01:26:22.552Z","avatar_url":"https://github.com/akarce.png","language":null,"readme":"# ELK Stack Mastery: Building a Scalable Log Management System\n\n## Youtube Tutorial\n[![Watch the tutorial](https://img.youtube.com/vi/U1i5sIZzEQM/0.jpg)](https://youtu.be/U1i5sIZzEQM)\n\n\n## Overview\n\nThis project sets up an Elastic Cluster with 3 nodes using Virtualbox virtual machines. It includes the setup of Elasticsearch, Logstash, and Kibana (ELK stack) for log management and analysis.\n\n## Project Goals\n\n- Set up Elastic Cluster with all necessary components.\n- Create an index with a retention period of 10 days in Hot, 10 days in Cold, and 10 days in Frozen tiers.\n- Load logs using one of the methods listed in the setup.\n- Create a Dashboard with drilldown capabilities.\n\n## Prerequisites\n\n- VirtualBox installed on your system\n- Debian 12 ISO image\n- Sufficient system resources to run 3 VMs\n\n## VM Configuration\n\nCreate 3 VMs with the following specifications:\n\n1. **elktest1** (Master + Data_Hot + Data_Content, Kibana, Logstash)\n    - 8 GB RAM, 4 CPU, 40 GB storage\n2. **elktest2** (Data_Cold, Logstash)\n    - 8 GB RAM, 4 CPU, 40 GB storage\n3. **elktest3** (Data_Frozen, Logstash)\n    - 6 GB RAM, 3 CPU, 50 GB storage\n\n## Setup Instructions\n\n### 1. VM Installation\n\n1. Download Debian 12 ISO:\n    \n    ```\n    \u003chttps://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.7.0-amd64-netinst.iso\u003e\n    ```\n    \n2. Install Debian on each VM.\n3. In VM settings, change network from NAT to Bridged Adapter.\n\n### 2. SSH Setup\n\nInstall SSH on each VM:\n\n```bash\nsu -\napt-get update\napt-get install openssh-server\nsystemctl start ssh\nsystemctl enable ssh\n```\n\nReboot and get IP addresses:\n\n```bash\nreboot now\nip addr show\n```\n\nConnect from host machine:\n\n```bash\nssh \u003cusername\u003e@\u003cyour_ip_address\u003e\n```\n\n### 3. Elasticsearch Installation\n\nOn all VMs:\n\n```bash\napt install curl\ncurl -fsSL \u003chttps://artifacts.elastic.co/GPG-KEY-elasticsearch\u003e | gpg --dearmor -o /usr/share/keyrings/elastic.gpg\necho \"deb [signed-by=/usr/share/keyrings/elastic.gpg] \u003chttps://artifacts.elastic.co/packages/8.x/apt\u003e stable main\" | tee -a /etc/apt/sources.list.d/elastic-8.x.list\napt update\napt install elasticsearch\n```\n\n### 4. Elasticsearch Configuration\n\n### On elktest1:\n\n1. Edit `/etc/elasticsearch/elasticsearch.yml`:\n    \n    ```yaml\n    cluster.name: elktestcluster\n    node.name: elktest1\n    node.roles: [\"master\",\"data_hot\",\"data_content\"]\n    cluster.initial_master_nodes: [\"elktest1\"]\n    path.data: /var/lib/elasticsearch\n    path.logs: /var/log/elasticsearch\n    network.host: 0.0.0.0\n    http.port: 9200\n    discovery.seed_hosts: [\"elktest1\"]\n    xpack.security.enabled: true\n    xpack.security.enrollment.enabled: true\n    xpack.security.http.ssl:\n      enabled: true\n      keystore.path: certs/http.p12\n    xpack.security.transport.ssl:\n      enabled: true\n      verification_mode: certificate\n      keystore.path: certs/transport.p12\n      truststore.path: certs/transport.p12\n    http.host: 0.0.0.0\n    ```\n    \n2. Start Elasticsearch:\n    \n    ```bash\n    systemctl start elasticsearch\n    ```\n    \n3. Reset elastic user password:\n    \n    ```bash\n    /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic\n    ```\n    \n4. Generate enrollment tokens for other nodes:\n    \n    ```bash\n    cd /usr/share/elasticsearch/bin\n    ./elasticsearch-create-enrollment-token -s node\n    ```\n    \n\n### On elktest2 and elktest3:\n\n1. Reconfigure node with enrollment token:\n    \n    ```bash\n    cd /usr/share/elasticsearch/bin\n    ./elasticsearch-reconfigure-node --enrollment-token \u003cyour_enrollment_token\u003e\n    ```\n    \n2. Edit `/etc/elasticsearch/elasticsearch.yml`:\n    \n    For elktest2:\n    \n    ```yaml\n    cluster.name: elktestcluster\n    node.name: elktest2\n    node.roles: [\"data_cold\"]\n    path.data: /var/lib/elasticsearch\n    path.logs: /var/log/elasticsearch\n    network.host: 0.0.0.0\n    http.port: 9200\n    ```\n    \n    For elktest3:\n    \n    ```yaml\n    cluster.name: elktestcluster\n    node.name: elktest3\n    node.roles: [\"data_frozen\"]\n    path.data: /var/lib/elasticsearch\n    path.logs: /var/log/elasticsearch\n    network.host: 0.0.0.0\n    http.port: 9200\n    xpack.searchable.snapshot.shared_cache.size: 30%\n    ```\n    \n3. Start Elasticsearch on both nodes:\n    \n    ```bash\n    systemctl start elasticsearch\n    ```\n    \n\n### 5. Index Lifecycle Management\n\nCreate ILM policy:\n\n```bash\nPUT _ilm/policy/elktestcluster_logs_policy\n{\n    \"policy\": {\n        \"phases\": {\n            \"hot\": {\n                \"actions\": {\n                    \"rollover\": {\n                        \"max_size\": \"40gb\",\n                        \"max_age\": \"10d\"\n                    }\n                }\n            },\n            \"warm\": {\n                \"min_age\": \"10d\",\n                \"actions\": {\n                    \"forcemerge\": {\n                        \"max_num_segments\": 1\n                    },\n                    \"allocate\": {\n                        \"require\": {\n                            \"data\": \"cold\"\n                        }\n                    }\n                }\n            },\n            \"cold\": {\n                \"min_age\": \"20d\",\n                \"actions\": {\n                    \"freeze\": {},\n                    \"allocate\": {\n                        \"require\": {\n                            \"data\": \"frozen\"\n                        }\n                    }\n                }\n            }\n        }\n    }\n}\n```\n\nAssign policy to index template:\n\n```bash\nPUT _index_template/elktestcluster_logs_template\n{\n  \"index_patterns\": [\"elktestcluster-logs-*\"],\n  \"template\": {\n    \"settings\": {\n      \"number_of_shards\": 1,\n      \"number_of_replicas\": 1,\n      \"index.lifecycle.name\": \"elktestcluster_logs_policy\",\n      \"index.lifecycle.rollover_alias\": \"elktestcluster-logs\"\n    }\n  }\n}\n```\n\n### 6. Logstash Setup\n\nInstall Logstash on all VMs:\n\n```bash\napt install logstash -y\n```\n\nAdd logstash user to elasticsearch group:\n\n```bash\nsudo usermod -aG elasticsearch logstash\n```\n\nCreate Logstash pipeline configuration:\n\n```bash\nnano /etc/logstash/conf.d/elktestcluster-logs.con\n```\n\nAdd the following content:\n\n```\ninput {\n  file {\n    path =\u003e [\n      \"/var/log/elasticsearch/elktestcluster*.json\"\n    ]\n    start_position =\u003e \"beginning\"\n    sincedb_path =\u003e \"/dev/null\"\n    codec =\u003e \"json\"\n  }\n}\n\noutput {\n  elasticsearch {\n    hosts =\u003e [\"\u003chttps://elktest1:9200\u003e\", \"\u003chttps://elktest2:9200\u003e\", \"\u003chttps://elktest3:9200\u003e\"]\n    index =\u003e \"elktestcluster-logs-%{+YYYY.MM.dd}\"\n    user =\u003e \"elastic\"\n    password =\u003e \"elastic\"\n    ssl =\u003e true\n    cacert =\u003e \"/etc/elasticsearch/certs/http_ca.crt\"\n  }\n}\n```\n\nStart Logstash on all VMs:\n\n```bash\nsystemctl start logstash\n```\n\n### 7. Kibana Setup\n\nInstall Kibana on one VM (preferably elktest1 or elktest2):\n\n```bash\napt install kibana -y\n```\n\nReset kibana_system user password:\n\n```bash\n/usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u kibana_system\n```\n\nConfigure Kibana:\n\n```bash\nnano /etc/kibana/kibana.yml\n```\n\nAdd/edit the following:\n\n```yaml\nserver.port: 5601\nserver.host: \"0.0.0.0\"\nelasticsearch.hosts: [\"\u003chttps://elktest1:9200\u003e\", \"\u003chttps://elktest2:9200\u003e\", \"\u003chttps://elktest3:9200\u003e\"]\nelasticsearch.username: \"kibana_system\"\nelasticsearch.password: \"kibana\"\nelasticsearch.ssl.verificationMode: none\n```\n\nStart Kibana:\n\n```bash\nsystemctl start kibana\n```\n\n### 8. Accessing Kibana\n\nOpen a web browser and go to:\n\n```\nhttp://\u003cyour_kibana_machine_ip\u003e:5601\n```\n\nUse the Elasticsearch credentials:\n\n- Username: elastic\n- Password: elastic\n\n## Final Steps\n\n1. Create a data view from cluster logs in Kibana.\n2. Create a dashboard from the data view.\n\n![Kibana Snapshot](kibana_snapshot.png)\n\nCongratulations! You have now set up a complete ELK stack for log management and analysis.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakarce%2Felk-stack-mastery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakarce%2Felk-stack-mastery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakarce%2Felk-stack-mastery/lists"}