{"id":22442191,"url":"https://github.com/colinbut/centralized-logging-with-elastic-stack","last_synced_at":"2025-07-03T14:07:51.975Z","repository":{"id":69158006,"uuid":"175708444","full_name":"colinbut/centralized-logging-with-elastic-stack","owner":"colinbut","description":"Setting up the Elastic Stack https://www.elastic.co/products/","archived":false,"fork":false,"pushed_at":"2019-03-17T17:07:24.000Z","size":3083,"stargazers_count":1,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-11T11:44:49.354Z","etag":null,"topics":["beats","centralized-logging","elasticsearch","elasticstack","filebeat","filebeats","heartbeat","kibana","log-management","logging","logstash","metricbeat"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/colinbut.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-14T22:19:00.000Z","updated_at":"2025-02-20T23:10:23.000Z","dependencies_parsed_at":null,"dependency_job_id":"352f0b49-b41f-4d81-93ce-f550474ab163","html_url":"https://github.com/colinbut/centralized-logging-with-elastic-stack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/colinbut/centralized-logging-with-elastic-stack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colinbut%2Fcentralized-logging-with-elastic-stack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colinbut%2Fcentralized-logging-with-elastic-stack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colinbut%2Fcentralized-logging-with-elastic-stack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colinbut%2Fcentralized-logging-with-elastic-stack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/colinbut","download_url":"https://codeload.github.com/colinbut/centralized-logging-with-elastic-stack/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colinbut%2Fcentralized-logging-with-elastic-stack/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263339941,"owners_count":23451518,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beats","centralized-logging","elasticsearch","elasticstack","filebeat","filebeats","heartbeat","kibana","log-management","logging","logstash","metricbeat"],"created_at":"2024-12-06T02:18:24.622Z","updated_at":"2025-07-03T14:07:51.943Z","avatar_url":"https://github.com/colinbut.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Centralized Logging with Elastic Stack\n\n## Table of Contents\n\n* [Preface](#preface)\n* [The Elastic Stack](#the-elastic-stack)\n* [Stack Overview](#stack-overview)\n* [Setup](#setup)\n\t* [Download](#download)\n\t* [ElasticSearch](#elasticsearch)\n    * [Logstash](#logstash)\n    * [Filebeat](#filebeat)\n    * [Metricbeat](#metricbeat)\n    * [Heartbeat](#heartbeat)\n    * [Kibana](#kibana)\n        * [Creating Index Pattern](#creating-index-pattern)\n        * [Viewing logs](#viewing-logs)\n        * [Searching logs](#searching-logs)\n\t\n\n## \u003ca name=\"preface\"\u003e\u003c/a\u003ePreface\n\nThis project of mines demonstrates how to setup and configure the Elastic Stack for a project. Note that I'm only going to showcase the most minimalistic options of configurations. \n\nelastic.co has in-depth tutorial \u0026 guides as part of their documentation along with reference examples shipped with each of their products to help you do the most advanced setup.\n\n## \u003ca name=\"the-elastic-stack\"\u003e\u003c/a\u003eThe Elastic Stack\n\nWhat is the elastic stack? \n\nThe Elastic Stack is the new name for the list of products offered by elastic.co\n\nPreviously this was commonly known as the ELK Stack (because it ONLY contained ElasticSearch-Logstash-Kibana). But now it has outgrown that to contain more software products most noticeably Beats (which consist of many individual components), APM (Application Performance Monitoring), Elastic on the Cloud offering and a few more. \n\nThe rationale behind calling it the Elastic Stack is simply because their stack has now many products and not just ElasticSearch, Logstash, and Kibana.\n\n## \u003ca name=\"stack-overview\"\u003e\u003c/a\u003eStack Overview\n\nHere's a brief overview of what each product of the stack does. I'm probably not going to do it justice myself or I might be a tad incorrect so please head over to elastic.co official site to get an explanation of what each product is.\n\n| Product | Description |\n|:-------------------- |:--------------- |\n| Elasticsearch | A highly available Search Engine |\n| Logstash | A lighweight log centalizer, transformer, and stasher |\n| Kibana | A visualization tool that views the elasticsearch's data |\n| Filebeat | A lightweight log shipper |\n| Metricbeat | A lightweight metric data shipper |\n| Heartbeat | A lightweight heartbeat pinger |\n\n## \u003ca name=\"setup\"\u003e\u003c/a\u003eSetup\n\nAs part of this demo I'm going to also use one of my other sample application as demonstration - https://github.com/colinbut/microservices-template\n\nThis is how it fits together:\n\n![Image of the stack](etc/stack-overview.png)\n\n### \u003ca name=\"download\"\u003e\u003c/a\u003eDownload \nDownload all the services below from elastic.co site's download pages of each of their products.\nUnpack all the services into their own distinct binary directories.\n\n### \u003ca name=\"elasticsearch\"\u003e\u003c/a\u003e1 ElasticSearch\n\nElasticSearch is the search engine which contains indices of data ingested from various sources and it is the core of the Elastic Stack. It is at the heart of everything. So the first thing is to start this service up.\n\nMinimal configuration needed and so can fallback to all defaults.\n\n```bash\n/bin/elasticsearch\n```\n\nby default, ElasticSearch runs on port 9200. So mines was running on localhost therefore - http://localhost:9200\n\n### \u003ca name=\"logstash\"\u003e\u003c/a\u003e2 Logstash\n\nLogstash is a transformation pipeline. You feed in input data and you can filter it by doing a serious of transformations on that data fed in finally outputting back to the connecting sink. In case of the Elastic Stack (and previous the ELK stack) this would be Elastic Search the search engine.\n\nThis is optional. Tradtionally this was somewhat mandatory as it was one of the main services that made up the previous ELK stack.\n\n```bash\n/bin/logstash -f logstash-conf.conf\n```\n\nLogstash default runs on port 5044.\n\nlogstash-conf.conf:\n\n```\ninput {\n  stdin {}\n  beats {\n    port =\u003e 5044\n  }\n}\n\noutput {\n  stdout {}\n  elasticsearch {\n    hosts =\u003e [\"http://localhost:9200\"]\n    index =\u003e \"%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}\"\n  }\n}\n```\n\nIn above I configured stdin \u0026 stdout plugins for input \u0026 output respectively simply for as a means of sanity test to ensure Logstash is up and running and appears fine to me. \n\n### \u003ca name=\"filebeat\"\u003e\u003c/a\u003e3 Filebeat\n\n```bash\n./filebeat\n```\n\nFilebeat is a lightweight log shipper. All it does is gather the logs from your application(s) and ships it to ELK stack. As an addition to the ELK stack, beat that made ELK stack became what is now known as Elastic Stack.\n\n`filebeat.yml`\n\n```yaml\n- type: log\n  enabled: true\n  paths:\n    - /Applications/dev-sandbox/projects/microservices-template/logs/microservice-template*.log\n```\n\n### \u003ca name=\"metricbeat\"\u003e\u003c/a\u003e4 Metricbeat\n\n```bash\n./metricbeat\n```\n\nThe following configuration setups predefined dashboards made already by Elastic on Kibana and tells Metricbeat to ship metric data onto ElasticSearch.\n\n```yaml\nsetup.dashboards.enabled: true \n\nsetup.kibana:\n  host: \"localhost:5601\"\n\n#-------------------------- Elasticsearch output ------------------------------\noutput.elasticsearch:\n  hosts: [\"localhost:9200\"]\n```\n\nMetricbeat needs to be run on the same host as the application that it monitors for metrics. If application is containerized (Docker) then Metricbeat can also gather metrics from within the Docker container and you can then view the metrics data on the predefined Kibana dashboards.\n\n![Image of Metricbeat dashboard](etc/metricbeat-dashboard.png)\n\n__Full Screen View__\n![Image of Metricbeat dashboard](etc/metricbeat-dashboard-fullscreen.png)\n\n### \u003ca name=\"heartbeat\"\u003e\u003c/a\u003e5 Heartbeat\n\n```bash\n./heartbeat\n```\n\nThe most important configuration is to tell Heartbeat where to monitor for health and where to send the gathered data to.\nAgain, just like other beats, you can choose to send to either ElasticSearch directly or via Logstash.\n\n```yaml\n# Configure monitors\nheartbeat.monitors:\n- type: http\n  # List or urls to query\n  urls: [\"http://localhost:54268\"]\n  # Configure task schedule\n  schedule: '@every 10s'\n  # Total test connection and data exchange timeout\n  #timeout: 16s\n\n#-------------------------- Elasticsearch output ------------------------------\noutput.elasticsearch:\n  hosts: [\"localhost:9200\"]\n```\n\nAbove, i've configured it so send it to ElasticSearch directly as I don't require any transformation processing in Logstash and setup the monitor to ping my demo application's url every 10s.\n\nJust like other beats, Heartbeat comes with predefined Kibana dashboard that can be loaded into Kibana on startup.\n\n### \u003ca name=\"kibana\"\u003e\u003c/a\u003e6 Kibana\n\nFinally, the visualization tool of the Elastic Stack allowing you to \"see\" the data ingested for analysis. \n\n```bash\n/bin/kibana\n```\n\nNo special configuration required and can use defaults as everything in Kibana setup is defaulted. The ElasticSearch url needs changing if Kibana is not run on the same host as ElasticSearch is.\n\nWith Kibana you can use it as a centralized logging tool where you can see all your logs for all your applications in one place.\nYou can also create various dashboards of different types of graphs of your log data.\n\n#### \u003ca name=\"creating-index-pattern\"\u003e\u003c/a\u003eCreating index pattern\n\nTo view logs or any other data you require to configure on the Kibana UI to tell it what ElasticSearch Indices to look for.\nAll data is stored in ElasticSearch as a series of indices with data.\n\n![Image of Creating Index Pattern](etc/creating-index-pattern.png)\n\n#### \u003ca name=\"viewing-logs\"\u003e\u003c/a\u003eViewing logs\n\n![Image of Kibana View Log](etc/kibana-view-log.png)\n\n__Inspecting the table view__\n![Image of Kibana Table View Log](etc/kibana-logs-table-view.png)\n\n__Looking at the JSON view__\n![Image of Kibana JSON View Log](etc/kibana-logs-json-view.png)\n\n#### \u003ca name=\"searching-logs\"\u003e\u003c/a\u003eSearching logs\n\n![Image of Kibana Search Log](etc/kibana-search-log.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcolinbut%2Fcentralized-logging-with-elastic-stack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcolinbut%2Fcentralized-logging-with-elastic-stack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcolinbut%2Fcentralized-logging-with-elastic-stack/lists"}