{"id":22388139,"url":"https://github.com/sermilrod/kafka-elk-docker-compose","last_synced_at":"2025-07-31T06:32:33.535Z","repository":{"id":241221734,"uuid":"95893533","full_name":"sermilrod/kafka-elk-docker-compose","owner":"sermilrod","description":"Deploy ELK stack and kafka with docker-compose","archived":false,"fork":false,"pushed_at":"2017-07-24T08:37:44.000Z","size":5,"stargazers_count":82,"open_issues_count":3,"forks_count":75,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-05-23T04:20:48.089Z","etag":null,"topics":["apache2","docker","docker-compose","elasticsearch","filebeat","kafka","logstash","zookeeper"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sermilrod.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-30T14:00:09.000Z","updated_at":"2024-05-23T04:20:50.036Z","dependencies_parsed_at":"2024-05-23T04:35:19.068Z","dependency_job_id":null,"html_url":"https://github.com/sermilrod/kafka-elk-docker-compose","commit_stats":null,"previous_names":["sermilrod/kafka-elk-docker-compose"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sermilrod%2Fkafka-elk-docker-compose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sermilrod%2Fkafka-elk-docker-compose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sermilrod%2Fkafka-elk-docker-compose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sermilrod%2Fkafka-elk-docker-compose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sermilrod","download_url":"https://codeload.github.com/sermilrod/kafka-elk-docker-compose/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228219671,"owners_count":17887058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache2","docker","docker-compose","elasticsearch","filebeat","kafka","logstash","zookeeper"],"created_at":"2024-12-05T02:12:30.029Z","updated_at":"2024-12-05T02:12:30.647Z","avatar_url":"https://github.com/sermilrod.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafka-elk-docker-compose\nThis repository deploys with *docker-compose* an ELK stack which has kafka cluster buffering the logs collection process. This repository tries to make your life easier while testing a similar architecture. It is highly discouraged to use this repository as a production ready solution of this stack.\n\n## Setup\n\n1.  [Install Docker engine](https://docs.docker.com/engine/installation/)\n2.  [Install Docker compose](https://docs.docker.com/compose/install/)\n3.  Clone this repository:\n    ```\n    git clone git@github.com:sermilrod/kafka-elk-docker-compose.git\n    ```\n4. [Configure File Descriptors and MMap](https://www.elastic.co/guide/en/elasticsearch/guide/current/_file_descriptors_and_mmap.html)\nTo do so you have to type the following command:\n    ```\n    sysctl -w vm.max_map_count=262144\n    ```\n    Be aware that the previous sysctl setting vanishes when your machine restarts.\n    If you want to make it permanent place `vm.max_map_count` setting in your `/etc/sysctl.conf`.\n5. Create the elasticsearch volume:\n    ```bash\n    $ cd kafka-elk-docker-compose\n    $ mkdir esdata\n    ```\n    By default the *docker-compose.yml* uses *esdata* as the host volumen path name. If you want to use another name you have to edit the *docker-compose.yml* file and create your own structure.\n6. Create the *apache-logs* folder:\n    ```bash\n    $ cd kafka-elk-docker-compose\n    $ mkdir apache-logs\n    ```\n    This repository uses a default *apache container* to generate logs and it is required for *filebeat* to be present. If you do not want to use this apache or you want to add new components to the system just use the *docker-compose.yml* as a base for your use case.\n\n## Usage\n\nDeploy your Kafka+ELK Stack using *docker-compose*:\n\n```bash\n$ docker-compose up -d\n```\nBy default the apache container generating logs is exposed through port 8888. You can perform some requests to generate a few log entries for later visualization in kibana:\n\n``` bash\n$ curl http://localhost:8888/\n```\n\nThe full stack takes around a minute to be fully functional as there are dependencies beteween services.\nAfter that you should be able to hit Kibana [http://localhost:5601](http://localhost:5601)\n\nBefore you see the log entries generated before you have to configure an index pattern in kibana. Make sure you configure it with these two options:\n* Index name or pattern: logstash-*\n* Time-field name: @timestamp\n\n## Configuration\nThe *docker-compose.yml* deploys an ELK solution using kafka as a buffer for log collection. This repository is shipped with the minimal amount of configuration needed to make the stack work. The default config files are:\n### filebeat.yml:\n```\nfilebeat.prospectors:\n- paths:\n    - /apache-logs/access.log\n  tags:\n    - testenv\n    - apache_access\n  input_type: log\n  document_type: apache_access\n  fields_under_root: true\n\n- paths:\n    - /apache-logs/error.log\n  tags:\n    - testenv\n    - apache_error\n  input_type: log\n  document_type: apache_error\n  fields_under_root: true\n\noutput.kafka:\n  hosts: [\"kafka1:9092\", \"kafka2:9092\", \"kafka3:9092\"]\n  topic: 'log'\n  partition.round_robin:\n    reachable_only: false\n  required_acks: 1\n  compression: gzip\n  max_message_bytes: 1000000\n```\nAs you can see it is configured to read the default apache logs and push them to kafka. Any addition or change to the filebeat agent should be perform in this config file.\n\n### logstash.conf:\n```\ninput {\n  kafka {\n    bootstrap_servers =\u003e \"kafka1:9092,kafka2:9092,kafka3:9092\"\n    client_id =\u003e \"logstash\"\n    group_id =\u003e \"logstash\"\n    consumer_threads =\u003e 3\n    topics =\u003e [\"log\"]\n    codec =\u003e \"json\"\n    tags =\u003e [\"log\", \"kafka_source\"]\n    type =\u003e \"log\"\n  }\n}\n\nfilter {\n  if [type] == \"apache_access\" {\n    grok {\n      match =\u003e { \"message\" =\u003e \"%{COMMONAPACHELOG}\" }\n    }\n    date {\n      match =\u003e [\"timestamp\", \"dd/MMM/yyyy:HH:mm:ss Z\"]\n      remove_field =\u003e [\"timestamp\"]\n    }\n  }\n  if [type] == \"apache_error\" {\n    grok {\n      match =\u003e { \"message\" =\u003e \"%{COMMONAPACHELOG}\" }\n    }\n    date {\n      match =\u003e [\"timestamp\", \"dd/MMM/yyyy:HH:mm:ss Z\"]\n      remove_field =\u003e [\"timestamp\"]\n    }\n  }\n}\n\noutput {\n  if [type] == \"apache_access\" {\n    elasticsearch {\n         hosts =\u003e [\"elasticsearch:9200\"]\n         index =\u003e \"logstash-apache-access-%{+YYYY.MM.dd}\"\n    }\n  }\n  if [type] == \"apache_error\" {\n    elasticsearch {\n         hosts =\u003e [\"elasticsearch:9200\"]\n         index =\u003e \"logstash-apache-error-%{+YYYY.MM.dd}\"\n    }\n  }\n}\n```\nAs you can, logstash is configured as a kafka consumer to parse apache logs and to insert them into elasticsearch. Any addition or change to the logstash behaviour should be perform in this config file.\n\n### kibana.yml:\n```\nserver.name: kibana\nserver.host: \"0\"\nelasticsearch.url: http://elasticsearch:9200\nxpack.monitoring.ui.container.elasticsearch.enabled: false\n```\n\nIt is remarkable the fact that by default that both kibana and elasticsearch docker images enable by default the xpack plugin, which you will have to pay for it after the trial. This repository disables this paid feature by default. Any addition or change to the kibana behaviour should be perform in this config file.\n\n### Other configuration:\nYou can configure much more each of the components of the stack. It is up to you and your use case to extend the configuration files and change the *docker-compose.yml* to make it so.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsermilrod%2Fkafka-elk-docker-compose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsermilrod%2Fkafka-elk-docker-compose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsermilrod%2Fkafka-elk-docker-compose/lists"}