{"id":20548052,"url":"https://github.com/aleroxac/elk-kafka","last_synced_at":"2026-04-16T05:03:50.173Z","repository":{"id":159050115,"uuid":"348109663","full_name":"aleroxac/elk-kafka","owner":"aleroxac","description":"ELK Stack with Apache Kafka as buffer.","archived":false,"fork":false,"pushed_at":"2021-03-16T19:24:46.000Z","size":1254,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-16T16:23:44.016Z","etag":null,"topics":["beats","docker","docker-compose","elasticsearch","filebeat","kafka","kibana","logging","logstash","observability","zookeeper"],"latest_commit_sha":null,"homepage":"https://github.com/aleroxac/elk-kafka","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aleroxac.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-15T20:14:18.000Z","updated_at":"2023-05-14T17:47:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"fbdce71e-37b2-4143-9456-9cb05cc682ed","html_url":"https://github.com/aleroxac/elk-kafka","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aleroxac%2Felk-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aleroxac%2Felk-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aleroxac%2Felk-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aleroxac%2Felk-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aleroxac","download_url":"https://codeload.github.com/aleroxac/elk-kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242151726,"owners_count":20080155,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beats","docker","docker-compose","elasticsearch","filebeat","kafka","kibana","logging","logstash","observability","zookeeper"],"created_at":"2024-11-16T02:11:49.798Z","updated_at":"2026-04-16T05:03:50.147Z","avatar_url":"https://github.com/aleroxac.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# elk-kafka\n![Elastic Stack](assets/img/elk-kafka.png)\n\nELK Stack with Apache Kafka as buffer.\n\n\n## Resources available\n- [elk-sample](https://github.com/aleroxac/elk-sample) files(with some changes)\n- filebeat\n- apache kafka\n- apache zookeeper\n\n\n## Setup\n``` shell\n## Installing Docker\ncurl -fsSL https://get.docker.com | bash\n\n### Installing docker-compose\nsudo curl -L \"https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)\" -o /usr/local/bin/docker-compose\nsudo chmod +x /usr/local/bin/docker-compose\n```\n\n\n## Use mode\n``` sh\n## Running the docker-compose\ndocker-compose up -d\n\n### Viewing messages inside kafka \ndocker-compose exec kafka bash -c 'watch -n1 kafka-consumer-groups --bootstrap-server localhost:9092 --group logstash --describe'\n\n### Restarting index process\nassets/scripts/restart apache\n\n### Checking if index was been created\ncurl -sf \"http://localhost:9200/_cat/shards/apache*?v\" --user elastic:changeme\n\n### Creating Kibana index-pattern\ncurl 'http://localhost:5601/api/saved_objects/index-pattern' \\\n  -H 'kbn-version: 7.10.0' \\\n  -H 'Content-Type: application/json' \\\n  --data-raw '{\"attributes\":{\"title\":\"apache*\",\"timeFieldName\":\"@timestamp\",\"fields\":\"[{\\\"count\\\":0,\\\"name\\\":\\\"@timestamp\\\",\\\"type\\\":\\\"date\\\",\\\"esTypes\\\":[\\\"date\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true},{\\\"count\\\":0,\\\"name\\\":\\\"@version\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"text\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"@version.keyword\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"keyword\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true,\\\"subType\\\":{\\\"multi\\\":{\\\"parent\\\":\\\"@version\\\"}}},{\\\"count\\\":0,\\\"name\\\":\\\"_id\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"_id\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"_index\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"_index\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"_score\\\",\\\"type\\\":\\\"number\\\",\\\"scripted\\\":false,\\\"searchable\\\":false,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"_source\\\",\\\"type\\\":\\\"_source\\\",\\\"esTypes\\\":[\\\"_source\\\"],\\\"scripted\\\":false,\\\"searchable\\\":false,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"_type\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"_type\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"host.name\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"text\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"host.name.keyword\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"keyword\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true,\\\"subType\\\":{\\\"multi\\\":{\\\"parent\\\":\\\"host.name\\\"}}},{\\\"count\\\":0,\\\"name\\\":\\\"http_version\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"text\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"http_version.keyword\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"keyword\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true,\\\"subType\\\":{\\\"multi\\\":{\\\"parent\\\":\\\"http_version\\\"}}},{\\\"count\\\":0,\\\"name\\\":\\\"input.type\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"text\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"input.type.keyword\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"keyword\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true,\\\"subType\\\":{\\\"multi\\\":{\\\"parent\\\":\\\"input.type\\\"}}},{\\\"count\\\":0,\\\"name\\\":\\\"log.file.path\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"text\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"log.file.path.keyword\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"keyword\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true,\\\"subType\\\":{\\\"multi\\\":{\\\"parent\\\":\\\"log.file.path\\\"}}},{\\\"count\\\":0,\\\"name\\\":\\\"log.offset\\\",\\\"type\\\":\\\"number\\\",\\\"esTypes\\\":[\\\"long\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true},{\\\"count\\\":0,\\\"name\\\":\\\"log_type\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"text\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"log_type.keyword\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"keyword\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true,\\\"subType\\\":{\\\"multi\\\":{\\\"parent\\\":\\\"log_type\\\"}}},{\\\"count\\\":0,\\\"name\\\":\\\"remoteaddr\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"text\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"remoteaddr.keyword\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"keyword\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true,\\\"subType\\\":{\\\"multi\\\":{\\\"parent\\\":\\\"remoteaddr\\\"}}},{\\\"count\\\":0,\\\"name\\\":\\\"request_method\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"text\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"request_method.keyword\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"keyword\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true,\\\"subType\\\":{\\\"multi\\\":{\\\"parent\\\":\\\"request_method\\\"}}},{\\\"count\\\":0,\\\"name\\\":\\\"request_uri\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"text\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"request_uri.keyword\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"keyword\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true,\\\"subType\\\":{\\\"multi\\\":{\\\"parent\\\":\\\"request_uri\\\"}}},{\\\"count\\\":0,\\\"name\\\":\\\"response_status\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"text\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"response_status.keyword\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"keyword\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true,\\\"subType\\\":{\\\"multi\\\":{\\\"parent\\\":\\\"response_status\\\"}}},{\\\"count\\\":0,\\\"name\\\":\\\"username\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"text\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":false,\\\"readFromDocValues\\\":false},{\\\"count\\\":0,\\\"name\\\":\\\"username.keyword\\\",\\\"type\\\":\\\"string\\\",\\\"esTypes\\\":[\\\"keyword\\\"],\\\"scripted\\\":false,\\\"searchable\\\":true,\\\"aggregatable\\\":true,\\\"readFromDocValues\\\":true,\\\"subType\\\":{\\\"multi\\\":{\\\"parent\\\":\\\"username\\\"}}}]\"}}' \\\n  --user elastic:changeme\n```\n\n## Screenshots\n![elk_kafka_img1](assets/img/2021-03-16_13-42.png)\n![elk_kafka_img2](assets/img/2021-03-16_14-40.png)\n![elk_kafka_img3](assets/img/2021-03-16_14-41.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleroxac%2Felk-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faleroxac%2Felk-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleroxac%2Felk-kafka/lists"}