{"id":21091320,"url":"https://github.com/lyouthzzz/efk-yaml","last_synced_at":"2025-07-08T15:06:51.493Z","repository":{"id":130549052,"uuid":"333674378","full_name":"lyouthzzz/efk-yaml","owner":"lyouthzzz","description":"EFK logging","archived":false,"fork":false,"pushed_at":"2021-02-23T14:53:50.000Z","size":83,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-14T06:24:54.873Z","etag":null,"topics":["efk","elasticsearch","kafka","kibana","logstash"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lyouthzzz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-28T07:03:06.000Z","updated_at":"2021-02-23T14:55:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"444fa730-c637-4725-98bf-1e108a78cb6d","html_url":"https://github.com/lyouthzzz/efk-yaml","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lyouthzzz/efk-yaml","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyouthzzz%2Fefk-yaml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyouthzzz%2Fefk-yaml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyouthzzz%2Fefk-yaml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyouthzzz%2Fefk-yaml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lyouthzzz","download_url":"https://codeload.github.com/lyouthzzz/efk-yaml/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyouthzzz%2Fefk-yaml/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264292946,"owners_count":23586062,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["efk","elasticsearch","kafka","kibana","logstash"],"created_at":"2024-11-19T21:45:03.748Z","updated_at":"2025-07-08T15:06:51.476Z","avatar_url":"https://github.com/lyouthzzz.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# 日志收集系统-ELK\n\n## 流程\n![](/ELK.jpg)\n\n## fluent-bit\n\nFluent Bit 必须作为 `DaemonSet` 部署，这样就可以在 `Kubernetes` 集群的每个节点上使用它。首先，请使用以下命令来创建名称空间，服务帐号和角色设置(`namespace`, `serviceaccount`, `role``)\n\n### 部署\n```\nkubectl create namespace logging\nkubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml\nkubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml\nkubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml\n```\n\n创建ConfigMap以及DaemonSet\n```\nkubectl create -f fluentbit/deployments/k8s.yaml\n```\n\n## elasticsearch\n日志持久化组件，以及查询组件。\n\n### 部署\n#### 主节点\n```\nkubectl apply -f elasticsearch/elasticsearch-master.deployment.yaml\n```\n\n#### 数据节点\n```\nkubectl apply -f elasticsearch/elasticsearch-data.statefulset.yaml\n```\n\n#### 客户端节点\n```\nkubectl apply -f elasticsearch/elasticsearch-client.deployment.yaml\n```\n\n### 查看状态\n```\nkubectl get pods -n logging -l app=elasticsearch\n```\n\n### 生成密码\n我们启用了 xpack 安全模块来保护我们的集群，所以我们需要一个初始化的密码。我们可以执行如下所示的命令，在客户端节点容器内运行 bin/elasticsearch-setup-passwords 命令来生成默认的用户名和密码\n```\nkubectl exec $(kubectl get pods -n logging | grep elasticsearch-client | sed -n 1p | awk '{print $1}') \\\n    -n logging \\\n    -- bin/elasticsearch-setup-passwords auto -b\n```\n保存密码到Secret对象中\n```\nkubectl create secret generic elasticsearch-pw-elastic \\\n    -n logging \\\n    --from-literal password=${生成的密码}\n```\n修改密码\n```\ncurl -H \"Content-Type:application/json\" -XPOST -u elastic 'http://localhost:9200/_xpack/security/user/elastic/_password' -d '{ \"password\" : \"new password\" }'\n```\n\n验证密码\n```\ncurl -u elastic 'http://localhost:9200/_xpack/security/_authenticate?pretty'\n```\n\n## logstash\n日志的过滤聚合组件。采集kafka日志消息队列，做日志的过滤聚合，并输出到es集群中去。\n\n### 部署\n```\nkubectl apply -f logstash/logstash.deployment.yaml\n```\n\n## metricbeat\n### 部署\n服务metric指标收集的强大利器，可以采集kafka，logstash，kube-metric，system，docker等状态指标，并且es，kibana交互，可以展示非常友好的Dashboard。  \n当你想要收集如kafka、logstash类似的deployment服务的时候，metricbeat可以选择deployment方式部署。  \n当你想要收集node节点指标时，那么metricbeat需要以daemonset方式部署。  \n\n```\nkubectl apply -f metricbeat/metricbeat.deployment.yaml\n```\n\n## kibana\n### 部署\n```\nkubectl apply -f kibana/kibana.deployment.yaml\n```\n### 查看状态\n```\nkubectl get svc kibana -n logging\n```\n\n## 可能存在的瓶颈\n- elasticsearch磁盘IO瓶颈，可采用固态盘硬件，热节点。（需注意硬件隔离）\n- logstash消费速度慢，可增加实例，注意consumer thread、pipeline thread的合理配置\n\n##  todo\n\n- 日志采集组件选用filebeat，ELK Stack可以非常友好的集成。\n- elasticsearch采用集群方式部署，可横向扩展\n- logstash多节点部署。需要合理配置consumer thread以保证最大的消费速度\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flyouthzzz%2Fefk-yaml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flyouthzzz%2Fefk-yaml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flyouthzzz%2Fefk-yaml/lists"}