{"id":19249521,"url":"https://github.com/opvexe/elk","last_synced_at":"2025-04-21T12:32:36.089Z","repository":{"id":65150640,"uuid":"241301057","full_name":"opvexe/elk","owner":"opvexe","description":"elk日志收集系统","archived":false,"fork":false,"pushed_at":"2020-02-19T04:35:17.000Z","size":990,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-06-20T16:58:16.896Z","etag":null,"topics":["es7","etcd","kafka","logagent"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opvexe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-18T07:42:40.000Z","updated_at":"2023-02-26T19:13:21.000Z","dependencies_parsed_at":"2023-01-02T15:28:59.521Z","dependency_job_id":null,"html_url":"https://github.com/opvexe/elk","commit_stats":null,"previous_names":["opvexe/elk","shumintao/elk"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opvexe%2Felk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opvexe%2Felk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opvexe%2Felk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opvexe%2Felk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opvexe","download_url":"https://codeload.github.com/opvexe/elk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223863866,"owners_count":17216234,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["es7","etcd","kafka","logagent"],"created_at":"2024-11-09T18:14:14.089Z","updated_at":"2024-11-09T18:14:14.776Z","avatar_url":"https://github.com/opvexe.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## 日志收集项目架构设计及kafka介绍\n\n### ELK 项目架构\n\n![image-20200218124307456](asset/image-20200218124307456.png)\n\n### ELK方案的问题\n\n- 运维成本高，每增加一个日志收集项，都需要手动修改配置\n- 监控缺失，无法准确获取logstash的状态\n- 无法做到定制化开发和维护\n\n### 架构设计\n\n![image-20200218124557412](asset/image-20200218124557412.png)\n\n### Kafka 原理:\n\nKafka选择分区模式(3种)：\n\n- 指定往哪个分区写\n- 指定key,kafka根据key做hash然后决定写哪个分区\n- 轮询方式\n\n生产者往kafka发送数据的模式(3种):\n\n- `0 :` 把数据发送给Leader就成功，效率高，安全性最低\n- `1 :`把数据发送给leader,等待leaader确认回ACK\n- `all :`把数据发送给leader，确保follow从leader拉取数据回复ACK给leader,leader在回复ACK，安全性最高\n\n### docker-compose 搭建elk\n\n```shell\n$ cd home/\n$ git clone https://github.com/deviantony/docker-elk.git # 下载配置文件\n$ docker-compose up -d  #启动elk \n$ docker-compose ps # 查看\n$ docker-compose down \n$ docker-compose restart\n```\n\n###  kibana\n\n```shell\nhttp://127.0.0.1:5601\nuser:elastic # 用户\npassword:changeme#密码\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopvexe%2Felk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopvexe%2Felk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopvexe%2Felk/lists"}