{"id":15622560,"url":"https://github.com/v5tech/elk","last_synced_at":"2025-04-04T06:06:34.186Z","repository":{"id":47728340,"uuid":"59103904","full_name":"v5tech/ELK","owner":"v5tech","description":"搭建ELK日志分析平台。","archived":false,"fork":false,"pushed_at":"2020-12-20T09:46:29.000Z","size":18942,"stargazers_count":796,"open_issues_count":3,"forks_count":315,"subscribers_count":58,"default_branch":"master","last_synced_at":"2025-03-28T05:08:50.357Z","etag":null,"topics":["elasticsearch","elk","filebeat","kibana","logstash","topbeat"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/v5tech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-05-18T09:58:33.000Z","updated_at":"2025-03-19T05:32:32.000Z","dependencies_parsed_at":"2022-08-12T13:50:27.249Z","dependency_job_id":null,"html_url":"https://github.com/v5tech/ELK","commit_stats":null,"previous_names":["v5tech/elk","ameizi/elk"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v5tech%2FELK","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v5tech%2FELK/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v5tech%2FELK/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v5tech%2FELK/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/v5tech","download_url":"https://codeload.github.com/v5tech/ELK/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247128743,"owners_count":20888235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","elk","filebeat","kibana","logstash","topbeat"],"created_at":"2024-10-03T09:54:30.943Z","updated_at":"2025-04-04T06:06:34.169Z","avatar_url":"https://github.com/v5tech.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# ELK\n\n```\n环境:\nVagrant 1.8.1\nCentOS 7.2 192.168.0.228\nElasticsearch 2.3.2\nlogstash 2.2.4\nKibana 4.4.2\nfilebeat 1.2.2\ntopbeat 1.2.2\n```\n\n搭建ELK日志分析平台。此处为其核心配置文件。具体搭建过程请参考[ELK环境搭建.docx](https://raw.githubusercontent.com/ameizi/ELK/master/ELK环境搭建.docx \"ELK环境搭建.docx文档\")文档\n\n# Screenshots\n\nelasticsearch索引列表\n\n![](https://raw.githubusercontent.com/ameizi/ELK/master/Screenshots/elasticsearch-head.png)\n\nNginx日志分析\n\n![](https://raw.githubusercontent.com/ameizi/ELK/master/Screenshots/Discover-Kibana-Nginx.png)\n\nSyslog系统日志分析\n\n![](https://raw.githubusercontent.com/ameizi/ELK/master/Screenshots/Discover-Kibana-Syslog.png)\n\nTomcat日志分析\n\n![](https://raw.githubusercontent.com/ameizi/ELK/master/Screenshots/Discover-Kibana-Tomcat.png)\n\n系统日志分析\n\n![](https://raw.githubusercontent.com/ameizi/ELK/master/Screenshots/Discover-Kibana-Topbeat.png)\n\nTopbeat Dashboard\n\n![](https://raw.githubusercontent.com/ameizi/ELK/master/Screenshots/Topbeat-Dashboard-Kibana.png)\n\n\n# logstash命令\n\nlogstash命令帮助\n\n```bash\n$ /opt/logstash/bin/logstash -h\nUsage:\n    /bin/logstash agent [OPTIONS]\n\nOptions:\n    -f, --config CONFIG_PATH      Load the logstash config from a specific file\n                                  or directory.  If a directory is given, all\n                                  files in that directory will be concatenated\n                                  in lexicographical order and then parsed as a\n                                  single config file. You can also specify\n                                  wildcards (globs) and any matched files will\n                                  be loaded in the order described above.\n    -e CONFIG_STRING              Use the given string as the configuration\n                                  data. Same syntax as the config file. If no\n                                  input is specified, then the following is\n                                  used as the default input:\n                                  \"input { stdin { type =\u003e stdin } }\"\n                                  and if no output is specified, then the\n                                  following is used as the default output:\n                                  \"output { stdout { codec =\u003e rubydebug } }\"\n                                  If you wish to use both defaults, please use\n                                  the empty string for the '-e' flag.\n                                   (default: \"\")\n    -w, --pipeline-workers COUNT  Sets the number of pipeline workers to run.\n                                   (default: 1)\n    -b, --pipeline-batch-size SIZE Size of batches the pipeline is to work in.\n                                   (default: 125)\n    -u, --pipeline-batch-delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling\n                                  for the next event.\n                                   (default: 5)\n    --filterworkers COUNT         DEPRECATED. Now an alias for --pipeline-workers and -w\n    -l, --log FILE                Write logstash internal logs to the given\n                                  file. Without this flag, logstash will emit\n                                  logs to standard output.\n    -v                            Increase verbosity of logstash internal logs.\n                                  Specifying once will show 'informational'\n                                  logs. Specifying twice will show 'debug'\n                                  logs. This flag is deprecated. You should use\n                                  --verbose or --debug instead.\n    --quiet                       Quieter logstash logging. This causes only \n                                  errors to be emitted.\n    --verbose                     More verbose logging. This causes 'info' \n                                  level logs to be emitted.\n    --debug                       Most verbose logging. This causes 'debug'\n                                  level logs to be emitted.\n    --debug-config                translation missing: en.logstash.runner.flag.debug_config (default: false)\n    -V, --version                 Emit the version of logstash and its friends,\n                                  then exit.\n    -p, --pluginpath PATH         A path of where to find plugins. This flag\n                                  can be given multiple times to include\n                                  multiple paths. Plugins are expected to be\n                                  in a specific directory hierarchy:\n                                  'PATH/logstash/TYPE/NAME.rb' where TYPE is\n                                  'inputs' 'filters', 'outputs' or 'codecs'\n                                  and NAME is the name of the plugin.\n    -t, --configtest              Check configuration for valid syntax and then exit.\n    --[no-]allow-unsafe-shutdown  Force logstash to exit during shutdown even\n                                  if there are still inflight events in memory.\n                                  By default, logstash will refuse to quit until all\n                                  received events have been pushed to the outputs.\n                                   (default: false)\n    -h, --help                    print help\n```\n\n检查指定logstash配置文件\n\n```bash\n$ /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/14-log4j_to_es.conf -t\n```\n\n收集指定配置文件日志\n\n```bash\n$ /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/14-log4j_to_es.conf\n```\n\n查看logstash服务状态\n\n```\n$ sudo service logstash status \nlogstash is running\n```\n\n```\n$ sudo service logstash start|stop|restart \n```\n\n# kibana\n\n修改/opt/kibana/config\n\n```\nserver.port: 5601\nserver.host: \"192.168.0.228\"\nelasticsearch.url: \"http://192.168.0.228:9200\"\nkibana.index: \".kibana\"\n```\n\n# ELK使用场景配置示例\n\n## syslog日志\n\nlogstash filter配置\n\n```\nfilter {\n  if [type] == \"syslog\" {\n    grok {\n      match =\u003e { \"message\" =\u003e \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}\" }\n      add_field =\u003e [ \"received_at\", \"%{@timestamp}\" ]\n      add_field =\u003e [ \"received_from\", \"%{host}\" ]\n    }\n    syslog_pri { }\n    date {\n      match =\u003e [ \"syslog_timestamp\", \"MMM  d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]\n    }\n  }\n}\n```\n\n## java日志收集\n\nhttp://kibana.logstash.es/content/logstash/examples/java.html\n\n* log4j SocketAppender\n\nlogstash配置\n\n```\ninput {\n  # log4j SocketAppender\n  log4j {\n    mode =\u003e \"server\"\n    host =\u003e \"192.168.0.228\"\n    port =\u003e 4560\n    type =\u003e \"log4j\"\n  }\n}\n\nfilter {\n}\n\noutput {\n  if [type] == \"log4j\" {\n    elasticsearch {\n      action =\u003e \"index\"\n      hosts  =\u003e \"192.168.0.228:9200\"\n      index  =\u003e \"log4j-access-%{+yyyy.MM.dd}\"\n    }\n    redis {\n      host =\u003e \"192.168.0.46\"\n      port =\u003e 6379\n      data_type =\u003e \"list\"\n      key =\u003e \"logstash:log4j\"\n    }\n  } \n}\n\n```\n\nlog4j.properties\n\n```\nlog4j.rootLogger=logstash\n\n###SocketAppender###\nlog4j.appender.logstash=org.apache.log4j.net.SocketAppender\n# logstash中log4j input中的端口号\nlog4j.appender.logstash.Port=4560\n# logstash所在机器IP\nlog4j.appender.logstash.RemoteHost=192.168.0.228\nlog4j.appender.logstash.ReconnectionDelay=60000\nlog4j.appender.logstash.LocationInfo=true\nlog4j.appender.logstash.Application=elk-log4j-simple\n```\n\n* log4j-jsonevent-layout\n\nlogstash配置\n\n```\ninput {\n  # log4j-jsonevent-layout\n  file {\n    codec =\u003e json\n    path =\u003e \"/home/vagrant/tomcat-7.0.69/bin/target/*.log\"\n    type =\u003e \"log4j\"\n    start_position =\u003e \"beginning\"\n    sincedb_path =\u003e \"/dev/null\"\n  }\n}\n\nfilter {\n}\n\noutput {\n  if [type] == \"log4j\" {\n    elasticsearch {\n      action =\u003e \"index\"\n      hosts  =\u003e \"192.168.0.228:9200\"\n      index  =\u003e \"log4j-access-%{+yyyy.MM.dd}\"\n    }\n    redis {\n      host =\u003e \"192.168.0.46\"\n      port =\u003e 6379\n      data_type =\u003e \"list\"\n      key =\u003e \"logstash:log4j\"\n    }\n  } \n}\n```\n\n```xml\n\u003c!-- 将lo4j日志输出为json --\u003e\n\u003cdependency\u003e\n    \u003cgroupId\u003enet.logstash.log4j\u003c/groupId\u003e\n    \u003cartifactId\u003ejsonevent-layout\u003c/artifactId\u003e\n    \u003cversion\u003e1.7\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n注:`output`中`elasticsearch`项中`index`为当前被索引文档在`elasticsearch`中索引名称。使用`kibana`搜索的时候需要事先根据该值创建一个`index pattern`\n\n## tomcat日志\n\nlogstash pattern配置\n\n```\nJAVACLASS (?:[a-zA-Z0-9-]+\\.)+[A-Za-z0-9$]+\n\nJAVALOGMESSAGE (.*)\n\n# MMM dd, yyyy HH:mm:ss eg: Jan 9, 2014 7:13:13 AM\nCATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)\n\n# yyyy-MM-dd HH:mm:ss,SSS ZZZ eg: 2014-01-09 17:32:25,527 -0800\nTOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}\n\nCATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}\n\n# 2014-01-09 20:03:28,269 -0800 | ERROR | com.example.service.ExampleService - something compeletely unexpected happened...\nTOMCATLOG %{TOMCAT_DATESTAMP:timestamp} \\| %{LOGLEVEL:level} \\| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}\n```\nlogstash filter配置\n\n```\nfilter {\n  if [type] == \"tomcat_access\" {\n    grok {\n      match =\u003e [ \"message\", \"%{TOMCATLOG}\", \"message\", \"%{CATALINALOG}\" ]\n    }\n    date {\n      match =\u003e [ \"timestamp\", \"yyyy-MM-dd HH:mm:ss,SSS Z\", \"MMM dd, yyyy HH:mm:ss a\" ]\n    }\n  }\n}\n```\n\n## apache日志\n\nlogstash filter配置\n\n```\nfilter {\n  if [type] == \"apache-access\" {\n    grok {\n      match =\u003e { \"message\" =\u003e \"%{COMBINEDAPACHELOG}\" }\n    }\n  }\n}\n```\n\n## nginx访问日志\n\nlogstash pattern配置\n\n```\nNGUSERNAME [a-zA-Z\\.\\@\\-\\+_%]+\nNGUSER %{NGUSERNAME}\nNGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \\[%{HTTPDATE:timestamp}\\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:\"(?:%{URI:referrer}|-)\"|%{QS:referrer}) %{QS:agent}\n```\n\nlogstash filter配置\n\n```\nfilter {\n  if [type] == \"nginx-access\" {\n    grok {\n      match =\u003e { \"message\" =\u003e \"%{NGINXACCESS}\" }\n    }\n    geoip {\n      source =\u003e \"clientip\"\n      target =\u003e \"geoip\"\n      database =\u003e \"/etc/logstash/GeoLiteCity.dat\"\n      add_field =\u003e [ \"[geoip][coordinates]\", \"%{[geoip][longitude]}\" ]\n      add_field =\u003e [ \"[geoip][coordinates]\", \"%{[geoip][latitude]}\"  ]\n    }\n    mutate {\n      convert =\u003e [ \"[geoip][coordinates]\", \"float\"]\n    }\n  }\n}\n```\n\n## nginx access日志转化为json格式\n\nhttp://kibana.logstash.es/content/logstash/examples/nginx-access.html\n\nhttp://kibana.logstash.es/content/logstash/plugins/codec/json.html\n\nnginx.conf\n\n```\nlog_format json '{\"@timestamp\":\"$time_iso8601\",'\n                 '\"host\":\"$server_addr\",'\n                 '\"clientip\":\"$remote_addr\",'\n                 '\"size\":$body_bytes_sent,'\n                 '\"responsetime\":$request_time,'\n                 '\"upstreamtime\":\"$upstream_response_time\",'\n                 '\"upstreamhost\":\"$upstream_addr\",'\n                 '\"http_host\":\"$host\",'\n                 '\"url\":\"$uri\",'\n                 '\"xff\":\"$http_x_forwarded_for\",'\n                 '\"referer\":\"$http_referer\",'\n                 '\"agent\":\"$http_user_agent\",'\n                 '\"status\":\"$status\"}';\naccess_log  /var/log/nginx/access.log  json;                 \n```\n\n\nlogstash配置\n\n```\ninput {\n  file {             #从nginx日志读入\n    type =\u003e \"nginx-access\"\n    path =\u003e \"/var/log/nginx/access.log\"\n    start_position =\u003e \"beginning\"\n    sincedb_path =\u003e \"/dev/null\"\n    codec =\u003e \"json\"  #这里指定codec格式为json\n  }\n}\n\nfilter {\n    mutate {\n        split =\u003e [ \"upstreamtime\", \",\" ]\n    }\n    mutate {\n        convert =\u003e [ \"upstreamtime\", \"float\" ]\n    }\n}\n\noutput {\n  if [type] == \"nginx-access\" {\n    elasticsearch {\n      hosts =\u003e [\"192.168.0.228:9200\"]\n      index =\u003e \"nginx-access-%{+yyyy.MM.dd}\"\n    }\n  }\n}\n```\n\n## mysql慢日志\n\nhttp://kibana.logstash.es/content/logstash/examples/mysql-slow.html\n\nlogstash配置\n\n```\ninput {\n  file {\n   type =\u003e \"mysql-slow\"\n   path =\u003e \"/var/log/mysql/mysql-slow.log\"\n   start_position =\u003e \"beginning\"\n   sincedb_path =\u003e \"/dev/null\"\n   codec =\u003e multiline {         #这里用到了logstash的插件功能，将本来属于一行的多行日志条目整合在一起，让他属于一条   \n     pattern =\u003e \"^# User@Host\"  #用到了正则去匹配\n     negate =\u003e true\n     what =\u003e \"previous\"\n   }\n  }\n}\n\nfilter {\n  # drop sleep events\n  grok {\n    match =\u003e { \"message\" =\u003e \"SELECT SLEEP\" }\n    add_tag =\u003e [ \"sleep_drop\" ]\n    tag_on_failure =\u003e [] # prevent default _grokparsefailure tag on real records\n  }\n  if \"sleep_drop\" in [tags] {\n    drop {}\n  }\n  grok {\n    match =\u003e [ \"message\", \"(?m)^# User@Host: %{USER:user}\\[[^\\]]+\\] @ (?:(?\u003cclienthost\u003e\\S*) )?\\[(?:%{IP:clientip})?\\]\\s*# Query_time: %{NUMBER:query_time:float}\\s+Lock_time: %{NUMBER:lock_time:float}\\s+Rows_sent: %{NUMBER:rows_sent:int}\\s+Rows_examined: %{NUMBER:rows_examined:int}\\s*(?:use %{DATA:database};\\s*)?SET timestamp=%{NUMBER:timestamp};\\s*(?\u003cquery\u003e(?\u003caction\u003e\\w+)\\s+.*)\\n# Time:.*$\" ]\n  }\n  date {\n    match =\u003e [ \"timestamp\", \"UNIX\" ]\n    remove_field =\u003e [ \"timestamp\" ]\n  }\n}\n\noutput {\n  if [type] == \"mysql-slow\" {\n     elasticsearch {\n        action =\u003e \"index\"\n        hosts  =\u003e \"192.168.0.228:9200\"\n        index  =\u003e \"mysql-slow-%{+yyyy.MM.dd}\"\n     }\n  }\n}\n```\n\n# 平台搭建参考文章\n\nhttps://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-7\n\nhttps://www.digitalocean.com/community/tutorials/how-to-gather-infrastructure-metrics-with-topbeat-and-elk-on-centos-7\n\nhttps://www.digitalocean.com/community/tutorials/adding-logstash-filters-to-improve-centralized-logging\n\nhttps://www.digitalocean.com/community/tutorials/how-to-use-kibana-dashboards-and-visualizations\n\nhttps://www.digitalocean.com/community/tutorials/how-to-map-user-location-with-geoip-and-elk-elasticsearch-logstash-and-kibana\n\n\n# yml语法校验\n\nhttp://yaml-online-parser.appspot.com/\n\nhttp://www.yamllint.com/\n\n\n# linux平台系统运维教程集\n\nhttps://www.digitalocean.com/community/tutorials\n\nhttp://www.unixmen.com/\n\nhttp://linoxide.com/\n\n\n# tomcat日志分析参考\n\nhttps://aggarwalarpit.wordpress.com/2015/12/03/configuring-elk-stack-to-analyse-apache-tomcat-logs/\n\nhttps://www.systemcodegeeks.com/web-servers/apache/configuring-elk-stack-analyse-apache-tomcat-logs/\n\nhttp://stackoverflow.com/questions/25429377/how-can-i-integrate-tomcat6s-catalina-out-file-with-logstash-elasticsearch\n\nhttps://blog.codecentric.de/en/2014/10/log-management-spring-boot-applications-logstash-elastichsearch-kibana/\n\nhttps://blog.lanyonm.org/articles/2014/01/12/logstash-multiline-tomcat-log-parsing.html\n\nhttps://spredzy.wordpress.com/2013/03/02/monitor-your-cluster-of-tomcat-applications-with-logstash-and-kibana/\n\n# log4j日志分析\n\nhttps://qbox.io/blog\n\nhttps://github.com/logstash/log4j-jsonevent-layout\n\nhttps://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html\n\nhttps://blog.lanyonm.org/articles/2015/12/29/log-aggregation-log4j-spring-logstash.html\n\nhttp://www.tianmaying.com/tutorial/elastic-logstash-kibana\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fv5tech%2Felk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fv5tech%2Felk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fv5tech%2Felk/lists"}