{"id":20986629,"url":"https://github.com/dpwgc/ringlog","last_synced_at":"2026-05-05T15:37:07.211Z","repository":{"id":144485553,"uuid":"480427665","full_name":"dpwgc/ringlog","owner":"dpwgc","description":"基于Spring Boot整合Elasticsearch、Kafka及缓冲队列实现的日志收集系统。","archived":false,"fork":false,"pushed_at":"2022-04-16T06:12:26.000Z","size":3667,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-20T07:33:33.875Z","etag":null,"topics":["elasticsearch","java","kafka","log","nosql","socket","spring-boot","tcp","udp"],"latest_commit_sha":null,"homepage":"https://gitee.com/dpwgc/ringlog","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpwgc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-11T14:57:31.000Z","updated_at":"2022-04-15T08:18:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"6b70ab24-295f-459b-a7e1-b6bfd4e6bf53","html_url":"https://github.com/dpwgc/ringlog","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpwgc%2Fringlog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpwgc%2Fringlog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpwgc%2Fringlog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpwgc%2Fringlog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpwgc","download_url":"https://codeload.github.com/dpwgc/ringlog/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243393379,"owners_count":20283783,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","java","kafka","log","nosql","socket","spring-boot","tcp","udp"],"created_at":"2024-11-19T06:14:23.607Z","updated_at":"2025-12-28T15:14:31.256Z","avatar_url":"https://github.com/dpwgc.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RingLog\n## 基于Spring Boot整合Elasticsearch、Kafka及缓冲队列实现的日志收集系统。\n![Go](https://img.shields.io/static/v1?label=JDK\u0026message=1.8\u0026color=orange)\n![Go](https://img.shields.io/static/v1?label=Spring-Boot\u0026message=2.4.1\u0026color=green)\n![Go](https://img.shields.io/static/v1?label=Elasticsearch\u0026message=7.13.2^\u0026color=blue)\n![Go](https://img.shields.io/static/v1?label=Kafka\u0026message=2.7\u0026color=yellow)\n***\n\n### 实现原理\n* 使用Elasticsearch存储日志数据。\n* 使用UDP/TCP监听接收外来日志数据，客户端将产生的日志数据存入内存中，定期将一定数量的日志数据（可以是单条日志，也可以是多条日志列表）转为JSON字符串格式发送到RingLog服务端。\n* 使用ConcurrentLinkedQueue实现服务端缓冲队列，所有外来日志数据先插入缓冲队列，再由后台线程（一个或者多个消费者线程）将日志数据异步发送至外部消息队列Kafka。\n* Kafka中的一条消息中可以包含一个或者多个日志信息（JSON字符串形式存储），由Kafka消费者将日志数据批量插入Elasticsearch。\n\n![avatar](./img/img1.jpg)\n***\n\n### 启动项目\n* 部署Elasticsearch/Elasticsearch集群，创建索引：ring_log。\n* 部署Kafka/Kafka集群，创建Topic：ring_log。\n* 填写resources/application.properties配置文件，填写Elasticsearch配置、Kafka配置、UDP/TCP监听服务配置。\n* 启动项目，访问控制台 http://127.0.0.1:9000/#/ ，控制台自带三条件通配符检索功能，条件value为空则不启用该条件检索，即最多只能设置三个检索条件，控制台登录账号密码为项目连接elasticsearch的账号密码。\n\n![avatar](./img/img3.jpg)\n\n（通配符检索规则：'hi*'：tag、content、note、host、file这些字段中有以'hi'为前缀的值，'*'代表一个或多个字符）\n\n* 或使用Kibana连接Elasticsearch并检索日志。\n\n![avatar](./img/img2.jpg)\n***\n\n### 日志收集方式\n可使用UDP或TCP监听接收外部日志。\n#### UDP/TCP客户端发送日志信息\n先将日志信息转为JSON字符串。\n* 客户端将多条日志信息合并发送\n```json\n[\n  {\n    \"lv\":6,\n    \"tag\":\"test-0.0.1\",\n    \"content\":\"redigo: nil returned\",\n    \"note\":\"this is a test log\",\n    \"host\":\"88.88.88.88:8888\",\n    \"file\":\"/service/test.go\",\n    \"line\":15,\n    \"time\":1648380678385\n  },\n  {\n    \"lv\":6,\n    \"tag\":\"test-0.0.1\",\n    \"content\":\"redigo: nil returned\",\n    \"note\":\"this is a test log\",\n    \"host\":\"88.88.88.88:8888\",\n    \"file\":\"/service/test.go\",\n    \"line\":15,\n    \"time\":1648380678385\n  }\n]\n```\n```json\n\"[{\\\"lv\\\":1,\\\"tag\\\":\\\"test-0.0.1\\\",\\\"content\\\":\\\"hello word\\\",\\\"note\\\":\\\"this is a test log\\\",\\\"host\\\":\\\"88.88.88.88:8888\\\",\\\"file\\\":\\\"/service/test.go\\\",\\\"line\\\":15,\\\"time\\\":1648380678385},{\\\"lv\\\":1,\\\"tag\\\":\\\"test-0.0.1\\\",\\\"content\\\":\\\"hello word\\\",\\\"note\\\":\\\"this is a test log\\\",\\\"host\\\":\\\"88.88.88.88:8888\\\",\\\"file\\\":\\\"/service/test.go\\\",\\\"line\\\":15,\\\"time\\\":1648380678385}]\"\n```\n\n* 客户端直接发送单条日志信息\n```json\n[\n  {\n    \"lv\":6,\n    \"tag\":\"test-0.0.1\",\n    \"content\":\"redigo: nil returned\",\n    \"note\":\"this is a test log\",\n    \"host\":\"88.88.88.88:8888\",\n    \"file\":\"/service/test.go\",\n    \"line\":15,\n    \"time\":1648380678385\n  }\n]\n```\n```json\n\"[{\\\"lv\\\":1,\\\"tag\\\":\\\"test-0.0.1\\\",\\\"content\\\":\\\"hello word\\\",\\\"note\\\":\\\"this is a test log\\\",\\\"host\\\":\\\"88.88.88.88:8888\\\",\\\"file\\\":\\\"/service/test.go\\\",\\\"line\\\":15,\\\"time\\\":1648380678385}]\"\n```\n* 再将该JSON字符串转为字节数组，使用UDP/TCP发送到RingLog服务端，服务端UDP监听端口默认为8999，TCP监听端口默认为8998。\n\n#### 日志字段说明\n* lv 日志等级（整数表示，范围：1~8）\n* tag 日志标签（可以是所属项目名+项目版本号）\n* content 日志内容\n* note 日志备注（可以是对该日志的简短描述，或是额外的标注补充）\n* host 日志所属的主机（可以是主机名称或主机地址+应用程序端口号）\n* file 产生该日志的文件路径\n* line 日志产生于该文件的第几行（int类型）\n* time 日志产生时间（毫秒级时间戳，13位，long整数）\n\n#### 日志等级说明\n日志等级 lv（int类型）\n* 1 Emergency: system is unusable 导致系统不可用的事故\n* 2 Alert: action must be taken immediately 必须立即处理的问题\n* 3 Critical: critical conditions 需要立即修复的紧急情况\n* 4 Error: error conditions 运行时出现的错误，不需要立即进行修复\n* 5 Warning: warning conditions 可能影响系统功能，需要提醒的重要事件\n* 6 Notice: normal but significant condition 不影响正常功能，但需要注意的消息\n* 7 Informational: informational messages 系统正常运行情况下的一般信息\n* 8 Debug: debug-level messages 开发时对系统进行诊断的信息\n\n#### 客户端代码示例\n* Go语言，UDP发送日志\n```\n//日志信息结构体\ntype Log struct {\n\tLv int `json:\"lv\"`\n\tTag string `json:\"tag\"`\n\tContent string `json:\"content\"`\n\tNote string `json:\"note\"`\n\tHost string `json:\"host\"`\n\tFile string `json:\"file\"`\n\tLine int `json:\"line\"`\n\tTime int64 `json:\"time\"`\n}\n\n//单条日志信息\nlog := Log{\n\tLv: 1,\n\tTag: \"test-0.0.1\",\n\tContent: \"hello word\",\n\tNote: \"this is a test log\",\n\tHost: \"88.88.88.88:8888\",\n\tFile: \"/service/test.go\",\n\tLine: 15,\n\tTime: 1648380678385,\n}\n\n//日志信息列表\nvar logList []Log\n\n//将多条日志信息插入日志信息列表\nfor i:=0;i\u003c10;i++ {\n\tlogList = append(logList, log)\n}\n\n//将日志信息列表转为字节数组[]byte\nlogListBytes, _ := json.Marshal(logList)\n\t\n//配置UDP连接信息\nsocket, _ := net.DialUDP(\"udp\", nil, \u0026net.UDPAddr{\n\tIP:   net.IPv4(127, 0, 0, 1),   //RingLog服务端地址\n\tPort: 8999,                     //RingLog服务端口\n})\n\n//发送日志数据\n_, _ = socket.Write(logListBytes)\n\n//关闭socket\n_ = socket.Close()\n```\n\n***\n\n### Elasticsearch数据存储\n#### ring_log索引（用于存储所有日志信息）\n* 存储文档格式\n```json\n{\n  \"_index\": \"ring_log\",\n  \"_type\": \"_doc\",\n  \"_id\": \"X4UUIoAB2Qo5g_1D2_nM\",\n  \"_version\": 1,\n  \"_score\": 0,\n  \"fields\": {\n    \"note\": [\n      \"this is a test log\"\n    ],\n    \"file\": [\n      \"/service/test.go\"\n    ],\n    \"line\": [\n      \"15\"\n    ],\n    \"host\": [\n      \"88.88.88.88:8888\"\n    ],\n    \"lv\": [\n      \"6\"\n    ],\n    \"time\": [\n      \"1648380678385\"\n    ],\n    \"tag\": [\n      \"test-0.0.1\"\n    ],\n    \"_class\": [\n      \"com.dpwgc.ringlog.dao.LogMsg\"\n    ],\n    \"_class.keyword\": [\n      \"com.dpwgc.ringlog.dao.LogMsg\"\n    ],\n    \"content\": [\n      \"redigo: nil returned\"\n    ]\n  }\n}\n```\n\n*** \n\n### 项目结构\n* config `配置类`\n   * EsConfig.java `Elasticsearch连接配置`\n   * TcpConfig.java `TCP监听配置`\n   * UdpConfig.java `UDP监听配置`\n* controller `控制器层`\n   * HelloController.java `web访问接口`\n* dao `模板层`\n   * LogMsg.java `日志消息模板`\n* interceptor `拦截器`\n   * ApiInterceptor.java `HTTP接口拦截器`\n* mapper `接口层`\n   * EsMapper.java `Elasticsearch操作接口`\n* server `监听服务层`\n   * KafkaServer.java `Kafka消息队列消费者服务`\n   * MqServer.java `本地缓冲队列消费者服务`\n   * TcpServer.java `TCP监听消息服务`\n   * UdpServer.java `UDP监听消息服务`\n* util ``\n   * LogUtil.java `日志操作工具`\n   * EsUtil.java `Elasticsearch操作工具`\n* RinglogApplication.java `启动类`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpwgc%2Fringlog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpwgc%2Fringlog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpwgc%2Fringlog/lists"}