{"id":42052203,"url":"https://github.com/xuesongtao/ps-log","last_synced_at":"2026-01-26T07:03:39.425Z","repository":{"id":65315537,"uuid":"587304695","full_name":"xuesongtao/ps-log","owner":"xuesongtao","description":"1. 支持定时/实时去解析多个 log 文件 2.支持多个匹配规则, 匹配的内容支持不同的处理方式(支持同步/异步处理) 3.采用文件池将频繁使用的句柄进行缓存, 采用tire树缓存匹配规则","archived":false,"fork":false,"pushed_at":"2024-12-26T07:35:10.000Z","size":335,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-26T08:26:11.837Z","etag":null,"topics":["collect","parse-log","real-time-collect","tail","tail-logs"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xuesongtao.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-10T12:52:23.000Z","updated_at":"2024-12-26T07:35:15.000Z","dependencies_parsed_at":"2024-06-27T17:05:26.810Z","dependency_job_id":"a0168647-3c31-4e7b-a63b-fe105b936627","html_url":"https://github.com/xuesongtao/ps-log","commit_stats":null,"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"purl":"pkg:github/xuesongtao/ps-log","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuesongtao%2Fps-log","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuesongtao%2Fps-log/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuesongtao%2Fps-log/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuesongtao%2Fps-log/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xuesongtao","download_url":"https://codeload.github.com/xuesongtao/ps-log/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuesongtao%2Fps-log/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28769226,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T06:37:25.426Z","status":"ssl_error","status_checked_at":"2026-01-26T06:37:23.039Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["collect","parse-log","real-time-collect","tail","tail-logs"],"created_at":"2026-01-26T07:03:19.267Z","updated_at":"2026-01-26T07:03:39.410Z","avatar_url":"https://github.com/xuesongtao.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ps-log 日志分析\r\n\r\n#### 项目背景\r\n\r\n1. 开发/测试/生产环境(**微服务数量在 40+ 情况下**), 出现了 error log等不能被开发感知, 当反馈到开发时时间间隔较长, 如何解决?\r\n    \u003e **解决**: 定时(如:10m)去解析 log 中每行包含 error 的内容, 再进行对应的处理(如: 发钉钉, 发邮件, 发es等)\r\n2. 定时去分析 log 吗, 需要实时感知 error log 怎么办呢?\r\n    \u003e **解决**: 通过文件事件通知来感知文件变化呀, 有变化的时候就去查看文件内容呀\r\n3. 实时感知的文件需要起多个监听任务吗? 需要多次打开相同的文件怎么处理呢?\r\n    \u003e **解决**: 不需要,只需要1个监听者,1个处理者; 可以通过池化文件句柄\r\n\r\n#### 介绍\r\n\r\n```go\r\ngo get -u gitee.com/xuesongtao/ps-log\r\n```\r\n\r\n1. 支持 **定时/实时** 去解析多个 log 文件; 采集完后会根据配置进行采集位置的持久化保存(即: 文件偏移量保存), 便于停机后重启防止出现重复采集现象(注: Change 设置的比较大时, 需要注意处理重启服务时, 偏移量未保存出现的重复数据)\r\n2. 支持 log `行内容` 多个匹配规则; 支持解析**错误堆栈**(即: 支持行内容合并); 匹配的内容支持不同的处理方式(支持同步/异步处理)\r\n3. 采用文件池将频繁使用的句柄进行缓存; 采用 `trie` 树缓存匹配规则提高匹配效率\r\n\r\n![简易流程图](https://gitee.com/xuesongtao/ps-log/raw/master/ps-log.png)\r\n\r\n#### 使用\r\n\r\n##### 实时监听\r\n\r\n```go\r\nfunc main() {\r\n\tps, err := pslog.NewPsLog(pslog.WithAsync2Tos())\r\n\tif err != nil {\r\n\t\tpanic(err)\r\n\t}\r\n\tdefer ps.Close()\r\n\r\n\t// 实时监听\r\n\tif err := ps.TailLogs(); err != nil {\r\n\t\tpanic(err)\r\n\t}\r\n\r\n\ttmp := \"log/test.log\"\r\n\thandler := \u0026pslog.Handler{\r\n\t\tCleanOffset: true,           // 重新加载时, 清理已保存的 文件偏移量\r\n\t\tChange:      -1,             // 每次都保存文件偏移量\r\n\t\tTail:        true,           // 实时监听\r\n\t\tExpireAt:    pslog.NoExpire, // 不过期\r\n\t\tTargets: []*pslog.Target{\r\n\t\t\t{\r\n\t\t\t\tContent:  \" \",        // 目标内容\r\n\t\t\t\tExcludes: []string{}, // 排查内容\r\n\t\t\t\tTo:       []pslog.PsLogWriter{\u0026pslog.Stdout{}},\r\n\t\t\t},\r\n\t\t},\r\n\t}\r\n\r\n\t// 注册\r\n\tif err := ps.Register(handler); err != nil {\r\n\t\tpanic(err)\r\n\t}\r\n\tcloseCh := make(chan int)\r\n\tgo func() {\r\n\t\tfh := xfile.NewFileHandle(tmp)\r\n\t\tif err := fh.Initf(os.O_RDWR | os.O_APPEND | os.O_TRUNC); err != nil {\r\n\t\t\tlog.Println(err)\r\n\t\t\treturn\r\n\t\t}\r\n\t\tdefer fh.Close()\r\n\t\tfor i := 0; i \u003c 10; i++ {\r\n\t\t\ttime.Sleep(10 * time.Millisecond)\r\n\t\t\t_, err := fh.AppendContent(time.Now().Format(base.DatetimeFmt+\".000\") + \" \" + fmt.Sprint(i) + \"\\n\")\r\n\t\t\tif err != nil {\r\n\t\t\t\tlog.Println(\"write err:\", err)\r\n\t\t\t}\r\n\t\t}\r\n\t\tclose(closeCh)\r\n\t}()\r\n\r\n\t// 添加待监听的 path\r\n\tif err := ps.AddPaths(tmp); err != nil {\r\n\t\tpanic(err)\r\n\t}\r\n\r\n\t// dump\r\n\tlog.Println(ps.List())\r\n\tfor range closeCh {\r\n\t}\r\n}\r\n```\r\n\r\n#### 其他\r\n\r\n- 采集服务示例: [gitee](https://gitee.com/xuesongtao/collect-log.git)\r\n\r\n- 欢迎大佬们指正, 希望大佬给❤️，to [gitee](https://gitee.com/xuesongtao/ps-log.git), [github](https://github.com/xuesongtao/ps-log.git)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxuesongtao%2Fps-log","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxuesongtao%2Fps-log","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxuesongtao%2Fps-log/lists"}