{"id":28729747,"url":"https://github.com/didi/falcon-log-agent","last_synced_at":"2025-06-15T17:10:52.912Z","repository":{"id":38428756,"uuid":"138727868","full_name":"didi/falcon-log-agent","owner":"didi","description":"用于监控系统的日志采集agent，可无缝对接open-falcon","archived":false,"fork":false,"pushed_at":"2023-05-05T02:29:35.000Z","size":10324,"stargazers_count":592,"open_issues_count":5,"forks_count":154,"subscribers_count":31,"default_branch":"master","last_synced_at":"2025-05-30T22:44:36.626Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/didi.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-06-26T11:22:43.000Z","updated_at":"2025-05-29T09:34:30.000Z","dependencies_parsed_at":"2023-09-29T08:52:05.161Z","dependency_job_id":null,"html_url":"https://github.com/didi/falcon-log-agent","commit_stats":{"total_commits":31,"total_committers":10,"mean_commits":3.1,"dds":"0.29032258064516125","last_synced_commit":"6c58b8e220d86b07e902a7db05273960ca1ec9ca"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2Ffalcon-log-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2Ffalcon-log-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2Ffalcon-log-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2Ffalcon-log-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/didi","download_url":"https://codeload.github.com/didi/falcon-log-agent/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2Ffalcon-log-agent/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259099667,"owners_count":22804940,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-15T17:10:51.802Z","updated_at":"2025-06-15T17:10:52.898Z","avatar_url":"https://github.com/didi.png","language":"Go","funding_links":["https://opencollective.com/falcon-log-agent"],"categories":[],"sub_categories":[],"readme":"# falcon-log-agent \n\n![log-agent](./pic/logo.png)\n\n[![Build Status](https://img.shields.io/github/stars/didi/falcon-log-agent.svg)](https://github.com/didi/falcon-log-agent)\n[![Build Status](https://img.shields.io/github/forks/didi/falcon-log-agent.svg)](https://github.com/didi/falcon-log-agent)\n[![Build Status](https://img.shields.io/github/license/mashape/apistatus.svg)](https://github.com/didi/falcon-log-agent)\n[![Backers on Open Collective](https://opencollective.com/falcon-log-agent/backers/badge.svg)](#backers) \n[![Sponsors on Open Collective](https://opencollective.com/falcon-log-agent/sponsors/badge.svg)](#sponsors) \n\n# 目录\n- [简介](#简介)\n- [Feature](#Feature)\n- [什么是日志采集](#什么是日志采集)\n- [falcon-log-agent如何工作](#falcon-log-agent如何工作)\n- [限定条件](#限定条件)\n- [开始使用log-agent](#开始使用log-agent)\n  * [构建](#)\n  * [修改配置文件](#)\n  * [启动/停止服务](#)\n- [基础配置项](#基础配置项)\n  * [日志相关](#)\n  * [worker相关](#)\n  * [资源限制](#)\n  * [策略相关](#)\n  * [其他](#)\n- [采集策略](#采集策略)\n  * [文件路径](#文件路径)\n  * [时间格式](#时间格式)\n  * [采集规则](#采集规则)\n  * [采集周期](#采集周期)\n  * [采集方式](#采集方式)\n  * [采集名称](#采集名称)\n  * [标签](#标签)\n  * [其他](#其他)\n- [检验日志格式](#检验日志格式)\n- [自身状态暴露](#自身状态暴露)\n- [自监控](#自监控)\n\n# 简介\nfalcon-log-agent是一个开源版的日志采集工具，旨在从流式的日志中抓取、统计日志中的特征信息。\n\n获取的特征信息，与开源版Open-Falcon监控系统打通。可用于业务指标的衡量、也可用于稳定性的建设。\n\n# Feature\n- **准确可依赖**：历经滴滴线上业务近一年考验，统计准确性高。\n- **性能高、资源消耗可控**：性能优化程度高，单核单策略可支撑日志分析:20W条/秒\n- **接入成本低**：外挂式采集，只需要标准化日志即可；输出数据直接对接open-falcon。\n\n附：我司agent升级前后资源占用对比图\n![资源对比图](./pic/resource.png)\n\n\n# 什么是日志采集\n日志采集，是一种外挂式的采集。通过读取进程打印的日志，来进行监控数据的采集与汇聚计算。\n\n# falcon-log-agent如何工作\n本agent即日志采集场景下的实时计算。实时读取文件内容，实时计算，将计算结果直接推送至falcon。\n\n# 限定条件\n- **要求日志必须包含时间**：不包含时间的日志，只能根据当前时间统计日志条数，结果非常不准确。\n- **不支持文件软链**\n- **日志时间必须有序**：为了应对日志延迟落盘等，agent会根据日志的时间来判断某一周期的数据是否采集完成，如果日志时间顺序错乱，可能导致采集不准。\n\n# 开始使用log-agent\n\n**构建**\n```\ngo get https://github.com/didi/falcon-log-agent.git \u0026\u0026 cd $GOPATH:/src/github.com/didi/falcon-log-agent\nmake build\n```\n\n**修改配置文件**\n```\n# base config\ncp cfg/dev.cfg cfg/cfg.json\nvim cfg/cfg.json\n\n# strategy config\ncp cfg/strategy.dev.json cfg/strategy.json\nvim cfg/strategy.json\n```\n\n**打包 \u0026 安装**\n```\nmake pack\nexport WorkDir=\"$HOME/falcon-log-agent\"\nmkdir -p $WorkDir\ntar -xzvf falcon-log-agent.tar.gz -C $WorkDir\ncd $WorkDir\n```\n\n\n**启动/停止服务**\n```\n # start\n./control start\n\n# stop\n./control stop\n\n# status\n./control status\n```\n\n# 基础配置项\n基础配置项，即程序本身的配置项。默认是cfg/cfg.json，可以通过-c参数来指定。\n\n**日志相关**\n```\nlog_path：程序输出的日志目录\nlog_level：日志等级\nlog_rotate_size：日志切割大小\nlog_rotate_num：按配置切割之后，保留多少个文件，其他的清理掉\n```\n\n**worker相关**\n```\nworker_num：每个日志文件，进行计算的并发数\nqueue_size：读文件和进行计算之间，有一个缓冲队列，如果队列满了，意味着计算能力跟不上，就要丢日志了。这个配置就是这个缓冲队列的大小。\npush_interval：循环判断将计算完成的数据推送至发送队列的时间\npush_url：推送的odin-agent的url\n```\n\n**资源限制**\n```\nmax_cpu_rate:最大使用的cpu百分比。（可用核数=ceil(总核数*max_cpu_rate))\nmax_mem_rate:最大使用内存百分比。(最大内存=(内存总大小*max_mem_rate)，最小为500M)\n```\n\n**策略相关**\n```\nupdate_duration:策略的更新周期\ndefault_degree:默认的采集精度\n```\n\n**其他**\n```\nhttp_port:自身状态对外暴露的接口\nendpoint:上报至open-falcon的endpoint配置。(可选host或ip,host为主机名,ip为本机ip)\n```\n\n# 采集策略\n\n## 文件路径\n\n文件路径，即file_path配置项。**必须要求启动agent的用户，对这个文件有可读权限**。\n\n文件路径支持固定路径和动态路径两种：\n- 固定路径：直接填写即可，如/var/log/falcon-log-agent.log\n- 动态路径：可支持按照规则配置的根据时间变化的路径。例如：\n\n```\n比如：线上有些模块自己按照小时写入文件，路径为：\n/xiaoju/application/log/20150723/application.log.2015072312\n \n对应的我们的配置方式可以填写为：\n/xiaoju/application/log/${%Y%m%d}/application.log.${%Y%m%d%H}    //  ${}中不能包含/\n```\n\n## 时间格式\n\n时间格式，即time_format配置项。\n\n如果日志中没有时间格式，一旦遇到日志延迟落盘、或者日志量太大计算延迟的情况。会直接导致我们的监控采集不准。\n\n因此，我们规定日志中必须有合法的时间格式。且在配置中time_format项指定。\n\n如果想要添加自己的时间格式，可以直接在[common/utils/util.go](https://github.com/didi/falcon-log-agent/blob/master/src/common/utils/util.go)里添加。\n\n目前已经支持的时间格式如下：\n```\ndd/mmm/yyyy:HH:MM:SS\ndd/mmm/yyyy HH:MM:SS\nyyyy-mm-ddTHH:MM:SS\ndd-mmm-yyyy HH:MM:SS\nyyyy-mm-dd HH:MM:SS\nyyyy/mm/dd HH:MM:SS\nyyyymmdd HH:MM:SS\nmmm dd HH:MM:SS\n\nPS：为了防止日志积压或性能不足导致的计算偏差，日志采集的计算，依赖于日志的时间戳。\n因此如果配置了错误的时间格式，将无法得到正确的结果。\n```\n\n## 采集规则\n\n采集正则，包含两个配置项：pattern和exclude。\n\n两个采集项都是正则表达式，正则表达式的支持情况见：[google/re2](https://github.com/google/re2/wiki/Syntax)\n\npattern代表需要完全匹配出来的表达式。\n\nexclude代表需要排除掉的表达式。\n\n```\neg. 例如，我希望统计code=500或400的日志数量，但是想排除掉关键字SpeciallyErrorNo。 配置如下：\n\npattern: code=[45]00\nexclude: SpeciallyErrorNo\n```\n\n## 采集周期\n\n采集周期(step)，对应着监控系统的上报周期。意味着多久合并上报一次。\n```\n假设每秒产生1条符合采集规则的日志，配置的采集方式为计数。\n如果step为10 : 则每10s上报一次，值为10\n如果step为60 : 则每60s上报一次，值为60\n```\n\n## 采集方式\n\n采集方式(func)的意思是，当我们从日志中筛选出一堆符合规则的日志之后，应该以哪种规则来计算拿到最后的值来上报。\n\n目前支持的采集方式有：\n- cnt\n- avg\n- sum\n- max\n- min\n\n举例：\n```\n假设：\n正则表达式配置为 Return Success : (\\d+)s Used\n \n某一个周期内日志滚动：\n2017/12/01 12:12:01 Return Success : 1s Used\n2017/12/01 12:12:02 Return Success : 2s Used\n2017/12/01 12:12:03 Return Success : 4s Used\n2017/12/01 12:12:04 Return Success : 2s Used\n2017/12/01 12:12:05 Return Success : 1s Used\n \n首先，根据正则获取到括号内的值：1、2、4、2、1\n接下来，根据不同的计算方式，会得到不同的结果：\navg   : (1 + 2 + 4 + 2 + 1) / 5 = 2\ncount : 5\nsum   : (1 + 2 + 4 + 2 + 1) = 10\nmax   : Max(1, 2, 4, 2, 1) = 4\nmin   : Min(1, 2, 4, 2, 1) = 1\n```\n\n## 采集名称\n\n**采集名称**(name)对应open-falcon中的metric，即监控项。\n\n## 标签\n\n**标签**(tags)与open-falcon中的tags相对应。可以理解为确定监控项的补充。\n```\n说明：机器A的第一个核的cpu空闲率。\n\n采集名称(metric): cpu空闲率(cpu.idle)\n标签(tags)：两个标签: host=机器A;核数=第一个核\n```\n在主正则匹配完成后，然后匹配出tag的值，一起进行上报。\n\n若无法匹配出tag的值，则视为该条数据未匹配到，该条日志将**不再计入统计**。\n\n## 其他\n\n- degree: 精度\n- comment: 备注\n\n# 检验日志格式\n启动agent，会自动加载所有策略。此时通过**/check**接口，可以实时验证日志是否可以匹配到策略。\n/check接口会将该条日志能命中的采集规则，一起返回，并返回命中详情。\n```\n方法：POST\n参数：log=${日志原文}  // postForm\n\neg.\ncurl -s -XPOST localhost:8003/check -d  'log=01/Jan/2018:12:12:12 service error 500, num=10 province=33' | python -m json.tool\n```\n\n\n# 自身状态暴露\nfalcon-log-agent本身对外提供了一个http服务用来暴露自身状态。\n\n主要提供的url如下：\n- /health  ： 自身存活状态\n- /strategy ：当前生效的策略列表\n- /cached ： 最近1min内上报的点\n\n\n# 自监控\n在[common/proc/metric/metric.go](https://github.com/didi/falcon-log-agent/blob/master/common/proc/metric/metric.go#L38)定义了一个自监控结构体。\n\n在程序运行过程中会不断收集信息，主要包括如下：\n```\nMemUsedMB       进程内存占用\nReadLineCnt     读日志行数\nDropLineCnt     队列打满后，扔掉的日志行数\nAnalysisCnt     分析完成的日志行数\nAnalysisSuccCnt 分析成功匹配的日志行数\nPushCnt         推送的监控数据点数\nPushErrorCnt    推送错误的监控数据点数\nPushLatency     推送监控数据延迟\n```\n这些数据，目前自监控的处理方式是：定时输出日志。\n\n如果需要对接自己公司的监控系统，在[common/proc/metric/metric.go](https://github.com/didi/falcon-log-agent/blob/master/common/proc/metric/metric.go#L81)修改HandleMetrics方法即可。\n\n# 贡献者\n- [**高家升**](https://github.com/GaoJiasheng)\n- [**安宝勇**](https://github.com/anbaoyong)\n- [wcc526](https://github.com/wcc526)\n- [mdh67899](https://github.com/mdh67899)\n- [1Feng](https://github.com/1Feng)\n\n\n## Contributors\n\nThis project exists thanks to all the people who contribute. \n\u003ca href=\"https://github.com/didi/falcon-log-agent/graphs/contributors\"\u003e\u003cimg src=\"https://opencollective.com/falcon-log-agent/contributors.svg?width=890\u0026button=false\" /\u003e\u003c/a\u003e\n\n\n## Backers\n\nThank you to all our backers! 🙏 [[Become a backer](https://opencollective.com/falcon-log-agent#backer)]\n\n\u003ca href=\"https://opencollective.com/falcon-log-agent#backers\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/falcon-log-agent/backers.svg?width=890\"\u003e\u003c/a\u003e\n\n\n## Sponsors\n\nSupport this project by becoming a sponsor. Your logo will show up here with a link to your website. [[Become a sponsor](https://opencollective.com/falcon-log-agent#sponsor)]\n\n\u003ca href=\"https://opencollective.com/falcon-log-agent/sponsor/0/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/falcon-log-agent/sponsor/0/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/falcon-log-agent/sponsor/1/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/falcon-log-agent/sponsor/1/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/falcon-log-agent/sponsor/2/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/falcon-log-agent/sponsor/2/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/falcon-log-agent/sponsor/3/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/falcon-log-agent/sponsor/3/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/falcon-log-agent/sponsor/4/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/falcon-log-agent/sponsor/4/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/falcon-log-agent/sponsor/5/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/falcon-log-agent/sponsor/5/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/falcon-log-agent/sponsor/6/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/falcon-log-agent/sponsor/6/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/falcon-log-agent/sponsor/7/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/falcon-log-agent/sponsor/7/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/falcon-log-agent/sponsor/8/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/falcon-log-agent/sponsor/8/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/falcon-log-agent/sponsor/9/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/falcon-log-agent/sponsor/9/avatar.svg\"\u003e\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdidi%2Ffalcon-log-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdidi%2Ffalcon-log-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdidi%2Ffalcon-log-agent/lists"}