{"id":37354018,"url":"https://github.com/kubeservice-stack/pingmesh-agent","last_synced_at":"2026-01-16T04:08:45.064Z","repository":{"id":60413066,"uuid":"541499673","full_name":"kubeservice-stack/pingmesh-agent","owner":"kubeservice-stack","description":"Pingmesh : A Large-Scale System for Data Center Network Latency. pingmesh-agent is prometheus exporter","archived":false,"fork":false,"pushed_at":"2024-11-28T03:59:10.000Z","size":9182,"stargazers_count":37,"open_issues_count":2,"forks_count":9,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-28T04:21:51.695Z","etag":null,"topics":["exporter","monitoring"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kubeservice-stack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-09-26T09:12:41.000Z","updated_at":"2024-11-28T03:58:30.000Z","dependencies_parsed_at":"2023-12-16T06:41:22.518Z","dependency_job_id":"4a207198-b2cb-4727-81e1-f5ed6f4d057f","html_url":"https://github.com/kubeservice-stack/pingmesh-agent","commit_stats":{"total_commits":23,"total_committers":3,"mean_commits":7.666666666666667,"dds":0.3913043478260869,"last_synced_commit":"da76f1bce59ce12588a7f0625bd6dc48f0c10f9a"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/kubeservice-stack/pingmesh-agent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeservice-stack%2Fpingmesh-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeservice-stack%2Fpingmesh-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeservice-stack%2Fpingmesh-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeservice-stack%2Fpingmesh-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kubeservice-stack","download_url":"https://codeload.github.com/kubeservice-stack/pingmesh-agent/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeservice-stack%2Fpingmesh-agent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28477207,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T03:13:13.607Z","status":"ssl_error","status_checked_at":"2026-01-16T03:11:47.863Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["exporter","monitoring"],"created_at":"2026-01-16T04:08:44.360Z","updated_at":"2026-01-16T04:08:45.047Z","avatar_url":"https://github.com/kubeservice-stack.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 基于blackbox构建的Pingmesh体系\n\n## 背景\n\n数据中心自身是极为复杂的，其中网络涉及到的设备很多就显得更为复杂，一个大型数据中心都有成百上千的节点、网卡、交换机、路由器以及无数的网线、光纤。在这些硬件设备基础上构建了很多软件，比如搜索引擎、分布式文件系统、分布式存储等等。在这些系统运行过程中，面临一些问题：如何判断一个故障是网络故障？如何定义和追踪网络的 SLA？出了故障如何去排查？\n\n![IDC](https://kubeservice.cn/img/devops/IDC_hu8ec2fdff58b0ea09e7358f84cbaf1df1_175984_filter_3454788233369042773.png)\n\n`网络性能数据监控` 就比较困难实现。 如果单纯直接使用 `ping` 命令收集结果，`每台`服务器去 ping 剩下 `(N-1)` 台，也就是 `N^2` 的复杂度，稳定性和性能都存在一些问题。\n\n举个例子： \n如果IDC中有10000台服务器，ping的任务就有，`10000*9999` 任务, 如果一台机器有多IP请求，结果再翻倍。\n\n对于数据存储也是一个问题，如果是每30s进行一次ping， 一次ping 需要 payload大小是64bytes\n数据存储量： `10000*9999*2*64*24*3600/30` = `3.6860314e+13 bytes` = `33.52TB`\n\n是否只记录`fail`和`timeout`的记录，可以节约`99.99%`的存储空间\n\n## 业界实现\n\n本体系是基于`微软Pingmesh论文`一种`增强`实现. \n\n原微软Pingmesh论文地址：\n[《Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis》](https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p139.pdf)\n\n\n对于`微软Pingmesh`是网络监控中一个很好突破。（具体可认真读原文）\n\n但是在实际使用中也有不少局限性：\n\n1. agent数据流： 对于`Agent` 每次ping完都是记录到log中，再通过基础设施进行`log`数据收集，使用`日志分析`系统加大了系统复杂性。\n\n2. Ping 模式支持： 只能支持`UDP`模式， 对于`DNS tcp`、`ICMP ping`等支持比较缺少。\n\n3. Ping维度：只能支持`IPv4`ping。 但很多场景需要支持 是否公网互联互通等 `domain/dns` ping\n\n4. 不支持手动实时尝试ping： 可基于`balckbox-exporter`网络探测实现\n\n5. 不支持ipv6\n\n## Pingmesh升级后的架构\n\n![Pingmesh+](https://kubeservice.cn/img/devops/pingmesh_hu8c196f2563a4108ff3fa8682517063fd_177531_filter_4759638724306006349.png)\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fkubeservice-stack%2Fpingmesh-agent.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fkubeservice-stack%2Fpingmesh-agent?ref=badge_shield)\n\n### Controller\n\n`Controller` 主要负责生成 `pinglist.yaml` 文件。 `pinglist` 的生成来源有3个方向：\n\n\u003e 通过`IP Controller`自动获取到整个集群的podIP 和 nodeIp list\n\n\u003e 通过`Pinglist Controller` 活动`Agent Setting`配置\n\n\u003e 通过`Custom Define Pinglist` 在 `pinglist.yaml` 文件中补充 外部地址。 支持`dns地址`、`外部http地址`、`domain地址`、`ntp地址`、`Kubenetes apiserver地址`等等\n\n`Controller` 在生成 `pinglist` 文件后，通过 `HTTP/HTTPS` 提供出去，`Agent` 会定期获取 `pinglist` 来更新 `agent` 自己的配置，也就是我们说的`拉`模式。`Controller `需要保证高可用，因此需要在 `Service` 后面配置多个实例，每个实例的算法一致，`pinglist` 文件内容也一致，保证可用性\n\n\n### Agent\n每个 ping 动作都开启一个新的连接，为了减少 `Pingmesh` 造成的 `TCP` 并发. 两个server ping 的周期最小是 10s，Packet 大小最大 64kb。\n\n```yaml\nsetting:\n  # the maximum amount of concurrent to ping, uint\n  concurrent_limit: 20\n  # interval to exec ping in seconds, float\n  interval: 60.0\n  # The maximum delay time to ping in milliseconds, float\n  delay: 200\n  # ping timeout in seconds, float\n  timeout: 2.0\n  # send ip addr\n  source_ip_addr: 0.0.0.0\n  # send ip protocal\n  ip_protocol: ip6\n\nmesh:\n  add-ping-public: \n    name: ping-public-demo\n    type: OtherIP\n    ips :\n      - 127.0.0.1\n      - 8.8.8.8\n      - www.baidu.com\n      - kubernetes.default.svc.cluster.local\n```\n\n并且做了`过载保护`\n1. 如果`pinglist`中 数据很多， 在一个周期（比如`10s`）处理不完， 会保证本次处理完成后，在执行下一次， 优先一个轮回完成\n2. 配置可以设置 `agent` 并发线程数，确保 `pingmesh agent` 对整个集群影响小于`千分之一`\n3. metrics中是通过`Promethrus Gauge`, 在每个周期中单独计算\n\n```metrics \n# HELP ping_fail ping fail\n# TYPE ping_fail gauge\nping_fail{target=\"8.8.8.8\",tor=\"ping-public-demo\"} 1\n```\n\n4. 为了确保 ping的请求在一个`时间窗口interval`中平均发出, 对请求job 做了内存态计算，在`并发协程`上做了`ratelimit`\n\n## 网络状况设计\n\n通过`pinglist.yaml`设置中的`interval`时间窗口：\n- 请求超过了`timeout`时间， 将请求标记为 `ping_fail`\n- 请求超过了`delay` 但没有超过`timeout`时间， 将请求标记为 `ping_duration_milliseconds`\n- 请求没有超过`delay` ，在metrics接口中不记录\n\n## 与promtheus集成\n\n将以下文本添加到promtheus.yaml的scrape_configs部分, `pingmeship`为server的ip\n\n```yaml\nscrape_configs:\n\n  - job_name: net_monitor\n    honor_labels: true\n    honor_timestamps: true\n    scrape_interval: 60s\n    scrape_timeout: 5s\n    metrics_path: /metrics\n    scheme: http\n    static_configs:\n    - targets:\n      - $pingmeship:9115\n```\n\n\n## License\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fkubeservice-stack%2Fpingmesh-agent.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fkubeservice-stack%2Fpingmesh-agent?ref=badge_large)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkubeservice-stack%2Fpingmesh-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkubeservice-stack%2Fpingmesh-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkubeservice-stack%2Fpingmesh-agent/lists"}