{"id":19096497,"url":"https://github.com/sapcc/go-pmtud","last_synced_at":"2025-04-30T14:11:54.748Z","repository":{"id":41954482,"uuid":"229103556","full_name":"sapcc/go-pmtud","owner":"sapcc","description":"go-pmtud is a simplified implementation of cloudflare/pmtud in Go.","archived":false,"fork":false,"pushed_at":"2025-04-17T01:28:13.000Z","size":18100,"stargazers_count":9,"open_issues_count":4,"forks_count":0,"subscribers_count":46,"default_branch":"master","last_synced_at":"2025-04-17T15:12:32.451Z","etag":null,"topics":["bpg","ecmp","kubernetes","path-mtu-discovery"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sapcc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-12-19T17:14:32.000Z","updated_at":"2025-04-04T09:27:55.000Z","dependencies_parsed_at":"2023-10-12T05:27:27.600Z","dependency_job_id":"1f103851-f75d-4ae9-9873-a7a6924942ae","html_url":"https://github.com/sapcc/go-pmtud","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sapcc%2Fgo-pmtud","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sapcc%2Fgo-pmtud/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sapcc%2Fgo-pmtud/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sapcc%2Fgo-pmtud/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sapcc","download_url":"https://codeload.github.com/sapcc/go-pmtud/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251717863,"owners_count":21632370,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bpg","ecmp","kubernetes","path-mtu-discovery"],"created_at":"2024-11-09T03:36:52.946Z","updated_at":"2025-04-30T14:11:54.692Z","avatar_url":"https://github.com/sapcc.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# go-pmtud\n\n[![CI](https://github.com/sapcc/go-pmtud/actions/workflows/ci.yaml/badge.svg)](https://github.com/sapcc/go-pmtud/actions/workflows/ci.yaml)\n[![Go Report Card](https://goreportcard.com/badge/github.com/sapcc/go-pmtud)](https://goreportcard.com/report/github.com/sapcc/go-pmtud)\n\n`go-pmtud` is a simplified implementation of [cloudflare/pmtud](https://github.com/cloudflare/pmtud) in Go.\n\n## Problem\n\nUsing ECMP (Equal Cost Multi Path) on bare metal Kubernetes clusters makes load sharing of traffic possible (e.g. by using service addresses of type `ExternalIP`).\n\nHosts (and Pods) try to leverage full MTU size that is derived from their interface configuration (e.g. 9000 bytes).\n\nIf `(1)` MTU is smaller somewhere in the path between sender and receiver and `(2)` packet has a DF (do-not-fragment) bit set, router sends ICMP Destination Unreachable message (type 3 code 4 message) to sender (originator of too large packets).\n\nIn case of ECMP it may not reach the original sender, thus breaking the communication.\n\nMore details in this blog post by Cloudflare: [Path MTU discovery in practice](https://blog.cloudflare.com/path-mtu-discovery-in-practice/).\n\ngo-pmtud replicates ICMP Destination Unreachable packets to all nodes in same Kubernetes cluster, so that the sender gets awareness that it has to use smaller packets for a particular destination.\n\n## Concept\n\n1. ICMP Destination Unreachable message (type 3 code 4 message) packets are filtered and sent to specific NFlog group by iptables.\n\n2. go-pmtud replicates ICMP packets to all nodes in same Kubernetes cluster, so that the sender pod gets awareness that it has to use smaller packets for particular destination.\n\n```\nExec into the pod:\n\n# ip route get 192.168.100.10\n192.168.100.10 via 192.100.0.1 dev eth0 src 192.100.0.50\n    cache  expires 484sec mtu 9000  \u003c\u003c\u003c\u003c connection is failing\n\n# ip route get 192.168.100.10\n192.168.100.10 via 192.100.0.1 dev eth0 src 192.100.0.50\n    cache  expires 484sec mtu 8996  \u003c\u003c\u003c\u003c correct MTU information, connection is working\n```\n\n## Build\n\nBuild from source:\n\n```\ngo mod download\ngo build -v -o /go-pmtud cmd/go-pmtud/main.go\n```\n\nBuild a Docker image:\n\n```\ndocker build -t go-pmtud .\n```\n\n## go-pmtud options\n\nFollowing options are available:\n\n1. peers - resend ICMP frag-needed packets to this peer list.\n2. iface - interface that listens for ICMP packets and resends them to other peers.\n3. nodename - node hostname, used for metric label.\n4. nflog-group - NFLOG group, set to 33 in our case.\n5. metrics-port - Port for Prometheus metrics (30040 by default).\n6. ttl - TTL of replicated ICMP packets.\n7. ignore-networks - Do not resend ICMP frag-needed packets originated from specified networks\n\nIf `iface` is empty, it finds out the outgoing interface based on the default route. \n\n## Example - go-pmtud Daemonset\n\ngo-pmtud can run as a [Daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/), [example](https://github.com/sapcc/helm-charts/blob/master/system/go-pmtud).\n\nExample values.yaml:\n\n```\nimages:\n  iptables:\n    repository: sapcc/iptables\n    tag: v20191226161919\n  pmtud:\n    repository: sapcc/go-pmtud\n    tag: latest\n\niptables:\n  nflogGroup: 33\n  ignoreSourceNetworks: 192.168.100.0/24\n\npmtud:\n  ttl: 10\n  metricsPort: 30040\n  interface: eth0\n  peers: 192.168.100.2, 192.168.100.3, 192.168.100.4, 192.168.100.5, 192.100.0.50\n```\n\n## Example - iptables and NFlog\n\nThere is an iptables rule on each node that redirects ICMP Destination Unreachable` packets to NFlog group nr. 33:\n\n`iptables -t raw -D PREROUTING -i \u003cinterface\u003e -p icmp -m icmp --icmp-type 3/4 --j NFLOG --nflog-group 33`\n\nImportant: we need ignore packets from summarized source networks of all nodes in the local cluster to avoid re-sending loops. Use `ignore-networks` option for this. \nThis means a node will not re-send already retransmitted ICMP messages. It will only resend messages that are usually originated by routers on the path. \n\n## License\nThis project is licensed under the Apache2 License - see the [LICENSE](LICENSE) file for details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsapcc%2Fgo-pmtud","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsapcc%2Fgo-pmtud","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsapcc%2Fgo-pmtud/lists"}