{"id":18074186,"url":"https://github.com/maxlaverse/snat-race-conn-test","last_synced_at":"2025-08-21T12:46:18.946Z","repository":{"id":42457437,"uuid":"142065068","full_name":"maxlaverse/snat-race-conn-test","owner":"maxlaverse","description":"Small piece of code used to trigger a race condition in Linux's SNAT","archived":false,"fork":false,"pushed_at":"2021-06-09T08:05:47.000Z","size":20,"stargazers_count":45,"open_issues_count":0,"forks_count":14,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-25T08:03:13.797Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maxlaverse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-23T20:20:14.000Z","updated_at":"2024-10-29T23:05:14.000Z","dependencies_parsed_at":"2022-07-09T15:35:05.030Z","dependency_job_id":null,"html_url":"https://github.com/maxlaverse/snat-race-conn-test","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/maxlaverse/snat-race-conn-test","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlaverse%2Fsnat-race-conn-test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlaverse%2Fsnat-race-conn-test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlaverse%2Fsnat-race-conn-test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlaverse%2Fsnat-race-conn-test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maxlaverse","download_url":"https://codeload.github.com/maxlaverse/snat-race-conn-test/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlaverse%2Fsnat-race-conn-test/sbom","scorecard":{"id":629673,"data":{"date":"2025-08-11","repo":{"name":"github.com/maxlaverse/snat-race-conn-test","commit":"66035af96698d7830f453f47dcebf71bc0f57114"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.5,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":1,"reason":"Found 1/7 approved changesets -- score normalized to 1","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":0,"reason":"license file not detected","details":["Warn: project does not have a license file"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating golang:1.16 to golang:1.16@sha256:5f6a4662de3efc6d6bb812d02e9de3d8698eea16b8eb7281f03e6f3e8383018e","Info:   0 out of   1 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 2 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-21T07:28:44.860Z","repository_id":42457437,"created_at":"2025-08-21T07:28:44.860Z","updated_at":"2025-08-21T07:28:44.860Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271484499,"owners_count":24767764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-21T02:00:08.990Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-31T10:11:33.679Z","updated_at":"2025-08-21T12:46:18.871Z","avatar_url":"https://github.com/maxlaverse.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reproducing SNAT insertion failures\n\nThis small Go program aims at reproducing the condition for a race when inserting records in the conntrack table.\nMore explanations about the underlying issue can be found in [this blog post](https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02).\n\n## Introduction\n\nThe Linux Kernel has a known race condition when doing source network address translation (SNAT) that can lead to SYN packets being dropped. SNAT is performed by default on outgoing connections with Docker and Flannel using iptables masquerading rules.\nThe race can happen when multiple containers try to establish new connections to the same external address concurrently.\nIn some cases, two connections can be allocated the same port for the translation which ultimately results in one or more packets being dropped and at least one second connection delay.\n\nThis problem can be reproduced with a default Docker installation and this README try to explains how.\n\n## Reproducing the issue\n\nThe easiest way to reproduce the issue requires two servers with Docker installed and root access:\n* The *destination server* runs a Docker container that listens for incoming tcp connections. It's the endpoint for the connections we're trying to break. It should be fast enough to accept and immediately close the connection but doesn't require to be multi-core.\n* The *source server* is where we try to reproduce the issue. It runs 3 containers:\n   1. A container to observe the insertion failure in `conntrack` and manipule `iptables`.\n   2. Two containers that try to connect to the destination server and race.\n\nThe likelihood for observing the issue increases with the number of cores on the *source server*.\n\n### Prepare the destination server\nStart a container that listens for tcp connections:\n```shellsession\n$ sudo docker run -p8080:8080 -ti maxlaverse/snat-race-conn-test server\n```\n\nIf you try to connect from another server, the connection should work and immediately be closed:\n```shellsession\n$ nc -vv \u003cip-of-dst-server\u003e 8080\nConnection to \u003cip-of-dst-server\u003e 8080 port [tcp/http-alt] succeeded!\n```\n\n### Prepare the util container on the source server\n\nStart an Alpine container on the *source server*, install `conntrack` and `iptables`:\n```shellsession\n$ sudo docker run --privileged --net=host -ti alpine:latest\n\n$ apk add conntrack-tools iptables\nfetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz\nfetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz\n(1/9) Installing libmnl (1.0.4-r1)\n(2/9) Installing libnfnetlink (1.0.1-r2)\n(3/9) Installing libnetfilter_conntrack (1.0.8-r0)\n(4/9) Installing libnetfilter_cthelper (1.0.0-r1)\n(5/9) Installing libnetfilter_cttimeout (1.0.0-r1)\n(6/9) Installing libnetfilter_queue (1.0.5-r0)\n(7/9) Installing conntrack-tools (1.4.6-r0)\n(8/9) Installing libnftnl-libs (1.1.8-r0)\n(9/9) Installing iptables (1.8.6-r0)\nExecuting busybox-1.32.1-r6.trigger\nOK: 9 MiB in 23 packages\n```\n\nThis containers requires access to the host network interface and has to be privileged in order to interact with it. \nFrom this container, have a first look at the conntrack statistics:\n```shellsession\n$ conntrack -S\ncpu=0   \tfound=0 invalid=0 ignore=2 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0\ncpu=1   \tfound=0 invalid=0 ignore=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0\ncpu=2   \tfound=0 invalid=0 ignore=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0\ncpu=3   \tfound=0 invalid=0 ignore=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0\ncpu=4   \tfound=0 invalid=0 ignore=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0\ncpu=5   \tfound=0 invalid=0 ignore=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0\ncpu=6   \tfound=0 invalid=0 ignore=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0\ncpu=7   \tfound=0 invalid=0 ignore=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0\ncpu=8   \tfound=0 invalid=0 ignore=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0\ncpu=9   \tfound=0 invalid=0 ignore=6 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0\ncpu=10   \tfound=0 invalid=0 ignore=6 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0\n[...] \n```\n\nThis command was executed after a reboot. Therefore all `insert_failed` counters are at 0.\n\nFrom this container, you can also check that the Docker masquerading iptables rule is present:\n```shellsession\n$ iptables-save\n[...]\n-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE\n[...]\n```\n\n### Run the test containers on the source server\nStart two containers that continuously establish connections to the destination server:\n```shellsession\n$ sudo docker run -ti maxlaverse/snat-race-conn-test client --remote-addr \u003cip-of-dst-server\u003e:8080\n\n$ sudo docker run -ti maxlaverse/snat-race-conn-test client --remote-addr \u003cip-of-dst-server\u003e:8080\n```\n\nThe test program periodically prints a summary of the response time and the number of errors:\n```\n2021-05-16T15:42:32.108 Summary of the last 5 seconds:\n             Max response time:   3.9ms\n                 99 percentile:   3.8ms\n                 95 percentile:   3.2ms\n                        median:   2.6ms\n                  Request Rate:   156req/s\n\nRequests since start (error/total):     0/784\n```\n\nAfter a couple of seconds you should already see errors being logged in one or both containers:\n```\n2021-05-16T15:42:36.819 error after 500ms, err: \"dial tcp :0-\u003e10.18.10.137:8080: i/o timeout\"\n2021-05-16T15:42:37.108 Summary of the last 5 seconds:\n             Max response time: 500.1ms\n                 99 percentile:   3.7ms\n                 95 percentile:   3.2ms\n                        median:   2.5ms\n                  Request Rate:   159req/s\n\nRequests since start (error/total):     1/1580\n```\n\nAfter a couple of minutes, press Ctrl+C to stop the two test containers and note down the amount of errors:\n```shellsession\nContainer 1: Requests since start (error/total):  1439/49571\nContainer 2: Requests since start (error/total):   829/51927\n\n# Approximately 2.05% of the connection failed\n```\n\nHave another look at the conntrack statistics from the util container. You can see that some `insert_failed` counters increased:\n\n```shellsession\n$ conntrack -S\ncpu=0   \tfound=4601 invalid=2157 ignore=0 insert=0 insert_failed=74 drop=74 early_drop=0 error=0 search_restart=100\ncpu=1   \tfound=7545 invalid=3952 ignore=0 insert=0 insert_failed=175 drop=175 early_drop=0 error=0 search_restart=159\ncpu=2   \tfound=3810 invalid=1871 ignore=2 insert=0 insert_failed=56 drop=56 early_drop=0 error=0 search_restart=72\ncpu=3   \tfound=8138 invalid=4610 ignore=0 insert=0 insert_failed=166 drop=166 early_drop=0 error=0 search_restart=181\ncpu=4   \tfound=4139 invalid=2725 ignore=0 insert=0 insert_failed=82 drop=82 early_drop=0 error=0 search_restart=129\ncpu=5   \tfound=6283 invalid=4059 ignore=6 insert=0 insert_failed=129 drop=129 early_drop=0 error=0 search_restart=220\ncpu=6   \tfound=9915 invalid=5704 ignore=0 insert=0 insert_failed=196 drop=196 early_drop=0 error=0 search_restart=251\ncpu=7   \tfound=9709 invalid=4931 ignore=0 insert=0 insert_failed=215 drop=215 early_drop=0 error=0 search_restart=235\ncpu=8   \tfound=7441 invalid=4626 ignore=0 insert=0 insert_failed=162 drop=162 early_drop=0 error=0 search_restart=246\ncpu=9   \tfound=12412 invalid=6959 ignore=0 insert=0 insert_failed=249 drop=249 early_drop=0 error=0 search_restart=306\ncpu=10  \tfound=11154 invalid=6175 ignore=0 insert=0 insert_failed=231 drop=231 early_drop=0 error=0 search_restart=272\n[...]\n```\n\n### Trying out a mitigation\n\nIn the util container, save the iptables rules:\n```shellsession\n$ iptables-save \u003e dump\n```\n\nAppend `--random-fully` to the masquerading rule:\n```diff\n- -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE\n+ -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE --random-fully\n```\n\nReload the rules:\n```shellsession\n$ iptables-restore \u003c dump\n```\n\nNow restart the two test containers and stop them after the same amount of requests than the first run:\n```\nContainer 1: Requests since start (error/total):    45/50210\nContainer 2: Requests since start (error/total):    57/50160\n\n# Approximately 0.10% of the connection failed with --random-fully\n# This is a 95% error decrease in this particular case, and can vary a lot from one setup from the other.\n```\n\n## Kubernetes\n\nYou can run the same test on Kubernetes. Your test Pods need to be collocated on the same Node in order to observe insertion failures.\n\n## Building the project\n\nThe test program is written using Go module. You need at least Go 1.16 to compile it:\n```shellsession\n$ go build\n```\n\nYou can build you own Docker image with:\n```shellsession\n$ sudo docker build -t testimage .\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxlaverse%2Fsnat-race-conn-test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxlaverse%2Fsnat-race-conn-test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxlaverse%2Fsnat-race-conn-test/lists"}