{"id":16859009,"url":"https://github.com/ross/haproxy-mapper","last_synced_at":"2025-10-20T12:40:39.865Z","repository":{"id":48379034,"uuid":"388671290","full_name":"ross/haproxy-mapper","owner":"ross","description":"Tools for generating HAProxy Maps from various sources","archived":false,"fork":false,"pushed_at":"2021-08-02T19:05:37.000Z","size":136,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-29T16:11:29.525Z","etag":null,"topics":["haproxy","maxmind","spamhaus"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ross.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-23T04:00:24.000Z","updated_at":"2025-04-04T10:24:04.000Z","dependencies_parsed_at":"2022-09-06T05:32:11.697Z","dependency_job_id":null,"html_url":"https://github.com/ross/haproxy-mapper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ross/haproxy-mapper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ross%2Fhaproxy-mapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ross%2Fhaproxy-mapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ross%2Fhaproxy-mapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ross%2Fhaproxy-mapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ross","download_url":"https://codeload.github.com/ross/haproxy-mapper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ross%2Fhaproxy-mapper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262739457,"owners_count":23356729,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["haproxy","maxmind","spamhaus"],"created_at":"2024-10-13T14:15:54.509Z","updated_at":"2025-10-20T12:40:39.791Z","avatar_url":"https://github.com/ross.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## HAProxy Mapper\n\nTools for generating [HAProxy\nMaps](https://www.haproxy.com/blog/introduction-to-haproxy-maps/) from various\nsources of data including [MaxMind](https://maxmind.com/),\n[Spamhaus](https://www.spamhaus.org/), and various Cloud and CDN providers. The\nresulting maps can be useful for many purposes: observability, legal\ncompliance, defending against DDoS attacks.\n\n### Sources\n\n* Providers\n  * AWS\n  * Azure - Hacky currently to avoid requiring auth\n  * Cloudflare\n  * Fastly\n  * Google Cloud\n  * Oracle\n* MaxMind\n  * ASN/ISP\n  * City\n* Spamhaus\n\nIf you have a source of data you'd like to see included [open an\nissue](https://github.com/ross/haproxy-mapper/issues/new) or even better submit\na PR. If there's a publicly available JSON or TXT file with data for the source\nplease include info about it in the issue, doing so will make it much more\nlikely to be implemented.\n\n### Mapfile Format\n\nMap files have 1 value per line. The format is a range of IP addresses in [CIDR\nnotation](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#CIDR_notation)\nfollowed by a space and then an associated value for that range. The value\ndetails vary by map, but consists of UTF-8 characters and punctuation and may\ninclude spaces.  Generally haproxy-mapper uses `/` as a separator when a value\nhas multiple components. See the map file headers for more details.\n\n```\n1.0.0.0/24 OC/AU\n1.0.1.0/24 AS/CN\n1.0.2.0/23 AS/CN\n1.0.4.0/22 OC/AU\n1.0.8.0/21 AS/CN\n1.0.16.0/20 AS/JP\n1.0.32.0/19 AS/CN\n1.0.64.0/24 AS/JP/Hiroshima/Hiroshima\n1.0.65.0/25 AS/JP/Kanagawa/Sagamihara\n1.0.65.128/25 AS/JP/Hiroshima/Hiroshima\n```\n\n## Using map files with HAProxy\n\n\n### General map lookups\n\nThe following [HAProxy\nconfiguration](https://cbonte.github.io/haproxy-dconv/2.4/configuration.html)\nsnippet will map the `src` to a location when one is available.\n\nIf you will be using the results of the map lookup for multiple purposes it\noften makes sense to store the value in a variable. `txn` variables allow\naccess to the value across the full life-cycle of the request/connection once\ntheir set. You can use `req` or `res` if your specific needs limit the times at\nwhich you will need access to the value.\n\n```\nhttp-request set-var(txn.client_ip_location) src,map_ip(/etc/haproxy/maps/ip_to_location)\n```\n\n### Passing values to backend servers\n\nPass the previously mapped client IP location to the backend server as a\nrequest header\n\n```\nhttp-request set-header x-client-ip-location %[var(txn.client_ip_location)]\n```\n\n### Making decisions based on lookups\n\nOnce you've looked up a value in a map it can be used to make decisions in\nHAProxy via ACL's. For example the following would disallow requests that come\nfrom IPs that have the country equal to Canada.\n\n```\nhttp-request set-var(txn.client_ip_country) src,map_ip(/etc/haproxy/maps/ip_to_country)\nacl is_country_ca var(txn.client_ip_country) -i CA\nuse_backend 403_forbidden backend_banned_banhammer if is_country_ca\n```\n\n### Logging values\n\nIf you're going to be mapping IPs it often makes sense to log the results for\nobservability and debugging purposes. The details of HAProxy logging are beyond\nthe scope of this README, see\n[ross/haproxied](https://github.com/ross/haproxied) for more information, but\nthe following snippet should provide the specific bits required to get a lookup\nresult into your logs. We'll first need to get the lookup value stored in a\nvariable.\n\n```\nhttp-request set-var(txn.client_ip_location) src,map_ip(/etc/haproxy/maps/ip_to_location)\n```\n\nWe can then include the variable's value in our `log-format` to emit its value as part of our log line.\n\n```\nlog-format \"backend_name=%b ... client_ip_location=%{+Q,+E}[var(txn.client_ip_location)] ...\"\n```\n\nFor map values that do not contain spaces or special characters the quoting and\nescaping can be omitted, e.g ip_to_country which uses the 2-letter ISO codes.\n\n```\nlog-format \"backend_name=%b ... client_ip_country=%[var(txn.client_ip_country)] ...\"\n```\n\n### src vs X-Forwarded-For\n\nThe above example used\n[`src`](https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#src), but\nin some situations that maybe the IP address of a proxy, either on the internet\nor within your network, that will mask the true origin of the request. In many\nsuch cases the original client IP address will be passed to the server using\nthe [`x-forwarded-for` header](https://en.wikipedia.org/wiki/X-Forwarded-For).\n\nIt is important that you \"trust\" the source of the header before you utilize it\nas it is trivial for clients to add the header to confuse or obfuscate. This\ncan potentially be accomplished by having proxies under your control at the\nedge of your network strip or clear any such headers they see. They can then\nadd a new x-forwarded-for header with the information as they see it.\nSubsequent proxies in your system can then choose to leave x-forwarded-for\nheaders in place when `src` is on a trusted network and thus an internal server\nunder your control. See [ross/haproxied](https://github.com/ross/haproxied) for\nmore information and examples.\n\nAssuming you trust the `x-forwarded-for` header you can make use of it during\nmapping.\n\n```\nacl has_existing_x_forwarded_for req.hdr(x-forwarded-for) -m found\nhttp-request set-header x-real-ip %[req.hdr(x-forwarded-for,1)] if has_existing_x_forwarded_for\nhttp-request set-header x-real-ip %ci if !has_existing_x_forwarded_for\nhttp-request set-header x-real-ip-location req.hdr(x-real-ip),map_ip(/etc/haproxy/maps/ip_to_location)\n```\n\nIf you'll be using the values of `x-real-ip` and/or `x-real-ip-location`\nelsewhere they can be set into variables.\n\n```\nhttp-request set-var(txn.real_ip) req.hdr(x-real-ip)\nhttp-request set-var(txn.real_ip_location) req.hdr(x-real-ip-location)\n```\n\n## Generating Maps\n\nThe shortcut `-all` will create all available maps. It requires both MaxMind\nCity and either ISP or ASN databases. If the ASN database is used ISP data is\nnot included in the generated maps.\n\n```console\n$ haproxy-mapper -outdir /tmp/mapper -all -isp-db GeoIP2-ISP.mmdb -city-db GeoIP2-City.mmdb\n$ wc -l /tmp/mapper/*\n  649426 /tmp/mapper/ip_to_asn\n    4933 /tmp/mapper/ip_to_aws\n   15868 /tmp/mapper/ip_to_azure\n 3397950 /tmp/mapper/ip_to_city\n      22 /tmp/mapper/ip_to_cloudflare\n 3961657 /tmp/mapper/ip_to_continent\n 3959544 /tmp/mapper/ip_to_country\n      19 /tmp/mapper/ip_to_fastly\n     433 /tmp/mapper/ip_to_google_cloud\n  662153 /tmp/mapper/ip_to_isp\n 3961657 /tmp/mapper/ip_to_location\n     346 /tmp/mapper/ip_to_oracle\n   21621 /tmp/mapper/ip_to_provider\n    1132 /tmp/mapper/ip_to_spamhaus\n 3399066 /tmp/mapper/ip_to_subdivisions\n 20035827 total\n```\n\n### Full list of options\n\n```console\n$ haproxy-mapper\nUsage of ./haproxy-mapper:\n  -all\n        Include All data\n  -asn\n        Include asn map, requires -asn-db or -isp-db\n  -asn-db string\n        MaxMind ASN database file\n  -aws\n        Include AWS data\n  -azure\n        Include Azure data\n  -city\n        Include city map, requires -city-db\n  -city-db string\n        MaxMind City database file\n  -cloudflare\n        Include Cloudflare data\n  -continent\n        Include continent map, requires -city-db\n  -country\n        Include country map, requires -city-db\n  -fastly\n        Include Fastly data\n  -google-cloud\n        Include Google Cloud data\n  -ipv6\n        Process IPv6 data, -ipv6=false to disable (default true)\n  -isp\n        Include isp map, requires -isp-db\n  -isp-db string\n        MaxMind ISP database file\n  -location\n        Include location map, requires -city-db\n  -oracle\n        Include Oracle data\n  -outdir string\n        Output directory\n  -provider\n        Include providers map\n  -spamhaus\n        Include spamhaus data\n  -subdivisions\n        Include subdivisions map, requires -city-db\n```\n\n## Example Script/Process\n\n[examples/generate.sh](examples/generate.sh) is a starting point on which to\nbuild a process for downloading MaxMind databases and building maps. It can be\nrun with the following. It uses the `-all` option, though as written relies on\nthe free databases and thus does not include the ISP map.\n\n```\n$ go build \u0026\u0026 PATH=$PATH:. ./examples/generate.sh\nx GeoLite2-City_20210727/\nx GeoLite2-City_20210727/README.txt\nx GeoLite2-City_20210727/COPYRIGHT.txt\nx GeoLite2-City_20210727/GeoLite2-City.mmdb\nx GeoLite2-City_20210727/LICENSE.txt\nx GeoLite2-ASN_20210727/\nx GeoLite2-ASN_20210727/COPYRIGHT.txt\nx GeoLite2-ASN_20210727/GeoLite2-ASN.mmdb\nx GeoLite2-ASN_20210727/LICENSE.txt\nMaps in out\n  555291 out/ip_to_asn\n    4936 out/ip_to_aws\n   15868 out/ip_to_azure\n 3451689 out/ip_to_city\n      22 out/ip_to_cloudflare\n 6117600 out/ip_to_continent\n 6115499 out/ip_to_country\n      19 out/ip_to_fastly\n     433 out/ip_to_google_cloud\n 6117600 out/ip_to_location\n     346 out/ip_to_oracle\n   21624 out/ip_to_provider\n    1132 out/ip_to_spamhaus\n 3766966 out/ip_to_subdivisions\n 26169025 total\n```\n\n## Related Links\n\n* https://www.haproxy.com/blog/introduction-to-haproxy-maps/\n* https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/syntax/\n* https://cbonte.github.io/haproxy-dconv/2.4/configuration.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fross%2Fhaproxy-mapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fross%2Fhaproxy-mapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fross%2Fhaproxy-mapper/lists"}