{"id":45435974,"url":"https://github.com/tn3w/ip2x","last_synced_at":"2026-06-06T06:00:44.994Z","repository":{"id":339786125,"uuid":"1163360475","full_name":"tn3w/IP2X","owner":"tn3w","description":"Fast IP geolocation and proxy detection with compressed binary databases","archived":false,"fork":false,"pushed_at":"2026-05-31T20:53:46.000Z","size":71,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-05-31T22:21:27.271Z","etag":null,"topics":["asn-lookup","binary-database","ip-geolocation","proxy-detection","python","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tn3w.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-21T14:11:04.000Z","updated_at":"2026-05-31T20:47:54.000Z","dependencies_parsed_at":"2026-06-06T06:00:42.211Z","dependency_job_id":null,"html_url":"https://github.com/tn3w/IP2X","commit_stats":null,"previous_names":["tn3w/ip2x"],"tags_count":49,"template":false,"template_full_name":null,"purl":"pkg:github/tn3w/IP2X","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tn3w%2FIP2X","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tn3w%2FIP2X/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tn3w%2FIP2X/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tn3w%2FIP2X/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tn3w","download_url":"https://codeload.github.com/tn3w/IP2X/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tn3w%2FIP2X/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33971107,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-06T02:00:07.033Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asn-lookup","binary-database","ip-geolocation","proxy-detection","python","rust"],"created_at":"2026-02-22T03:00:47.347Z","updated_at":"2026-06-06T06:00:44.957Z","avatar_url":"https://github.com/tn3w.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# IP2X\n\n[![Build](https://img.shields.io/github/actions/workflow/status/tn3w/IP2X/build.yml?label=build)](https://github.com/tn3w/IP2X/actions)\n[![Release](https://img.shields.io/github/v/release/tn3w/IP2X?label=release)](https://github.com/tn3w/IP2X/releases/latest)\n[![Updated](https://img.shields.io/github/release-date/tn3w/IP2X?label=updated)](https://github.com/tn3w/IP2X/releases/latest)\n[![Artifacts](https://img.shields.io/badge/artifacts-11-blue)](#artifacts)\n[![Sources](https://img.shields.io/badge/sources-IP2Location_LITE_%2B_GeoLite2_%2B_RIR-informational)](#attribution)\n[![License](https://img.shields.io/badge/license-Apache_2.0-lightgrey)](LICENSE)\n\n[![geo.bin](https://img.shields.io/badge/geo.bin-42MB-blue)](https://github.com/tn3w/IP2X/releases/latest/download/geo.bin)\n[![proxy.bin](https://img.shields.io/badge/proxy.bin-12MB-blue)](https://github.com/tn3w/IP2X/releases/latest/download/proxy.bin)\n[![geofeed.bin](https://img.shields.io/badge/geofeed.bin-11MB-blue)](https://github.com/tn3w/IP2X/releases/latest/download/geofeed.bin)\n[![proxy_pub.netset](https://img.shields.io/badge/proxy__pub.netset-31MB-blue)](https://github.com/tn3w/IP2X/releases/latest/download/proxy_pub.netset)\n[![usage.buckets](https://img.shields.io/badge/usage.buckets-27MB-blue)](https://github.com/tn3w/IP2X/releases/latest/download/usage.buckets)\n[![threat.buckets](https://img.shields.io/badge/threat.buckets-0.5MB-blue)](https://github.com/tn3w/IP2X/releases/latest/download/threat.buckets)\n[![isp.tsv](https://img.shields.io/badge/isp.tsv-34MB-blue)](https://github.com/tn3w/IP2X/releases/latest/download/isp.tsv)\n[![domain.tsv](https://img.shields.io/badge/domain.tsv-33MB-blue)](https://github.com/tn3w/IP2X/releases/latest/download/domain.tsv)\n[![last_seen.tsv](https://img.shields.io/badge/last__seen.tsv-38MB-blue)](https://github.com/tn3w/IP2X/releases/latest/download/last_seen.tsv)\n[![provider.tsv](https://img.shields.io/badge/provider.tsv-0.3MB-blue)](https://github.com/tn3w/IP2X/releases/latest/download/provider.tsv)\n[![fraud_score.tsv](https://img.shields.io/badge/fraud__score.tsv-37MB-blue)](https://github.com/tn3w/IP2X/releases/latest/download/fraud_score.tsv)\n\nPublic IP intel repacked for fast offline use. Three crates: mmap binary\nDBs (`geo.bin`, `proxy.bin`, `geofeed.bin`) and plain-text proxy views\n(≤ 38 MB each). Sources: IP2Location LITE, MaxMind GeoLite2, RIR geofeeds.\n\n```bash\nwget https://github.com/tn3w/IP2X/releases/latest/download/geo.bin\nwget https://github.com/tn3w/IP2X/releases/latest/download/proxy.bin\nwget https://github.com/tn3w/IP2X/releases/latest/download/geofeed.bin\nwget https://github.com/tn3w/IP2X/releases/latest/download/proxy_pub.netset\nwget https://github.com/tn3w/IP2X/releases/latest/download/usage.buckets\nwget https://github.com/tn3w/IP2X/releases/latest/download/threat.buckets\nwget https://github.com/tn3w/IP2X/releases/latest/download/isp.tsv\nwget https://github.com/tn3w/IP2X/releases/latest/download/domain.tsv\nwget https://github.com/tn3w/IP2X/releases/latest/download/last_seen.tsv\nwget https://github.com/tn3w/IP2X/releases/latest/download/provider.tsv\nwget https://github.com/tn3w/IP2X/releases/latest/download/fraud_score.tsv\n```\n\nUpdated daily via GitHub Actions.\n\n## Artifacts\n\n| file | role | size |\n| ---- | ---- | ---: |\n| `geo.bin`            | mmap DB, IP → (lat, lon) at 0.001° | ~42 MB |\n| `proxy.bin`          | mmap DB, IP → (isp, domain)        | ~12 MB |\n| `geofeed.bin`        | mmap DB, IP → (country, region, city, postal, feed) | ~11 MB |\n| `proxy_pub.netset`   | CIDR netset, public proxies (proxy_type == PUB) | ~31 MB |\n| `usage.buckets`      | IP → usage  (bucketed per value)    | ~27 MB |\n| `threat.buckets`     | IP → threat (bucketed per value)    | ~0.5 MB |\n| `isp.tsv`            | IP → ISP            (dict + ranges) | ~34 MB |\n| `domain.tsv`         | IP → domain         (dict + ranges) | ~33 MB |\n| `last_seen.tsv`      | IP → last-seen days (dict + ranges) | ~38 MB |\n| `provider.tsv`       | IP → VPN provider   (dict + ranges) | ~0.3 MB |\n| `fraud_score.tsv`    | IP → fraud score    (dict + ranges) | ~37 MB |\n\n# geo.bin\n\nBuilt by [`geox/`](geox/) from IP2Location DB11 LITE (preferred) +\nMaxMind GeoLite2-City (fallback). Coordinates quantised to 0.001°\n(~111 m, village-scale). Self-describing little-endian, magic `GEO1`.\n\n## Layout\n\n24 B header. IPv4 stored as `(base u32) + (delta u24)` blocks of ≤ 256\nrows; IPv6 keyed on the upper 64 bits. Bit-packed point indices into a\ndeduped `(lat, lon)` table of i24/1000.\n\n| offset | size | field |\n| -----: | ---- | ----- |\n| 0  | 4    | magic `GEO1` |\n| 4  | u8   | version (1) |\n| 5  | u8   | minor (3) |\n| 6  | u8   | idx_bits |\n| 7  | u8   | reserved |\n| 8  | u32  | point_count |\n| 12 | u32  | v4_row_count |\n| 16 | u32  | v6_row_count |\n| 20 | u32  | v4_block_count |\n\nThen: points (`6 B × point_count`), v4 bases (`4 B × blocks`), v4 offsets\n(`4 B × (blocks+1)`), v4 deltas (`3 B × rows`), v4 packed idx,\nv6 keys (`8 B × rows`), v6 packed idx.\n\nLookup v4: bisect `v4_bases`, bisect deltas inside the matched block,\nread packed idx, decode point. Lookup v6: bisect upper-64 keys, read\npacked idx, decode point. ~0.2 MB resident at open; pages fault on demand.\n\n## Build\n\n```bash\ncd geox\ncargo build --release\n\n./target/release/geox build \\\n    --ip2l IP2LOCATION-LITE-DB11.IPV6.BIN \\\n    --mmdb GeoLite2-City.mmdb \\\n    --out  geo.bin\n\n./target/release/geox lookup --db geo.bin 8.8.8.8\n# 37.386, -122.084\n```\n\n## Python lookup ([`geo_lookup.py`](geo_lookup.py))\n\nmmap + numpy `searchsorted` on v4 bases / v6 upper-64 keys; manual\nbit-packed idx + i24 decode. No preload, near-instant startup.\n\n```bash\npython3 geo_lookup.py 8.8.8.8 2001:4860:4860::8888\n# 8.8.8.8                  37.386, -122.084\n# 2001:4860:4860::8888     37.386, -122.084\n```\n\n`--db PATH` to point at a non-default `geo.bin`.\n\n# proxy.bin\n\nBuilt by [`proxyx/`](proxyx/) from IP2Location IP2PROXY-LITE-PX12.\nCompact mmap DB, IP → (isp, domain). Magic `PRX2`, little-endian, ~12 MB\nfor the full PX12 dataset (3.88M v4 rows + 7.8k v6 rows after\nadjacent-equal merge).\n\n## Layout\n\n36 B header. Strings interned once into a single offset/blob table;\n(isp_idx, dom_idx) pairs interned into a pair table, freq-sorted so hot\npairs get tiny indices. IPv4 stored as fixed-size blocks of 256 rows\nwith per-block variable bit-width deltas and pair-index packing; IPv6\nkeyed on the upper 64 bits.\n\n| offset | size | field |\n| -----: | ---- | ----- |\n| 0  | 4   | magic `PRX2` |\n| 4  | u8  | version (2) |\n| 5  | u8  | block_shift (8 → 256 rows) |\n| 6  | u8  | v6_bits |\n| 7  | u8  | reserved |\n| 8  | u32 | pair_count |\n| 12 | u32 | str_count |\n| 16 | u32 | v4_row_count |\n| 20 | u32 | v6_row_count |\n| 24 | u32 | v4_block_count |\n| 28 | u32 | v4_delta_blob_len |\n| 32 | u32 | v4_idx_blob_len |\n\nThen: pairs (`6 B × n_pairs`, u24 isp_idx + u24 dom_idx), str offsets\n(`4 B × (n_strs+1)`), str blob, v4 bases (`4 B × blocks`), per-block\n`dbits` / `ibits` (`1 B × blocks` each), v4 delta byte-offsets and\nidx byte-offsets (`4 B × (blocks+1)` each), v4 delta blob + 8 B pad,\nv4 idx blob + 8 B pad, v6 keys (`8 B × rows`), v6 packed idx + 8 B pad.\n\nAvg per-block widths on full PX12: ~14 delta-bits, ~8 idx-bits.\n\nLookup v4: bisect `bases4`, bisect deltas in the matched block at that\nblock's `dbits`, read packed pair-idx at that block's `ibits`, resolve\npair → (isp, domain). Lookup v6: bisect upper-64 keys, read packed idx,\nresolve pair. Native lookup ~170 ns v4 / ~80 ns v6; load ~10 µs;\nresident struct 208 B (mmap shared, paged on demand).\n\n## Build\n\n```bash\ncd proxyx\ncargo build --release\n\n./target/release/proxyx build-db \\\n    --px12 IP2PROXY-LITE-PX12.BIN \\\n    --out  proxy.bin\n\n./target/release/proxyx lookup --db proxy.bin 1.0.19.98\n# isp     I2TS Inc.\n# domain  mediaindex.co.jp\n```\n\n## Python lookup ([`proxy_db_lookup.py`](proxy_db_lookup.py))\n\nmmap + numpy `searchsorted` on bases4 / v6 upper-64 keys; manual\nbit-packed delta + idx decode against per-block widths. No preload,\nnear-instant startup.\n\n```bash\npython3 proxy_db_lookup.py 1.0.19.98 2001:dead::1\n# 1.0.19.98     isp=I2TS Inc.            domain=mediaindex.co.jp\n# 2001:dead::1  isp=FDCservers.net LLC   domain=fdcservers.net\n```\n\n`--db PATH` to point at a non-default `proxy.bin`.\n\n# geofeed.bin\n\nBuilt by [`geofeedx/`](geofeedx/) from operator-published geolocation.\nThe builder downloads the RIR bulk WHOIS dumps (RIPE, APNIC, AFRINIC),\nextracts every `geofeed:` / `remarks: Geofeed` reference, fetches each\nreferenced [RFC 8805](https://www.rfc-editor.org/rfc/rfc8805) feed\nconcurrently, and merges the LACNIC consolidated feed. Self-describing\nlittle-endian, magic `GFD3`, IPv4 + IPv6.\n\nFeed rows are accepted only when contained in the authority range of the\nRIR object that referenced them. Each row contributes\n`(country, region, city, postal, feed, rir)`; `feed` is the source URL.\n\n## Layout\n\n28 B header. `(country, region, city, postal, feed, rir)` tuples are\ninterned into a freq-sorted record table (hot records get small ids), and\nevery string is interned once into an offset/blob table. IPv4 and IPv6\nranges are each flattened into a sorted breakpoint array (`start → record\nid`); adjacent-equal ids are merged. Id and field-index widths are the\nminimum bytes the cardinalities require (typically 2 B each).\n\n| offset | size | field |\n| -----: | ---- | ----- |\n| 0  | 4   | magic `GFD3` |\n| 4  | u8  | version (3) |\n| 5  | u8  | id_width |\n| 6  | u8  | field_count (6) |\n| 7  | u8  | field_width |\n| 8  | u32 | v4_break_count |\n| 12 | u32 | v6_break_count |\n| 16 | u32 | record_count |\n| 20 | u32 | string_count |\n| 24 | u32 | blob_len |\n\nThen: v4 starts (`4 B × v4_breaks`), v4 ids (`id_width × v4_breaks`),\nv6 starts (`16 B × v6_breaks`), v6 ids (`id_width × v6_breaks`),\nrecords (`field_count × field_width × records`), string offsets\n(`4 B × (strings+1)`), string blob.\n\nLookup: bisect the matching family's starts, read the packed record id,\nresolve the tuple. Native load ~6 µs (mmap, ~0 resident); ~120 ns/lookup\nover ~1.2 M v4 breakpoints.\n\n## Build\n\n```bash\ncd geofeedx\ncargo build --release\n\n./target/release/geofeedx fetch --out geofeeds_data.csv\n./target/release/geofeedx build --data geofeeds_data.csv --out geofeed.bin\n\n./target/release/geofeedx lookup --db geofeed.bin 213.21.192.5\n# country  LV\n# region   LV-RIX\n# city     Riga\n# ...\n```\n\n`fetch` caches the RIR bulk dumps under `.cache/rir-bulk` and re-downloads\nonly what is missing. `geofeeds_data.csv` is the intermediate\n`cidr,country,region,city,postal,feed,rir` join, regenerated on each fetch.\n\n## Python lookup ([`geofeed_lookup.py`](geofeed_lookup.py))\n\nmmap + `bisect` on the v4 / v6 start arrays; variable-width record and\nfield decode. No preload, near-instant startup. v4 + v6 in one call.\n\n```bash\npython3 geofeed_lookup.py 213.21.192.5 2001:ad0::1\n```\n\n`--db PATH` to point at a non-default `geofeed.bin`.\n\n# proxyx outputs\n\nBuilt by [`proxyx/`](proxyx/) from IP2Location IP2PROXY-LITE-PX12.\nAll files plain UTF-8, `#`-prefixed metadata header, ≤ 38 MB each\n(no compression, no splitting). Empty source fields dropped; adjacent\nranges with identical value merged.\n\nThree shapes used across the files:\n\n### Netset (`proxy_pub.netset`)\n\nStandard CIDR list, one network per line, single IPs as bare addresses.\n`#`-prefixed metadata header. Drop-in for `ipset hash:net`,\n`iptables`/`nftables`, `ufw`, pfSense and similar.\n\n```bash\nipset create proxy_pub hash:net family inet\nawk '!/^#/ \u0026\u0026 /\\./' proxy_pub.netset | xargs -n1 ipset add proxy_pub\n```\n\n### Bucketed form (`usage.buckets`, `threat.buckets`)\n\n```\n[VALUE]\n\u003cstart_ip\u003e[+\u003cspan\u003e]\n\u003cstart_ip\u003e[+\u003cspan\u003e]\n[NEXT_VALUE]\n...\n```\n\nFor low-cardinality categorical fields. IP → value = scan sections,\nbisect ranges. The string is written once per category, not per range.\n\n### Dict + ranges form (`*.tsv`)\n\n```\n#dict\n\u003cidx\u003e\\t\u003cvalue\u003e\n\u003cidx\u003e\\t\u003cvalue\u003e\n#data\n\u003cstart_ip\u003e[+\u003cspan\u003e]\\t\u003cidx\u003e\n```\n\n`#dict` is frequency-sorted (smaller idx = more common, so popular\nvalues cost 1-2 chars per row). `#data` is v4 block then v6, ascending.\nLookup: load the dict into a `Vec\u003cString\u003e`, bisect `#data` by `start_ip`,\nindex into the dict.\n\n## Field source\n\nPX12 columns kept by `proxyx` (others ignored):\n\n| file | PX12 column |\n| ---- | ----------- |\n| `proxy_pub.netset` | `proxy_type` filtered to `PUB` |\n| `usage.buckets`    | `usage_type` |\n| `threat.buckets`   | `threat` |\n| `isp.tsv`          | `isp` |\n| `domain.tsv`       | `domain` |\n| `last_seen.tsv`    | `last_seen` (days) |\n| `provider.tsv`     | `provider` |\n| `fraud_score.tsv`  | `fraud_score` (0-99) |\n\nCountry/region/city/ASN/AS-name are intentionally omitted — `geo.bin`\nalready covers location, ASN lives elsewhere.\n\n## Build\n\n```bash\ncd proxyx\ncargo build --release\n\n./target/release/proxyx build \\\n    --px12 IP2PROXY-LITE-PX12.BIN \\\n    --out  out/\n\nls -lh out/\n```\n\n## Python lookup ([`proxy_lookup.py`](proxy_lookup.py))\n\nParses all 8 outputs once into sorted (start, end, val) arrays; bisects\nper file on query. v4 + v6 in one call. Load ~8 s for the full bundle,\nlookup O(log n) per file thereafter.\n\n```bash\npython3 proxy_lookup.py 1.0.19.98\n# proxy_pub    True\n# isp          I2TS Inc.\n# domain       mediaindex.co.jp\n# last_seen    30\n# fraud_score  80\n# usage        DCH\n# ...\n```\n\n`--dir PATH` to point at a directory other than `.`.\n\n# Pipeline\n\n```mermaid\nflowchart LR\n    D1[IP2Location DB11 LITE] --\u003e G[geox/]\n    D2[GeoLite2-City] --\u003e G\n    G --\u003e GB[geo.bin]\n    D3[IP2Location PX12 LITE] --\u003e P[proxyx/]\n    P --\u003e PB[proxy.bin]\n    P --\u003e R[proxy_pub.netset]\n    P --\u003e U[usage.buckets]\n    P --\u003e T[threat.buckets]\n    P --\u003e TSV[isp / domain / last_seen / provider / fraud_score .tsv]\n    D4[RIR bulk WHOIS] --\u003e F[geofeedx/]\n    D5[RFC 8805 feeds + LACNIC] --\u003e F\n    F --\u003e FB[geofeed.bin]\n```\n\n# Automated updates\n\n[`.github/workflows/build.yml`](.github/workflows/build.yml):\n\n1. Loops over IP2Location LITE downloads (`DB11LITEBINIPV6`,\n   `PX12LITEBIN`) using `IP2LOCATION_TOKEN`.\n2. Pulls `GeoLite2-City.mmdb` from a public mirror.\n3. Builds `geo.bin` with `geox`, plus `proxy.bin` and the eight\n   plain-text views with `proxyx`.\n4. Runs `geofeedx fetch` (RIR bulk + RFC 8805 feeds) then `geofeedx\n   build` to produce `geofeed.bin`.\n5. Publishes a timestamped release with all eleven assets; prunes to the\n   latest 5.\n\n# Attribution\n\nGeo data: [IP2Location LITE](https://lite.ip2location.com) DB11 +\n[MaxMind GeoLite2](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data).\nProxy data: IP2Location LITE PX12.\nGeofeed data: RIR bulk WHOIS (RIPE, APNIC, AFRINIC, LACNIC) +\noperator-published [RFC 8805](https://www.rfc-editor.org/rfc/rfc8805) feeds.\n\n# License\n\n[Apache-2.0](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftn3w%2Fip2x","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftn3w%2Fip2x","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftn3w%2Fip2x/lists"}