{"id":13620106,"url":"https://github.com/ixty/wnm","last_synced_at":"2025-04-14T19:31:02.635Z","repository":{"id":49698338,"uuid":"119668160","full_name":"ixty/wnm","owner":"ixty","description":"World Internet Map, internet visualization, networks, AS, BGP, ...","archived":false,"fork":false,"pushed_at":"2022-12-08T00:43:41.000Z","size":2593,"stargazers_count":15,"open_issues_count":6,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-08T06:40:27.929Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://wnm.ixty.net","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ixty.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-31T09:52:21.000Z","updated_at":"2024-07-21T12:17:39.000Z","dependencies_parsed_at":"2023-01-25T01:31:32.009Z","dependency_job_id":null,"html_url":"https://github.com/ixty/wnm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ixty%2Fwnm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ixty%2Fwnm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ixty%2Fwnm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ixty%2Fwnm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ixty","download_url":"https://codeload.github.com/ixty/wnm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248945777,"owners_count":21187384,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T21:00:52.297Z","updated_at":"2025-04-14T19:31:02.621Z","avatar_url":"https://github.com/ixty.png","language":"HTML","funding_links":[],"categories":["Geography and maps"],"sub_categories":["Looking for software, projects, courses or inspiration?"],"readme":"# World Net Map\n2017-2018 - Martin Balch - \u003ca href=\"mailto:martin.balch+wnm@gmail.com\"\u003econtact\u003c/a\u003e\n\n## intro\n\nWorld Net Map (WNM) is a project to visualize geographically the internet infrastructure, networks and connectivity.\nIt uses data from multiple sources to display information on networks, countries, facilities, exchanges, submarine cables, ... and represent everything in a single interactive map.\nThis repository hosts the code used to download, agregate, consolidate \u0026 format the data (`wnm_*`) in addition to the code of a web server that displays the data (`pwui_*`).\n\nSee it in action here: [http://wnm.ixty.net/](http://wnm.ixty.net/)\n\n![World Net Map Screenshot](https://github.com/ixty/wnm/raw/master/screenshot.png \"World Net Map Screenshot\")\n\n## installation\n\n```shell\n# pre-requisites\n$ sudo apt-get install git virtualenv unzip\n\n# install with data (recommended)\n$ git clone --recursive https://github.com/ixty/wnm\n\n# or install without data (you need to regenerate databases)\n$ # git clone https://github.com/ixty/wnm\n\n# python requirements\n$ cd wnm\n$ virtualenv venv\n$ source venv/bin/activate\n$ pip install -r requirements.txt\n```\n\n## run server\n\nIn debug mode (default), the default bound address is `127.0.0.1:5000`, flask is used as a webserver, templates are automatically reloaded and there is no cache on pages.\n\n```shell\n$ source venv/bin/activate\n$ ./pwui/pwui.py\n```\n\nThe default bound address is `0.0.0.0:80` in release mode, which may requires root acess.\n\n```shell\n$ source venv/bin/activate\n$ ./pwui/pwui.py release\n```\n\n## regenerate databases\n\nIf you didnt clone the repository recursively, you need to regenerate the database.\n\n### automated update\nIf you want to generate or update the database used by WNM, you can use the following command:\n```shell\n$ source venv/bin/activate\n$ ./update.py\n```\nThis operation can take a long time based on your bandwith \u0026 cpu.\n\n### manual sub-database updates\nYou can also update specific databases by using the following scripts:\n```shell\n$ ./wnm/wnm_asnames.py\n$ ./wnm/wnm_bgp.py\n$ ./wnm/wnm_geo.py\n$ ./wnm/wnm_nics.py\n$ ./wnm/wnm_scd.py\n$ ./wnm/wnm_world.py\n```\nThey all have the same usage: `./script \u003cdownload | rebuild | update\u003e`\n- `download` will download the raw source data\n- `rebuild` will update our local database (./data/*) with the previously downloaded source data.\n- `update` is equivalent to running `download` and `rebuild`\n\n### geo-location\nTo get accurate positioning for facilities, you will need to get an OpenCageData API key from [OpenCageData.com](https://geocoder.opencagedata.com/api)\n```shell\n# for a first build\n$ OCD_KEY=01234567890123456789012345678901 ./update.py\n\n# or just regenerate WNM data\n$ OCD_KEY=01234567890123456789012345678901 ./wnm/wnm_merge.py\n```\n\n### update log\n\u003cdetails\u003e\n    \u003csummary\u003eExample output of the full updating process\u003c/summary\u003e\n\n```shell\n========================================\n\u003e updating AS names database\n========================================\n\u003e ftp://ftp.radb.net/radb/dbase/radb.db.gz\n\u003e downloaded ./data-raw/as-radb.db.gz in 3.2 secs\n\u003e ftp://ftp.arin.net/pub/rr/arin.db\n\u003e downloaded ./data-raw/as-arin.db in 5.4 secs\n\u003e ftp://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz\n\u003e downloaded ./data-raw/as-ripe.db.gz in 2.2 secs\n\u003e ftp://ftp.afrinic.net/pub/dbase/afrinic.db.gz\n\u003e downloaded ./data-raw/as-afrinic.db.gz in 4.4 secs\n\u003e ftp://rr.level3.net/pub/rr/level3.db.gz\n\u003e downloaded ./data-raw/as-level3.db.gz in 3.1 secs\n\u003e https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz\n\u003e downloaded ./data-raw/as-apnic.db.gz in 6.6 secs\n========================================\n\u003e done.\n\u003e parsing as-radb.db.gz\n\u003e parsing as-arin.db.gz\n\u003e parsing as-ripe.db.gz\n\u003e parsing as-afrinic.db.gz\n\u003e parsing as-level3.db.gz\n\u003e parsing as-apnic.db.gz\n\u003e saving to ./data/asnames.json.gz (52112 items)\n\n========================================\n\u003e updating BGP database\n========================================\n\u003e http://routeviews.org/route-views.linx/bgpdata/2018.01/RIBS/rib.20180129.0000.bz2\n\u003e downloaded ./data-raw/bgp-linx.bz2 in 9.3 secs\n\u003e downloaded [linx] bgp database.\n\u003e http://routeviews.org/route-views.eqix/bgpdata/2018.01/RIBS/rib.20180129.0000.bz2\n\u003e downloaded ./data-raw/bgp-eqix.bz2 in 4.7 secs\n\u003e downloaded [eqix] bgp database.\n\u003e http://routeviews.org/route-views.jinx/bgpdata/2018.01/RIBS/rib.20180129.0000.bz2\n\u003e downloaded ./data-raw/bgp-jinx.bz2 in 2.3 secs\n\u003e downloaded [jinx] bgp database.\n\u003e http://routeviews.org/route-views.saopaulo/bgpdata/2018.01/RIBS/rib.20180129.0000.bz2\n\u003e downloaded ./data-raw/bgp-saopaulo.bz2 in 4.1 secs\n\u003e downloaded [saopaulo] bgp database.\n\u003e http://routeviews.org/route-views.sg/bgpdata/2018.01/RIBS/rib.20180129.0000.bz2\n\u003e downloaded ./data-raw/bgp-sg.bz2 in 3.3 secs\n\u003e downloaded [sg] bgp database.\n\u003e done 5/5\n\u003e loading ./data-raw/bgp-linx.bz2\n\u003e processing ./data-raw/bgp-linx.bz2\n100%|████████████████████████████████████████████| 16948716/16948716 [32:26\u003c00:00, 8707.87it/s]\n\u003e loading ./data-raw/bgp-eqix.bz2\n\u003e processing ./data-raw/bgp-eqix.bz2\n100%|██████████████████████████████████████████████| 8601056/8601056 [18:27\u003c00:00, 7766.23it/s]\n\u003e loading ./data-raw/bgp-jinx.bz2\n\u003e processing ./data-raw/bgp-jinx.bz2\n100%|██████████████████████████████████████████████| 1157353/1157353 [02:29\u003c00:00, 7748.20it/s]\n\u003e loading ./data-raw/bgp-saopaulo.bz2\n\u003e processing ./data-raw/bgp-saopaulo.bz2\n100%|██████████████████████████████████████████████| 9463089/9463089 [24:37\u003c00:00, 6404.86it/s]\n\u003e loading ./data-raw/bgp-sg.bz2\n\u003e processing ./data-raw/bgp-sg.bz2\n100%|██████████████████████████████████████████████| 4093937/4093937 [09:39\u003c00:00, 7061.49it/s]\n\u003e saving to ./data/bgp.json.gz (60826 items)\n\n========================================\n\u003e updating world cities database\n========================================\n\u003e http://download.maxmind.com/download/worldcities/worldcitiespop.txt.gz\n\u003e downloaded ./data-raw/worldcitiespop.txt.gz in 3.4 secs\n\u003e done\n\u003e processing ./data-raw/worldcitiespop.txt.gz\n\u003e saving to ./data/cities.json.gz (234 items)\n\n========================================\n\u003e updating nics database\n========================================\n\u003e ftp://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-latest\n\u003e downloaded ./data-raw/nic-lacnic.txt in 6.3 secs\n\u003e ftp://ftp.apnic.net/pub/stats/apnic/delegated-apnic-latest\n\u003e downloaded ./data-raw/nic-apnic.txt in 10.1 secs\n\u003e ftp://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-latest\n\u003e downloaded ./data-raw/nic-afrinic.txt in 3.1 secs\n\u003e ftp://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest\n\u003e downloaded ./data-raw/nic-arin.txt in 16.6 secs\n\u003e ftp://ftp.ripe.net/ripe/stats/delegated-ripencc-latest\n\u003e downloaded ./data-raw/nic-ripe.txt in 1.5 secs\n========================================\n\u003e consolidating database..\n[ lacnic] date: 20180126 records 29128\n[  apnic] date: 20180129 records 54420\n[afrinic] date: 20180129 records 5652\n[   arin] date: 20180129 records 137492\n[ripencc] date: 20180128 records 114677\n\u003e total rirs info [asn: 85628, ipv4: 183252, ipv6: 81081]\n\u003e saving to ./data/nics.json.gz (4 items)\n\n========================================\n\u003e updating submarine cable database\n========================================\n\u003e https://github.com/telegeography/www.submarinecablemap.com/archive/master.zip\n\u003e error downloading ./data-raw/cables.zip\n\u003e https://github.com/telegeography/www.submarinecablemap.com/archive/master.zip\n\u003e downloaded ./data-raw/cables.zip in 1.3 secs\n\u003e unknown landing 9595 for country ID\n\u003e unknown landing 9595 for cable 1895\n\u003e saving to ./data/scdb.json.gz (3 items)\n\n========================================\n\u003e updating country database\n========================================\n\u003e http://download.geonames.org/export/dump/countryInfo.txt\n\u003e downloaded ./data-raw/worldinfo.txt in 0.1 secs\n\u003e saving to ./data/worldinfo.json.gz (252 items)\n\n========================================\n\u003e updating PeeringDB\n========================================\nOperations to perform:\n  Synchronize unmigrated apps: django_peeringdb\n  Apply all migrations: (none)\nSynchronizing apps without migrations:\n  Creating tables...\n    Running deferred SQL...\n  Installing custom SQL...\nRunning migrations:\n  No migrations to apply.\norg last update 1517240845 0 changed\ndata to be processed 0\nfac last update 1517069653 0 changed\ndata to be processed 0\nnet last update 1517244409 3 changed\ndata to be processed 3\nix last update 1517005124 0 changed\ndata to be processed 0\nixfac last update 1516870316 0 changed\ndata to be processed 0\nixlan last update 1516870415 0 changed\ndata to be processed 0\nixpfx last update 1516707274 0 changed\ndata to be processed 0\npoc last update 1517066523 5 changed\ndata to be processed 5\nnetfac last update 1517235749 0 changed\ndata to be processed 0\nnetixlan last update 1517242057 3 changed\ndata to be processed 3\n\n========================================\n\u003e Merging databases\n========================================\n\u003e loading data from data/nics.json.gz\n\u003e loading data from data/bgp.json.gz\n\u003e loading data from data/scdb.json.gz\n\u003e loading data from data/asnames.json.gz\n\u003e loading data from data/cities.json.gz\n\u003e loading data from data/worldinfo.json.gz\n\u003e loading peering db ..\n\u003e merging network info ..\n\u003e enriching networks ..\n\u003e enriching facilities ..\n\u003e loading data from data/geofacs.json.gz\n\u003e error loading data/geofacs.json.gz\n\u003e warn: set OpenCageData API key (export OCD_KEY=...) for accurate geolocation\n\u003e saving to data/geofacs.json.gz (0 items)\n\n\u003e enriching IXs ..\n\u003e building country stats ..\n\u003e saving to data/final.json.gz (6 items)\n\n\u003e all done :) (102m 4s)\n```\n\n\u003c/details\u003e\n\n\n## data sources\nThis is the list of all \"data-raw\" files that we use as data sources for WNM.\n\n### RIRs\n```shell\n./data-raw/nic-afrinic.txt          # ftp://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-latest\n./data-raw/nic-apnic.txt            # ftp://ftp.apnic.net/pub/stats/apnic/delegated-apnic-latest\n./data-raw/nic-arin.txt             # ftp://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest\n./data-raw/nic-lacnic.txt           # ftp://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-latest\n./data-raw/nic-ripe.txt             # ftp://ftp.ripe.net/ripe/stats/delegated-ripencc-latest\n```\nInfo is consolidated to `./data/nics.json.gz`\n\n### peeringdb (peeringdb.org)\n```shell\n./data-raw/peeringdb.sqlite         # https://peeringdb.github.io/peeringdb-py/cli/\n```\nUsed by `wnm_merge.py` as one of the most important databases\n\n### telegeography (telegeography.com)\n```shell\n./data-raw/cables/*                 # https://github.com/telegeography/www.submarinecablemap.com\n```\nInfo is consolidated to `./data/scdb.json.gz`\n\n### BGP views (routeviews.org)\n```shell\n./data-raw/bgp-eqix.bz2             # http://routeviews.org/route-views.eqix/bgpdata/YYYY.MM/RIBS/rib.YYYYMMDD.HHMM.bz2\n./data-raw/bgp-jinx.bz2             # http://routeviews.org/route-views.jinx/bgpdata/YYYY.MM/RIBS/rib.YYYYMMDD.HHMM.bz2\n./data-raw/bgp-linx.bz2             # http://routeviews.org/route-views.linx/bgpdata/YYYY.MM/RIBS/rib.YYYYMMDD.HHMM.bz2\n./data-raw/bgp-saopaulo.bz2         # http://routeviews.org/route-views.saopaulo/bgpdata/YYYY.MM/RIBS/rib.YYYYMMDD.HHMM.bz2\n./data-raw/bgp-sg.bz2               # http://routeviews.org/route-views.sg/bgpdata/YYYY.MM/RIBS/rib.YYYYMMDD.HHMM.bz2\n```\nInfo is consolidated to `./data/bgp.json.gz`\n\n### world info (geonames.org)\n```shell\n./data-raw/worldinfo.txt            # http://download.geonames.org/export/dump/countryInfo.txt\n```\nInfo is consolidated to `./data/worldinfo.json.gz`\n\n### world cities (maxmind.org)\n```shell\n./data-raw/worldcitiespop.txt.gz    # http://download.maxmind.com/download/worldcities/worldcitiespop.txt.gz\n```\nInfo is consolidated to `./data/worldcitiespop.txt.gz`\n\n### AS names\n```shell\n./data-raw/as-afrinic.db.gz         # ftp://ftp.afrinic.net/pub/dbase/afrinic.db.gz\n./data-raw/as-apnic.db.gz           # https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz\n./data-raw/as-arin.db.gz            # ftp://ftp.arin.net/pub/rr/arin.db\n./data-raw/as-level3.db.gz          # ftp://rr.level3.net/pub/rr/level3.db.gz\n./data-raw/as-radb.db.gz            # ftp://ftp.radb.net/radb/dbase/radb.db.gz\n./data-raw/as-ripe.db.gz            # ftp://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz\n```\nInfo is consolidated to `./data/asnames.json.gz`\n\n\n## Consolidated data\n\nAll our consolidated data is stored in `./wnm/data/`.\nThe only file directly used by the web interface is `final.json.gz` which is the consolidation of all the others in the folder.\n\nHere is the list of all WNM databases:\n    - final.json.gz             # merged database directly usable by pwui\n\n    - asnames.json.gz           # AS number to AS name\n    - bgp.json.gz               # AS bgp routes, associated prefix \u0026 routing chains\n    - cities.json.gz            # world cities with country \u0026 gps coordinates\n    - geofacs.json.gz           # gps coordinates for all peeringdb facilities\n    - nics.json.gz              # RIR info: ASN, IPv4 \u0026 IPv6 association to countries\n    - scdb.json.gz              # extract of submarinecablemap.org database in our format\n    - worldinfo.json.gz         # countries surface, population, etc.\n\n## display / pwui.py\n\nTo display the map, the `pwui` python flask app was created.\nIt uses [datamaps.js](http://datamaps.github.io/) (which itself uses [d3.js](https://d3js.org/)) for the world map.\nIt uses [bootstrap](https://getbootstrap.com/) for the layout.\n\n## repository tree overview\n\n```shell\nwnm\n╷\n├── bins                # bgpdump utility\n├── wnm                 # data fetching, consolidation \u0026 processing python scripts\n│\n├── data                # our consolidated databases in json format\n│                       # (git clone https://github.com/ixty/wnm_data)\n├── data-raw            # raw data from open sources\n│                       # (git clone https://github.com/ixty/wnm_data-raw)\n│\n├── update.sh           # script to update all our local databases\n│\n├── html                # web interface flask templates\n├── pwui                # web interface python code\n├── res                 # web interface static files\n│\n├── requirements.txt    # python dependencies\n└── README.md           # documentation\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fixty%2Fwnm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fixty%2Fwnm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fixty%2Fwnm/lists"}