{"id":13586617,"url":"https://github.com/firefart/network_info","last_synced_at":"2025-03-15T14:30:31.843Z","repository":{"id":12805378,"uuid":"42139203","full_name":"firefart/network_info","owner":"firefart","description":"Parse Network Info Databases (ARIN/APNIC/LACNIC/AfriNIC/RIPE)","archived":false,"fork":false,"pushed_at":"2024-05-13T16:23:25.000Z","size":174,"stargazers_count":88,"open_issues_count":3,"forks_count":32,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-05-13T17:47:35.924Z","etag":null,"topics":["afrinic","apnic","arin","lacnic","network-info","parser","ripe"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/firefart.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-09-08T21:09:55.000Z","updated_at":"2024-05-29T08:08:41.789Z","dependencies_parsed_at":"2023-02-18T23:31:06.076Z","dependency_job_id":"ed8e268f-124e-4604-83c8-bce1ba8f33ca","html_url":"https://github.com/firefart/network_info","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firefart%2Fnetwork_info","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firefart%2Fnetwork_info/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firefart%2Fnetwork_info/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firefart%2Fnetwork_info/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/firefart","download_url":"https://codeload.github.com/firefart/network_info/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243742522,"owners_count":20340662,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["afrinic","apnic","arin","lacnic","network-info","parser","ripe"],"created_at":"2024-08-01T15:05:41.337Z","updated_at":"2025-03-15T14:30:31.837Z","avatar_url":"https://github.com/firefart.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Network Info Parser\n\nThis script parses the ARIN/APNIC/LACNIC/AfriNIC/RIPE databases into a local PostgreSQL database.\nAfter the parsing is finished you can get the infos for any IPv4 or IPv6 by querying the database.\n\nThis project was used in analysing some data dumps and cross referencing the IPs with the networks.\nIt can also be used to easily search for netranges assigned to a company in interest.\n\nI recommend using the docker setup because it removes the hassle of installing everything manually.\n\nHint: The Database can grow fast so be sure to have enough space. On docker my postgres database uses 4.066GB of space.\n\n# Requirements\n\n- Python3 \u003e= 3.3\n- postgresql\n- python3-netaddr\n- python3-psycopg\n- python3-sqlalchemy\n\n# Docker\n\nYou can simply pull the image from Docker Hub and connect it to a local database via\n\n```sh\ndocker pull firefart/network_info\ndocker run --rm firefart/network_info -c postgresql://user:pass@db:5432/network_info\n```\n\nOr you can connect the docker container to another database container.\n\n```sh\ndocker run --name network_info_db -e POSTGRES_DB=network_info -e POSTGRES_USER=network_info -e POSTGRES_PASSWORD=network_info -d postgres:9-alpine\ndocker run --rm --link network_info_db:postgres firefart/network_info -c postgresql://user:pass@db:5432/network_info\n```\n\nIf you have checked out the GIT repo you can run the script via `docker compose`.\nI included some binstubs so you don't have to deal with all the docker commands.\n\nIf you run\n\n```sh\n./bin/network_info\n```\n\nthe image will be built, a postgres database is connected, the files are downloaded and the parsing begins.\nThe database stays up after the run (you can see it via `docker ps`) so you can connect it to your script.\n\nFor a one shot query you can run\n\n```\n./bin/query IPv4\n```\n\nor\n\n```\n./bin/query IPv6\n```\n\nOr for a psql prompt\n\n```\n./bin/psql\n```\n\n# Manual Installation\n\nInstallation of needed packages (Example on Ubuntu 16.04):\n\n```sh\napt install postgresql python3 python3-netaddr python3-psycopg2 python3-sqlalchemy\n```\n\nor -\n\n```sh\napt install postgresql python3 python-pip\npip install -r requirements.txt\n```\n\nCreate PostgreSQL database (Use \"network_info\" as password):\n\n```sh\nsudo -u postgres createuser --pwprompt --createdb network_info\nsudo -u postgres createdb --owner=network_info network_info\n```\n\nPrior to starting this script you need to download the database dumps by executing:\n\n```sh\n./download_dumps.sh\n```\n\nAfter importing you can lookup an IP address like:\n\n```sql\nSELECT block.inetnum, block.netname, block.country, block.description, block.maintained_by, block.created, block.last_modified, block.source FROM block WHERE block.inetnum \u003e\u003e '2001:db8::1' ORDER BY block.inetnum DESC;\nSELECT block.inetnum, block.netname, block.country, block.description, block.maintained_by, block.created, block.last_modified, block.source FROM block WHERE block.inetnum \u003e\u003e '8.8.8.8' ORDER BY block.inetnum DESC;\n```\n\nor -\n\n```bash\n./query_db.sh 192.0.2.1\n```\n\n# Sample run (docker compose)\n\n```\n$ ./bin/network_info\nCreating network \"ripe_default\" with the default driver\nCreating volume \"ripe_pg_data\" with local driver\nCreating ripe_db_1\nDownloading afrinic.db.gz...\nConnecting to ftp.afrinic.net (196.216.2.24:21)\nafrinic.db.gz        100% |****************************************************************************************************************************|  5419k  0:00:00 ETA\nDownloading apnic.db.inetnum.gz...\nConnecting to ftp.apnic.net (202.12.29.205:21)\napnic.db.inetnum.gz  100% |****************************************************************************************************************************| 37065k  0:00:00 ETA\nDownloading apnic.db.inet6num.gz...\nConnecting to ftp.apnic.net (202.12.29.205:21)\napnic.db.inet6num.gz 100% |****************************************************************************************************************************|  1113k  0:00:00 ETA\nDownloading arin.db...\nConnecting to ftp.arin.net (199.71.0.151:21)\narin.db              100% |****************************************************************************************************************************| 12314k  0:00:00 ETA\nDownloading delegated-lacnic-extended-latest...\nConnecting to ftp.lacnic.net (200.3.14.11:21)\ndelegated-lacnic-ext 100% |****************************************************************************************************************************|  2161k  0:00:00 ETA\nDownloading ripe.db.inetnum.gz...\nConnecting to ftp.ripe.net (193.0.6.140:21)\nripe.db.inetnum.gz   100% |****************************************************************************************************************************|   228M  0:00:00 ETA\nDownloading ripe.db.inet6num.gz...\nConnecting to ftp.ripe.net (193.0.6.140:21)\nripe.db.inet6num.gz  100% |****************************************************************************************************************************| 24589k  0:00:00 ETA\n2020-06-23 20:21:11,836 - create_db - INFO     - MainProcess - afrinic.db.gz - parsing database file: ./databases/afrinic.db.gz\n2020-06-23 20:21:20,688 - create_db - INFO     - MainProcess - afrinic.db.gz - Got 171337 blocks\n2020-06-23 20:21:20,688 - create_db - INFO     - MainProcess - afrinic.db.gz - database parsing finished: 8.85 seconds\n2020-06-23 20:21:20,688 - create_db - INFO     - MainProcess - afrinic.db.gz - parsing blocks\n2020-06-23 20:21:40,904 - create_db - INFO     - MainProcess - afrinic.db.gz - block parsing finished: 20.22 seconds\n2020-06-23 20:21:40,906 - create_db - INFO     - MainProcess - apnic.db.inet6num.gz - parsing database file: ./databases/apnic.db.inet6num.gz\n2020-06-23 20:21:43,364 - create_db - INFO     - MainProcess - apnic.db.inet6num.gz - Got 73190 blocks\n2020-06-23 20:21:43,376 - create_db - INFO     - MainProcess - apnic.db.inet6num.gz - database parsing finished: 2.47 seconds\n2020-06-23 20:21:43,376 - create_db - INFO     - MainProcess - apnic.db.inet6num.gz - parsing blocks\n2020-06-23 20:21:52,059 - create_db - INFO     - MainProcess - apnic.db.inet6num.gz - block parsing finished: 8.68 seconds\n2020-06-23 20:21:52,061 - create_db - INFO     - MainProcess - apnic.db.inetnum.gz - parsing database file: ./databases/apnic.db.inetnum.gz\n2020-06-23 20:22:21,802 - create_db - INFO     - MainProcess - apnic.db.inetnum.gz - Got 1079011 blocks\n2020-06-23 20:22:21,809 - create_db - INFO     - MainProcess - apnic.db.inetnum.gz - database parsing finished: 29.75 seconds\n2020-06-23 20:22:21,809 - create_db - INFO     - MainProcess - apnic.db.inetnum.gz - parsing blocks\n2020-06-23 20:24:43,940 - create_db - INFO     - MainProcess - apnic.db.inetnum.gz - block parsing finished: 142.13 seconds\n2020-06-23 20:24:43,942 - create_db - INFO     - MainProcess - arin.db - parsing database file: ./databases/arin.db\n2020-06-23 20:24:43,946 - create_db - INFO     - MainProcess - arin.db - Got 0 blocks\n2020-06-23 20:24:44,008 - create_db - INFO     - MainProcess - arin.db - database parsing finished: 0.07 seconds\n2020-06-23 20:24:44,008 - create_db - INFO     - MainProcess - arin.db - parsing blocks\n2020-06-23 20:24:44,030 - create_db - INFO     - MainProcess - arin.db - block parsing finished: 0.02 seconds\n2020-06-23 20:24:44,032 - create_db - INFO     - MainProcess - delegated-lacnic-extended-latest - parsing database file: ./databases/delegated-lacnic-extended-latest\n2020-06-23 20:24:44,034 - create_db - WARNING  - MainProcess - delegated-lacnic-extended-latest - line does not start with lacnic: b'2.3|lacnic|20200621|73188|19870101|20200619|-0300'\n2020-06-23 20:24:44,035 - create_db - WARNING  - MainProcess - delegated-lacnic-extended-latest - Invalid line: b'lacnic|*|ipv4|*|18136|summary'\n2020-06-23 20:24:44,035 - create_db - WARNING  - MainProcess - delegated-lacnic-extended-latest - Invalid line: b'lacnic|*|ipv6|*|43276|summary'\n2020-06-23 20:24:44,035 - create_db - WARNING  - MainProcess - delegated-lacnic-extended-latest - Invalid line: b'lacnic|*|asn|*|11776|summary'\n2020-06-23 20:24:44,489 - create_db - INFO     - MainProcess - delegated-lacnic-extended-latest - Got 61412 blocks\n2020-06-23 20:24:44,489 - create_db - INFO     - MainProcess - delegated-lacnic-extended-latest - database parsing finished: 0.46 seconds\n2020-06-23 20:24:44,489 - create_db - INFO     - MainProcess - delegated-lacnic-extended-latest - parsing blocks\n2020-06-23 20:24:51,207 - create_db - INFO     - MainProcess - delegated-lacnic-extended-latest - block parsing finished: 6.72 seconds\n2020-06-23 20:24:51,209 - create_db - INFO     - MainProcess - ripe.db.inetnum.gz - parsing database file: ./databases/ripe.db.inetnum.gz\n2020-06-23 20:27:46,474 - create_db - INFO     - MainProcess - ripe.db.inetnum.gz - Got 4191047 blocks\n2020-06-23 20:27:46,476 - create_db - INFO     - MainProcess - ripe.db.inetnum.gz - database parsing finished: 175.27 seconds\n2020-06-23 20:27:46,476 - create_db - INFO     - MainProcess - ripe.db.inetnum.gz - parsing blocks\n2020-06-23 20:37:17,835 - create_db - INFO     - MainProcess - ripe.db.inetnum.gz - block parsing finished: 571.36 seconds\n2020-06-23 20:37:17,837 - create_db - INFO     - MainProcess - ripe.db.inet6num.gz - parsing database file: ./databases/ripe.db.inet6num.gz\n2020-06-23 20:37:57,369 - create_db - INFO     - MainProcess - ripe.db.inet6num.gz - Got 1081478 blocks\n2020-06-23 20:37:57,706 - create_db - INFO     - MainProcess - ripe.db.inet6num.gz - database parsing finished: 39.87 seconds\n2020-06-23 20:37:57,706 - create_db - INFO     - MainProcess - ripe.db.inet6num.gz - parsing blocks\n2020-06-23 20:40:21,534 - create_db - INFO     - MainProcess - ripe.db.inet6num.gz - block parsing finished: 143.83 seconds\n2020-06-23 20:40:21,534 - create_db - INFO     - MainProcess - empty - script finished: 1149.83 seconds\n\n$ ./bin/query 8.8.8.8\nSELECT block.inetnum, block.netname, block.country, block.description, block.maintained_by, block.created, block.last_modified, block.source FROM block WHERE block.inetnum \u003e\u003e '8.8.8.8' ORDER BY block.inetnum DESC;\n-[ RECORD 1 ]-+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\ninetnum       | 8.0.0.0/8\nnetname       | IANA-NETBLOCK-8\ncountry       | AU\ndescription   | This network range is not allocated to APNIC. If your whois search has returned this message, then you have searched the APNIC whois database for an address that is allocated by another Regional Internet Registry (RIR). Please search the other RIRs at whois.arin.net or whois.ripe.net for more information about that range.\nmaintained_by | MAINT-APNIC-AP\ncreated       |\nlast_modified | 2008-09-04 06:51:28\nsource        | apnic\n-[ RECORD 2 ]-+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\ninetnum       | 8.0.0.0/6\nnetname       | NON-RIPE-NCC-MANAGED-ADDRESS-BLOCK\ncountry       | EU # Country is really world wide\ndescription   | IPv4 address block not managed by the RIPE NCC\nmaintained_by | RIPE-NCC-HM-MNT\ncreated       | 2019-01-07 10:49:33\nlast_modified | 2019-01-07 10:49:33\nsource        | ripe\n-[ RECORD 3 ]-+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\ninetnum       | 0.0.0.0/0\nnetname       | IANA-BLK\ncountry       | EU # Country is really world wide\ndescription   | The whole IPv4 address space\nmaintained_by | AFRINIC-HM-MNT\ncreated       |\nlast_modified |\nsource        | afrinic\n-[ RECORD 4 ]-+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\ninetnum       | 0.0.0.0/0\nnetname       | IANA-BLK\ncountry       | EU # Country field is actually all countries in the world and not just EU countries\ndescription   | The whole IPv4 address space\nmaintained_by | RIPE-NCC-HM-MNT\ncreated       | 2002-06-25 14:19:09\nlast_modified | 2018-11-23 10:30:34\nsource        | ripe\n-[ RECORD 5 ]-+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\ninetnum       | 0.0.0.0/0\nnetname       | IANA-BLOCK\ncountry       | AU\ndescription   | General placeholder reference for all IPv4 addresses\nmaintained_by | MAINT-APNIC-AP\ncreated       |\nlast_modified | 2008-09-04 06:51:49\nsource        | apnic\n```\n\n# Export block table\n\nIf you need to export data from PG to another source (Clickhouse, Elasticsearch, etc.) you can use:\n```\n./bin/export_to_gzip \n```\nA compressed CSV file will be created.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirefart%2Fnetwork_info","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffirefart%2Fnetwork_info","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirefart%2Fnetwork_info/lists"}