{"id":19934176,"url":"https://github.com/quartz/ark","last_synced_at":"2026-04-11T22:49:33.968Z","repository":{"id":79263210,"uuid":"55785303","full_name":"Quartz/ark","owner":"Quartz","description":"Tools for processing Ark traceroute data","archived":false,"fork":false,"pushed_at":"2016-06-10T18:10:09.000Z","size":1248,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-01T11:45:43.816Z","etag":null,"topics":["ark","caida","postgresql","qz-things","tool"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Quartz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-04-08T14:31:12.000Z","updated_at":"2017-05-12T14:45:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"b528681a-7297-48cc-9ca4-822a0d265d6b","html_url":"https://github.com/Quartz/ark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Quartz/ark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quartz%2Fark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quartz%2Fark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quartz%2Fark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quartz%2Fark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Quartz","download_url":"https://codeload.github.com/Quartz/ark/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quartz%2Fark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31698152,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-11T21:17:31.016Z","status":"ssl_error","status_checked_at":"2026-04-11T21:17:24.556Z","response_time":54,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ark","caida","postgresql","qz-things","tool"],"created_at":"2024-11-12T23:16:03.108Z","updated_at":"2026-04-11T22:49:33.928Z","avatar_url":"https://github.com/Quartz.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ark\n\nTools for processing traceroute data from CAIDA's [Ark project](http://www.caida.org/projects/ark/).\n\nTools in the `ark-tools` folder were provided by CAIDA's [Young Hyun](http://www.caida.org/~youngh/).\n\n## Setup\n\n```\nmkvirtualenv ark\npip install -r requirements.txt\n\ncd ark-tools\ngem install rb-asfinder-0.10.1.gem rb-wartslib-1.4.2.gem\ncd ..   \n```\n\nThis project also requires a running, local instance of Postgres with a no-password user named `ark` who owns a geo-enabled database named `ark`:\n\n```\ncreatedb -O ark ark\npsql -q ark -c \"CREATE EXTENSION postgis;\"\n```\n\n## Sourcing the data\n\nThis script uses data from CAIDA's [Ark IPv4 Routed /24 Topology Dataset](http://www.caida.org/data/active/ipv4_routed_24_topology_dataset.xml). The following script will will download all data, for all three monitoring teams, for every day in March of 2014. Caution: **This is 87GB of data**. The script can be stopped and started without starting over.\n\n```\n./fetch.sh\n```\n\nYou will also need to download the following files to the root project directory:\n\n* [MaxMind GeoLite2 Country database](http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz) (unzip it)\n* [BGP Reports mapping of AS numbers to names](http://bgp.potaroo.net/as6447/asnames.txt)\n\n## Building the database\n\nWarning: This script will run for around a half hour **per day of data**. If you're loading a month of data it could easily take a full day. (Assuming you even have the disk space to hold it all.)\n\n```\n./process.py\n```\n\n## Running queries\n\n```\ncat by_country.sql | psql -q ark\ncat by_monitor.sql | psql -q ark\n```\n\n## Analyzing a trace path\n\nYou can pass a `trace` from the database into `parse_trace.py` to generate detailed path data in CSV format:\n\n```\n./parse_trace.py \"216.66.30.102:6939,216.66.30.101:6939,213.248.67.125:1299,213.155.130.34:1299,157.130.60.13:701,:q,:r,108.51.141.48:701\" \u003e nyc_to_dc.csv\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquartz%2Fark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquartz%2Fark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquartz%2Fark/lists"}