https://github.com/ooni/historical-geoip
Generate historical IP to country + ASN databases for processing historical OONI data
https://github.com/ooni/historical-geoip
Last synced: 3 months ago
JSON representation
Generate historical IP to country + ASN databases for processing historical OONI data
- Host: GitHub
- URL: https://github.com/ooni/historical-geoip
- Owner: ooni
- Created: 2022-07-14T16:25:21.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2024-03-22T13:16:06.000Z (over 2 years ago)
- Last Synced: 2024-04-14T09:04:43.666Z (over 2 years ago)
- Language: Python
- Homepage:
- Size: 43.9 KB
- Stars: 2
- Watchers: 7
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: Readme.md
Awesome Lists containing this project
README
# Historical GeoIP databases
The purpose of this repo is to build historical IP to country + AS databases
for use in the OONI data processing pipeline, but potentially for use by the
probes as well.
The data sources used for IP to country mappings are:
* Historical copies of [Maxmind GeoIP2 Country Lite](https://dev.maxmind.com/geoip/geolite2-free-geolocation-) from [20180206 to 20191224](https://archive.org/download/maxmind-geolite2-country).
* From 2020-02 onward we use monthly [DB-IP IP2Country Lite](https://db-ip.com/db/download/ip-to-country-lite), which are also [publicly archived](https://archive.org/download/dbip-country-lite).
For mapping IP ranges to ASNs we use the [prefix2as mappings from CAIDA](https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/).
For mapping ASNs with metadata about the organization, we use the as [AS to Organization mappings from CAIDA](https://publicdata.caida.org/datasets/as-organizations/).
The primary entry point for running the full workflow is the following:
```
./update_databases.sh
```
In order to upload the built artifacts to archive.org, you should have the set
`IA_ACCESS_KEY` and `IA_SECRET_KEY` environment variables.
The workflow for generating the final artifacts (the IP to country + ASN mmdb
files) is the following:
```mermaid
graph
A[AS Organizations] --> E{{AS to ORG Map}}
E --> D{Enrich country DB}
B[Maxmind GeoIP2 Country] --> D
C[DB-IP IP2Country] --> D
F[prefix2AS] --> D
D --> O{{IP to Country + ASN mmdb}}
```
Both the AS to ORG Map and the timestamped IP to Country + ASN mmdb files are
[published as artifacts on archive.org](https://archive.org/download/ip2country-as).
Note: if the repository is inactive for more than 60 days, the GitHub action
that publishes artifacts is disabled and we don't publish artifacts.
The IP to Country + ASN is compatible with the mmdb file format, but country
and ASN lookup are supported inside of the same call.
The keys used in the result for returning metadata information are the following:
* `autonomous_system_number`, this is an INT indicating the ASN. It's a standard key.
* `autonomous_system_organization`, this is a string indicating the
organization name for the given ASN. It's a standard key.
* `autonomous_system_country`, is the country of registration of the AS
organization. This key is non-standard.
* `autonomous_system_name`, is the name of the AS, which in most cases is
different from the organization name. This key is non-standard.
## Skipped workflows
If the workflow happens to be skipped for more than a month you may need to backfill the missing older dates.
This can be done by passing the missing timestamp to the `./update_databases.sh` script, like so:
```
./update_databases.sh YYYYMM
```
You can also manually trigger the workflow for debug purposes using the [github CLI](https://cli.github.com/) command:
```
gh workflow run .github/workflows/update_databases.yml
```