{"id":19900085,"url":"https://github.com/m-lab/annotation-service","last_synced_at":"2025-08-09T12:40:32.416Z","repository":{"id":22168181,"uuid":"95483792","full_name":"m-lab/annotation-service","owner":"m-lab","description":"Annotation integration service for M-Lab data","archived":false,"fork":false,"pushed_at":"2024-06-10T05:51:06.000Z","size":139850,"stargazers_count":3,"open_issues_count":53,"forks_count":5,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-10-29T17:19:12.210Z","etag":null,"topics":["etl","pipeline"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-26T19:51:40.000Z","updated_at":"2022-03-08T17:08:58.000Z","dependencies_parsed_at":"2024-04-18T03:26:02.706Z","dependency_job_id":"1eb583a5-7c28-4803-882b-46512dec7f60","html_url":"https://github.com/m-lab/annotation-service","commit_stats":{"total_commits":692,"total_committers":13,"mean_commits":53.23076923076923,"dds":0.7832369942196532,"last_synced_commit":"f83cb7a5233c08243a77c8ed1d26243bdb3e488e"},"previous_names":[],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fannotation-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fannotation-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fannotation-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fannotation-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/annotation-service/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224015467,"owners_count":17241510,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","pipeline"],"created_at":"2024-11-12T20:11:06.962Z","updated_at":"2024-11-12T20:11:07.615Z","avatar_url":"https://github.com/m-lab.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# annotation-service\n\n[![GoDoc](https://godoc.org/github.com/m-lab/annotation-service?status.svg)](https://godoc.org/github.com/m-lab/annotation-service) [![Build Status](https://travis-ci.org/m-lab/annotation-service.svg?branch=master)](https://travis-ci.org/m-lab/annotation-service) [![Coverage Status](https://coveralls.io/repos/github/m-lab/annotation-service/badge.svg?branch=master)](https://coveralls.io/github/m-lab/annotation-service?branch=master) [![Go Report Card](https://goreportcard.com/badge/github.com/m-lab/annotation-service)](https://goreportcard.com/report/github.com/m-lab/annotation-service)\n\nAnnotation integration service provides geolocation for IPv4 and IPv6 MaxMind databases from Google Cloud Storage.\n\n## API\n\n### v1 - Deprecated\n\nThis is the original, deprecated API, which includes an unwrapped array of\nRequestData objects. Its use is discouraged.  Please use the v2 API, which\nhas better support for batch requests, including returning the date of the\nAnnotator used to provide the annotations.\n\n### v2\n\nThe v2 api introduces a standard wrapper struct, beginning with a string\nspecifying the version identifier, and an Info string that may include\narbitrary request info, e.g. for use in tracing or debugging.\nIt is described in the api/v2 package in the api/v2 directory.  The recommended\nGetAnnotations function is only available in the v2 package.\n\n### Response contents\n\nAnnotatation service will respond with the following data:\n- IP Address range\n- Postal Code\n- Latitude\n- Longitude\n- Continent Code\n- Country Code\n- Country Name\n- Metro Code\n- City Name\n\n---\n\n## Code structure\n\nThe code is divided into the following packages (organized in rough order of dependencies, except for api package):\n\n- api - defines external API, including GetAnnotations() call which handles composing and sending requests, with retries.\n- manager - handles caching of Annotators\n- directory - used by manager to create and keep track of CompositeAnnotators.\n- handler - receives incoming requests, handles marshalling, unmarshalling, interpretation of requests.\n- geoloader - maintains directory of available MaxMind (GEO) and Routeview (ASN) files, and selects which file(s) to use for a given date.  (Needs a lot of renaming)\n- asn - handles details of interpreting RouteViews ASN files, and creating ASN annotators.\n- geolite2v2 and legacy - handle details of interpreting MaxMind files and creating annotators.\nCurrently this is divided into two packages, but should be merged.\n- loader - handles files downloads and decompression\n- iputil - general IP utility functions that are used across asn, legacy geo,\nand geolite2 datasets.\n- metrics - all metric definitions.\n\n### Dependencies (as of April 2019)\n\n(higher depends on lower, left -\u003e depends on right)\n\n- main.go\n- manager -\u003e handler, directory\n- geoloader -\u003e asn, geolite2v2, legacy\n- iputils -\u003e loader\n- api, metrics\n\n---\n\n## Maxmind Dataset details\n\nIf an annotation request is dated prior to August 2017, location data will be\nderived from MaxMind GeoLiteLatest databases. Otherwise, data will be provided\nby MaxMind GeoLite2 databases. The discrepencies between provided databases are\nprovided below.\n\nMaxMind GeoLiteLatest databases include:\n\n1. GeoLiteCity-Blocks.csv\n\n    - StartIPNum IPv4\n    - EndIPNum  IPv4\n    - GeonameID\n\n1. GeoLiteCity-Location.csv\n\n    - GeonameID\n    - Country Code\n    - City\n    - Postal Code\n    - Logitude\n    - Latitude\n    - MetroCode\n\nMaxMind GeoLite2 databases include:\n\n1. GeoLite2-City-Blocks-IPv4.csv \u0026 GeoLite2-City-Blocks-IPv6.csv\n\n    - IP network (CIDR Format)\n    - GeonameID (identifies end user location)\n    - Registered Country Geoname ID (identifies country where IP address is\n      registered to an ISP)\n    - Latitude\n    - Longitude\n\n1. GeoLite2-City-Locations-en.csv\n\n    - GeonameID\n    - Continent Code\n    - Country ISO\n    - Country Name\n    - City Name\n    - Metro Code\n\nGeonameID is the same with Registered Country Geoname ID most of time, but with some exceptions.\nEither GeonameID or Registered Country Geoname ID could be not available for some IP addresses.\n\nImportant discrepencies to note include:\n\n1. GeoLite2 databases provides a network IP range in CIDR format while\n   GeoLiteLatest databases provide an IP range in decimal format.\n1. GeoLite2 provides both end user location as well as country registration\n   information while GeoLiteLatest includes only end user location.\n   www.maxmind.com/en/geoip2-precision-city-service\n\n### CompositeAnnotator\n\nCompositeAnnotator encapsulates multiple component api.Annotator objects, and satisfies the\napi.Annotator interface.\n\nMergeAnnotators takes two or more []api.Annotator, and merges them, creating CompositeAnnotators for\neach distinct date, using the most recent Annotator from each list prior to that date.\n\n### Directory\n\nDirectory wraps a []api.Annotator, and provides the GetAnnotator(date time.Time) function.\n\n### CachingLoader\n\nCachingLoader specifies the interface provided by loaders that load and cache a list of annotators.\n\n1. Maintains list of loaded Annotator objects.\n1. Refreshes the list of loaded objects on demand.\n\n```go\ntype CachingLoader interface {\n  UpdateCache() error\n  Fetch() []Annotator\n}\n```\n\n## Local Testing\n\nInstall dependencies for geoip-dev and pkg-config using your local package\nmanager. See .travis.yml and Dockerfile for examples.\n\nBecause the default operation of the annotation service is to load *all*\nhistorical data, the RAM requirements are significant. Instead, for local\ntesting, specify alterante date patterns for maxmind and routeview files.\n\n```sh\ngo get .\n~/bin/annotation-service -maxmind_dates '2013/10/07' -routeview_dates '2013/10'\n```\n\nThe annotation service supports two endpoints: `/batch_annotate` and\n`/annotate`. The `/annotate` resource accepts HTTP GET with parameters.\n\n- `since_epoch=` specifies the timestamp of the annotation.\n- `ip_addr=` specifies the IP address that should be annotated.\n\nNOTE: for local testing, only the loaded RouteView and Maxmind databases are\nused for all dates.\n\nPerform an adhoc query using `curl`:\n\n```sh\ncurl 'http://localhost:8080/annotate?since_epoch=1380600000\u0026ip_addr=67.86.65.1' | jq\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fannotation-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Fannotation-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fannotation-service/lists"}