{"id":50181974,"url":"https://github.com/ipanalytics/asnforge","last_synced_at":"2026-06-07T09:01:26.453Z","repository":{"id":359981751,"uuid":"1248246817","full_name":"ipanalytics/ASNforge","owner":"ipanalytics","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-24T13:00:44.000Z","size":79,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-24T13:25:42.288Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ipanalytics.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-24T11:42:59.000Z","updated_at":"2026-05-24T13:00:48.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ipanalytics/ASNforge","commit_stats":null,"previous_names":["ipanalytics/asnforge"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/ipanalytics/ASNforge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipanalytics%2FASNforge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipanalytics%2FASNforge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipanalytics%2FASNforge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipanalytics%2FASNforge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ipanalytics","download_url":"https://codeload.github.com/ipanalytics/ASNforge/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipanalytics%2FASNforge/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33464014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-25T06:32:55.349Z","status":"ssl_error","status_checked_at":"2026-05-25T06:32:35.322Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-25T07:03:43.047Z","updated_at":"2026-05-25T07:05:02.270Z","avatar_url":"https://github.com/ipanalytics.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ASNForge\n\nASNForge builds reproducible ASN and prefix-origin intelligence artifacts for IP enrichment, routing analytics, and security data pipelines. It compiles public registry and routing inputs into a compact IP-to-ASN MaxMind DB, canonical ASN tables, prefix-origin snapshots, build metadata, checksums, and release-ready archives.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./site/banner.svg\" alt=\"ASNForge routing and registry intelligence compiler\" width=\"100%\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/ipanalytics/ASNforge/actions/workflows/ci.yml\"\u003e\u003cimg alt=\"CI\" src=\"https://img.shields.io/github/actions/workflow/status/ipanalytics/ASNforge/ci.yml?branch=main\u0026label=ci\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/ipanalytics/ASNforge/releases\"\u003e\u003cimg alt=\"Release\" src=\"https://img.shields.io/github/v/release/ipanalytics/ASNforge?include_prereleases\u0026label=release\"\u003e\u003c/a\u003e\n  \u003ca href=\"./LICENSE\"\u003e\u003cimg alt=\"License\" src=\"https://img.shields.io/badge/license-Apache%202.0-blue\"\u003e\u003c/a\u003e\n  \u003cimg alt=\"Go\" src=\"https://img.shields.io/badge/go-1.22%2B-00ADD8\"\u003e\n  \u003cimg alt=\"Dataset\" src=\"https://img.shields.io/badge/dataset-public--safe-success\"\u003e\n  \u003ca href=\"https://ipanalytics.github.io/ASNforge/\"\u003e\u003cimg alt=\"GitHub Pages\" src=\"https://img.shields.io/badge/pages-docs-brightgreen\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## Overview\n\nASNForge is a local compiler for ASN profile and prefix-origin datasets. It is designed for teams that need deterministic, inspectable artifacts instead of ad hoc enrichment files assembled by scripts.\n\nThe v0.1 pipeline ingests RIR delegated stats, bgp.tools prefix-origin and ASN catalog exports, static ipanalytics signal feeds, normalized local inputs, and curated overrides. It emits stable JSONL/CSV tables for analytics and joins, plus a compact MaxMind DB for latency-sensitive IP enrichment.\n\nThe compiler records build identifiers, schema versions, source hashes, artifact hashes, quality results, and release manifests so generated data can be traced and compared across builds.\n\n## System Behavior\n\n```text\nRIR delegated stats     manual overrides\n        │                    │\n        ▼                    ▼\n   ASN allocation table ── ASN profile normalization\n        │                    │\n        │                    ▼\nBGP prefix-origin feed ── prefix-origin aggregation ── MOAS policy\n        │                    │\n        ├───────────────┬────┴───────────────┐\n        ▼               ▼                    ▼\n  ASN JSONL/CSV   Prefix JSONL/CSV       Compact MMDB\n  ASN -\u003e profile  prefix -\u003e origins      IP -\u003e ASN profile\n```\n\nASNForge produces three related artifact families:\n\n| Artifact | Access pattern | Role |\n| --- | --- | --- |\n| `asnforge.mmdb` | IP address -\u003e origin ASN profile | Prefix-keyed MaxMind DB for local enrichment |\n| `asnforge-asn.jsonl` / `.csv` | ASN -\u003e ASN profile | Canonical table for direct ASN lookup and joins |\n| `asnforge-prefixes.jsonl` / `.csv` | Prefix -\u003e observed origin state | Prefix-origin snapshot with MOAS and collector state |\n\n## Features\n\n- RIR delegated extended parser for ASN allocation records.\n- Normalized BGP prefix-origin CSV/TSV parser with collector-aware aggregation.\n- bgp.tools bulk table parser for production prefix-origin snapshots.\n- bgp.tools ASN catalog parser for ASN names and coarse source classes.\n- Static ASN signal enrichment from IP-Knowledge-Layer and ASN-Signal-Graph.\n- Conservative name-based classification fallback for obvious ASN categories.\n- Manual ASN overrides for curated name, organization, type, tags, confidence, and field sources.\n- MOAS handling with deterministic policies: `mark_ambiguous`, `most_observed`, `lowest_asn`.\n- Private and reserved ASN handling with `flag`, `drop`, and `keep` policies.\n- Compact MaxMind DB writer for IP-prefix lookups.\n- Stable JSONL and CSV outputs with fixed headers and sorted records.\n- Build metadata with source hashes, artifact hashes, schema version, build id, and quality verdict.\n- Smoke tests, validation command, checksums, release manifest, and baseline diff output.\n- GitHub Actions CI and release workflow for scheduled or tag-driven artifact publication.\n\n## Quick Start\n\nThe local development profile is deterministic and does not require network access.\n\n```sh\ngo run ./cmd/asnforge build \\\n  --config config/local-dev.yaml \\\n  --out release/current \\\n  --build-id local-dev\n\ngo run ./cmd/asnforge validate --out release/current --strict\n```\n\nInspect generated data:\n\n```sh\ngo run ./cmd/asnforge inspect-ip 8.8.8.8 \\\n  --mmdb release/current/asnforge.mmdb \\\n  --format json\n\ngo run ./cmd/asnforge inspect-asn 15169 \\\n  --asn-table release/current/asnforge-asn.jsonl\n\ngo run ./cmd/asnforge stats --out release/current --format json\n```\n\n## Installation\n\nBuild from source:\n\n```sh\ngit clone https://github.com/ipanalytics/ASNforge.git\ncd ASNforge\ngo build -o asnforge ./cmd/asnforge\n```\n\nRun the compiler:\n\n```sh\n./asnforge build --config config/local-dev.yaml --out release/current\n./asnforge validate --out release/current --strict\n```\n\nRequirements:\n\n| Component | Version |\n| --- | --- |\n| Go | 1.22 or newer |\n| OS | Linux, macOS, or any environment supported by Go |\n| Network | Required only for configured HTTP source downloads |\n\n## CLI\n\n```text\nasnforge build\nasnforge download\nasnforge validate\nasnforge inspect-ip \u003cip\u003e\nasnforge inspect-asn \u003casn\u003e\nasnforge stats\nasnforge version\n```\n\nCommon flags:\n\n| Flag | Default | Description |\n| --- | --- | --- |\n| `--config` | `config/public-safe.yaml` | Build configuration |\n| `--out` | `release/current` | Release output directory |\n| `--cache` | `data/cache` | Source cache directory |\n| `--build-id` | UTC timestamp | Explicit build identifier |\n| `--schema-version` | `asnforge.v0.1` | Artifact schema version |\n| `--private-asn-policy` | config value | `flag`, `drop`, or `keep` |\n| `--moas-policy` | config value | `mark_ambiguous`, `most_observed`, or `lowest_asn` |\n| `--mmdb` | `\u003cout\u003e/asnforge.mmdb` | MMDB path for build or inspect |\n| `--skip-download` | `false` | Use cached/local source files |\n| `--strict` | `false` | Treat quality warnings as build failures |\n| `--format` | `text` | `text` or `json` where supported |\n\n## Outputs\n\nA successful build writes a complete release directory:\n\n```text\nrelease/current/\n├── asnforge.mmdb\n├── asnforge.mmdb.gz\n├── asnforge-asn.jsonl\n├── asnforge-asn.jsonl.gz\n├── asnforge-asn.csv\n├── asnforge-asn.csv.gz\n├── asnforge-prefixes.jsonl\n├── asnforge-prefixes.jsonl.gz\n├── asnforge-prefixes.csv\n├── asnforge-prefixes.csv.gz\n├── metadata.json\n├── checksums.txt\n├── quality-report.md\n├── asnforge-diff.json\n└── manifest.json\n```\n\n| File | Description |\n| --- | --- |\n| `asnforge.mmdb` | Compact MaxMind DB for IP -\u003e ASN profile lookup |\n| `asnforge-asn.jsonl` | Canonical ASN profile table |\n| `asnforge-asn.csv` | CSV form of the ASN profile table |\n| `asnforge-prefixes.jsonl` | Canonical prefix-origin snapshot |\n| `asnforge-prefixes.csv` | CSV form of prefix-origin state |\n| `metadata.json` | Build metadata, source hashes, artifact hashes, summary, quality verdict |\n| `checksums.txt` | SHA256 checksums for release artifacts |\n| `quality-report.md` | Human-readable build and quality report |\n| `asnforge-diff.json` | Baseline or release-to-release diff shape |\n| `manifest.json` | Machine-readable artifact manifest |\n\n## Data Formats\n\nEvery primary record includes:\n\n- `schema_version`\n- `build_id`\n\nThese fields make joins explicit and prevent accidental mixing of incompatible builds.\n\n### ASN Profile\n\n`asnforge-asn.jsonl` and `asnforge-asn.csv` contain one row per ASN:\n\n```json\n{\n  \"schema_version\": \"asnforge.v0.1\",\n  \"build_id\": \"local-dev\",\n  \"asn\": 15169,\n  \"asn_name\": \"Google LLC\",\n  \"asn_org\": \"Google\",\n  \"asn_type\": \"cloud\",\n  \"asn_tags\": [\"cloud\", \"dns\", \"manual-override\", \"search\"],\n  \"registration_country\": \"US\",\n  \"rir\": \"arin\",\n  \"asn_confidence\": 100\n}\n```\n\n`registration_country` is the registry allocation country from RIR delegated data. It is not user, host, or service geolocation.\n\n### Prefix Origin\n\n`asnforge-prefixes.jsonl` and `asnforge-prefixes.csv` preserve routing observation state:\n\n```json\n{\n  \"prefix\": \"8.8.8.0/24\",\n  \"origin_asns\": [15169],\n  \"selected_origin_asn\": 15169,\n  \"moas\": false,\n  \"origin_policy\": \"most_observed\",\n  \"observation_count\": 2,\n  \"source_collectors\": [\"ris-rrc00\", \"routeviews2\"],\n  \"prefix_confidence\": 90,\n  \"rpki_state\": \"unknown\"\n}\n```\n\n### MMDB Record\n\nThe MMDB is prefix-keyed and optimized for local IP enrichment:\n\n```json\n{\n  \"schema_version\": \"asnforge.v0.1\",\n  \"build_id\": \"local-dev\",\n  \"asn\": 15169,\n  \"asn_name\": \"Google LLC\",\n  \"asn_org\": \"Google\",\n  \"asn_type\": \"cloud\",\n  \"asn_tags\": [\"cloud\", \"dns\", \"manual-override\", \"search\"],\n  \"registration_country\": \"US\",\n  \"rir\": \"arin\",\n  \"moas\": false,\n  \"asn_confidence\": 100\n}\n```\n\nDetailed MOAS state, origin arrays, collector observations, and field-level provenance belong in the prefix and ASN tables. Keeping those fields out of the MMDB preserves data-section deduplication and keeps the database compact.\n\n## Operational Notes\n\n- Builds are deterministic for the same inputs, config, schema version, and build id.\n- ASN rows are sorted by numeric ASN.\n- Prefix rows are sorted by IP family, address bytes, and prefix length.\n- CSV list fields use semicolon-separated stable ordering.\n- `metadata.json` records source paths, URLs, SHA256 hashes, sizes, generated time, artifact hashes, and quality summary.\n- `validate --strict` is intended for CI and release workflows.\n\n## Source Profiles\n\n| Profile | Purpose | Network required |\n| --- | --- | --- |\n| `config/local-dev.yaml` | Deterministic development and CI fixture build | No |\n| `config/public-safe.yaml` | Public-safe release profile using RIR delegated files, bgp.tools exports, and static ipanalytics ASN signal feeds | Yes |\n| `config/research-caida.yaml` | Public-safe sources plus optional CAIDA ASRank, AS2Org, and AS relationships bulk files | Yes, plus operator-provided CAIDA files |\n\nThe public-safe profile downloads `https://bgp.tools/table.jsonl` for production prefix-origin input, `https://bgp.tools/asns.csv` for ASN names and source classes, and static raw CSV signal exports from IP-Knowledge-Layer and ASN-Signal-Graph. The deterministic fixture under `examples/testdata` is intentionally scoped to `config/local-dev.yaml`.\n\nThe research CAIDA profile is separate because CAIDA datasets have their own acceptable-use, citation, and redistribution terms. CAIDA fields are written to the ASN JSONL/CSV artifacts and are intentionally excluded from the compact MMDB.\n\nDefault CAIDA research inputs:\n\n| Dataset | File |\n| --- | --- |\n| AS2Org | `https://publicdata.caida.org/datasets/as-organizations/latest.as-org2info.txt.gz` |\n| AS relationships | Latest `*.as-rel2.txt.bz2` resolved from `https://publicdata.caida.org/datasets/as-relationships/serial-2/` |\n| ASRank | Operator-provided CSV path or URL; ASRank API crawling is not used |\n\nThe repository includes a monthly/manual `release-caida` workflow that publishes a prerelease tagged `research-caida-YYYYMMDD-HHMMSSZ`.\n\nThe v0.1 public-safe profile does not include CAIDA data by default. Optional PeeringDB, CAIDA, RPKI, and native MRT support are tracked as later source profiles and parser extensions.\n\n## Use Cases\n\n- IP enrichment in SIEM, fraud, abuse, and traffic analytics systems.\n- Local ASN profile joins in data warehouses and stream processors.\n- Prefix-origin snapshots for routing analytics and MOAS review.\n- Reproducible release artifacts for internal security data pipelines.\n- Build-time validation of third-party registry and routing source changes.\n\n## Scope\n\nASNForge v0.1 focuses on the pipeline shape: parsers, normalized models, deterministic outputs, compact MMDB generation, metadata, validation, and release automation.\n\nClassification is conservative. `asn_type` is a scored operational classification, not an authoritative registry fact. Confidence describes source agreement, completeness, or observation strength; it is not a risk score.\n\n## Limitations\n\n- Native MRT parsing is not implemented in v0.1.\n- Prefix-origin input is normalized CSV/TSV.\n- RPKI state defaults to `unknown`.\n- ASN classification is intentionally sparse without optional enrichment sources.\n- Live routing data varies by collector and collection time.\n\n## Directory Structure\n\n```text\n.\n├── cmd/asnforge/          # CLI entry point\n├── internal/asn/          # ASN models, classification, private/reserved policy\n├── internal/bgp/          # Prefix-origin parser and aggregation\n├── internal/build/        # Build pipeline, metadata, quality, diff\n├── internal/config/       # Config loading and CLI options\n├── internal/download/     # Source download, hashing, source state\n├── internal/mmdb/         # MaxMind DB writer and inspector\n├── internal/output/       # JSONL, CSV, gzip, checksums\n├── internal/rir/          # RIR delegated parser\n├── internal/smoke/        # Smoke test runner\n├── config/                # Build profiles\n├── schemas/               # JSON Schemas\n├── examples/              # Overrides, smoke cases, deterministic testdata\n├── docs/                  # Data source and artifact documentation\n└── .github/workflows/     # CI and release automation\n```\n\n## Deployment\n\nRelease artifacts are intended to be published through GitHub Releases, not committed to the repository.\n\nThe release workflow builds the CLI, runs the configured public-safe build, validates the output, computes checksums, creates a release tag, and uploads compressed data artifacts plus metadata:\n\n```sh\n./asnforge build --config config/public-safe.yaml --out release/current\n./asnforge validate --out release/current --strict\n```\n\nFor internal deployments, run the same commands in CI and publish `release/current/*` to object storage, package registries, or internal artifact repositories.\n\n## Documentation\n\n- [Data Sources](docs/DATA_SOURCES.md)\n- [Third-Party Data](docs/THIRD_PARTY_DATA.md)\n- [Classification](docs/CLASSIFICATION.md)\n- [MMDB Output](docs/MMDB_OUTPUT.md)\n- [Release Artifacts](docs/RELEASE_ARTIFACTS.md)\n- [Confidence](docs/CONFIDENCE.md)\n\n## License\n\nASNForge is licensed under the [Apache License 2.0](LICENSE).\n\n## Disclaimer\n\nASNForge aggregates registry and observed routing data for defensive, analytical, and operational use. Routing data is observational and may differ by collector and collection time.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fipanalytics%2Fasnforge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fipanalytics%2Fasnforge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fipanalytics%2Fasnforge/lists"}