{"id":50733384,"url":"https://github.com/timfanda35/aws-pricing-to-bq","last_synced_at":"2026-06-10T11:01:34.412Z","repository":{"id":362291592,"uuid":"1254858539","full_name":"timfanda35/aws-pricing-to-bq","owner":"timfanda35","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-03T12:33:32.000Z","size":99,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-03T14:17:08.339Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/timfanda35.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-31T04:49:22.000Z","updated_at":"2026-06-03T12:33:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/timfanda35/aws-pricing-to-bq","commit_stats":null,"previous_names":["timfanda35/aws-pricing-to-bq"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/timfanda35/aws-pricing-to-bq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timfanda35%2Faws-pricing-to-bq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timfanda35%2Faws-pricing-to-bq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timfanda35%2Faws-pricing-to-bq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timfanda35%2Faws-pricing-to-bq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/timfanda35","download_url":"https://codeload.github.com/timfanda35/aws-pricing-to-bq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timfanda35%2Faws-pricing-to-bq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34149132,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-10T02:00:07.152Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-10T11:01:33.433Z","updated_at":"2026-06-10T11:01:34.406Z","avatar_url":"https://github.com/timfanda35.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# aws-pricing-to-bq\n\nDaily loader for the [AWS Price List Bulk API](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/using-price-list-api.html), sinking to **BigQuery** via GCS. Runs as a Cloud Run Job; consumers query the dataset directly via SQL.\n\nSibling of [`azure-pricing-to-bq`](../azure-pricing-to-bq) and the (Postgres-targeted) [`aws-pricing-list-loader`](https://github.com/timfanda35/aws-pricing-list-loader) — same shape, same query mental model, just different cloud source.\n\n## What lands in BigQuery\n\n| Table | Shape | Who reads it |\n|---|---|---|\n| `aws_pricing` | Latest snapshot. Not partitioned. Clustered by `service_code, region_code, sku`. | **Default for consumers.** Plain `SELECT *` returns today's prices. |\n| `aws_pricing_history` | Append-only history. `PARTITION BY ingestion_date`. **`require_partition_filter = TRUE`.** Same clustering. | Time-travel queries (price changes over time). |\n| `aws_pricing_versions` | One row per (service, region, offer_type) — the AWS-supplied `version` we last loaded. | Internal: powers the incremental diff so the loader skips unchanged offers. |\n| `pricing_runs` | Audit: one row per loader invocation. | Operations / monitoring. |\n\nThe live `aws_pricing` table is rebuilt at the end of every successful run by a single atomic `CREATE OR REPLACE TABLE` — consumers either see yesterday's snapshot or today's, never a half-loaded mix.\n\n## How incremental loading works (the 20 GB problem)\n\nAWS publishes pricing as ~6000 `(service, region)` JSON files, ~5–10 GB total. **AWS does not publish deltas** — each file is monolithic. But each file carries a `version` timestamp embedded in its URL, and most files don't change on any given day.\n\nSo every run:\n\n1. Walks the [master index](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json) and every per-service `region_index.json`.\n2. Joins the discovered `(service, region, version)` tuples against `aws_pricing_versions`.\n3. Downloads only the rows whose version changed.\n4. LOADs those into today's history partition.\n5. Carries forward the unchanged `(service, region)` rows from the previous partition with a single `INSERT ... SELECT * REPLACE (ingestion_date)`.\n6. Rebuilds the live `aws_pricing` table from today's partition.\n\nA typical day moves a few hundred MB at most.\n\n### Same-day reruns\n\nThe loader is safe to run multiple times on the same UTC day. On the second\nrun, today's partition already has data from the first run, so:\n\n1. The loader DELETEs only the rows for `(service, region)` pairs whose AWS\n   version changed since the first run.\n2. The LOAD JOB then APPENDs fresh rows for those pairs.\n3. The live `aws_pricing` table is rebuilt at the end via `CREATE OR REPLACE`.\n\nUnchanged pairs are never touched. (Earlier versions of the loader used\nWRITE_TRUNCATE on the LOAD JOB and would wipe unchanged data on same-day\nreruns — fixed.)\n\n## Schema strategy: one table, JSON for the variable parts\n\nAWS pricing data is notoriously irregular — `products[*].attributes` varies per service (EC2 has `instanceType/vcpu/memory`; S3 has `storageClass`; RDS has `engineCode`; ...) and AWS adds new keys without warning.\n\nRather than 200+ per-service tables, this project lands **one unified `aws_pricing_history` table** with stable columns promoted to typed columns and the service-specific bag stored in a native BigQuery `JSON` column:\n\n```sql\nSELECT\n  region_code,\n  JSON_VALUE(attributes, '$.instanceType') AS instance_type,\n  JSON_VALUE(attributes, '$.vcpu')         AS vcpu,\n  price_per_unit\nFROM `\u003cproject\u003e.aws_pricing.aws_pricing`\nWHERE service_code = 'AmazonEC2'\n  AND term_type    = 'OnDemand'\n  AND region_code  = 'us-east-1'\nLIMIT 20;\n```\n\nNew AWS attributes just appear inside the JSON. Zero DDL evolution.\n\n**Price columns use `BIGNUMERIC`, not `NUMERIC`.** `NUMERIC` only carries 9 fractional digits, but AWS routinely publishes prices like `0.0001000000` (10 digits) and `1.0E-4` — `NUMERIC` would silently round those.\n\n## Local dev\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate\npip install -r requirements-dev.txt\ncp .env.example .env   # fill GCP_PROJECT and GCS_STAGING_BUCKET\n\nruff check .\npytest -q\n```\n\n### Filtered smoke against a real GCP project\n\n```bash\ngcloud auth application-default login\nexport GCP_PROJECT=\u003cdev-project\u003e\nexport BQ_DATASET=aws_pricing_dev\nexport GCS_STAGING_BUCKET=\u003cdev-project\u003e-aws-pricing-staging\n# MVP: validate end-to-end with just a few services\nexport AWS_SERVICE_FILTER=\"AmazonEC2,AmazonRDS,AmazonS3\"\nexport AWS_INCLUDE_SAVINGS_PLANS=false\n\npython -m aws_pricing_to_bq setup\npython -m aws_pricing_to_bq load\npython -m aws_pricing_to_bq runs --limit 5\n\nbq query --use_legacy_sql=false \\\n  'SELECT service_code, region_code, COUNT(*) AS rows\n   FROM `'\"$GCP_PROJECT\"'.aws_pricing_dev.aws_pricing`\n   GROUP BY 1, 2 ORDER BY rows DESC LIMIT 10'\n```\n\nA second run shortly after should be a near-noop: `changed=0`, `skipped≈300+`, elapsed under a minute. That's the version-diff doing its job.\n\nThe history table will reject any query without a `WHERE ingestion_date = …` filter — that protects all consumers from accidental full-table scans.\n\n## CLI\n\n```bash\npython -m aws_pricing_to_bq setup\npython -m aws_pricing_to_bq load [--force] [--service-filter \"AmazonEC2,AmazonRDS\"]\npython -m aws_pricing_to_bq runs [--limit N]\n```\n\n`--force` bypasses the version diff and re-downloads every `(service, region)`. Useful for backfills or recovering from a corrupted partition.\n\n## Docker / Cloud Run Job\n\n```bash\ndocker build -t aws-pricing-to-bq:dev .\n\n# Local smoke (uses your gcloud ADC creds):\ndocker compose up\n```\n\nThe image's default `CMD` is `python run_job.py`, which is exactly what Cloud Run Job invokes.\n\n## Deployment (GCP)\n\n1. **GitHub Actions → GHCR** — on every push to `main` (or a `v*` tag), the workflow in `.github/workflows/docker-publish.yml` runs `pytest` and, on success, builds and pushes the image to `ghcr.io/\u003cowner\u003e/\u003crepo\u003e`:\n   - `latest` — tracks the `main` branch\n   - `\u003csemver\u003e` — created when you push a `v*` tag (e.g. `v1.2.3`)\n\n2. **GCS staging bucket** — same region as the BQ dataset. **Add a lifecycle rule to delete objects older than 7 days** so failed-run debris cleans itself up.\n3. **Service account** for the Cloud Run Job:\n   - `roles/bigquery.dataEditor` on the dataset\n   - `roles/bigquery.jobUser` on the project\n   - `roles/storage.objectAdmin` on the staging bucket\n4. **Cloud Run Job** `aws-pricing-loader-job`:\n   - **task timeout 3600s** (default 600s is too short for a full first load)\n   - parallelism 1, max retries 1\n   - Image: `ghcr.io/\u003cowner\u003e/\u003crepo\u003e:latest` (or pin a semver tag)\n   - `CMD [\"python\",\"run_job.py\"]`\n5. **Cloud Scheduler**: daily 02:00 UTC → Cloud Run Job admin API with OIDC token (scheduler SA needs `roles/run.invoker`).\n\nThere is no Cloud Run Service deployment.\n\n## Sample consumer queries\n\nEC2 m5.large OnDemand by region:\n\n```sql\nSELECT\n  region_code,\n  JSON_VALUE(attributes, '$.instanceType') AS instance_type,\n  price_per_unit,\n  unit\nFROM `\u003cproject\u003e.aws_pricing.aws_pricing`\nWHERE service_code = 'AmazonEC2'\n  AND term_type    = 'OnDemand'\n  AND JSON_VALUE(attributes, '$.instanceType') = 'm5.large'\n  AND JSON_VALUE(attributes, '$.operatingSystem') = 'Linux'\n  AND JSON_VALUE(attributes, '$.tenancy') = 'Shared'\n  AND JSON_VALUE(attributes, '$.preInstalledSw') = 'NA'\nORDER BY region_code;\n```\n\nS3 storage classes pricing:\n\n```sql\nSELECT\n  region_code,\n  JSON_VALUE(attributes, '$.storageClass') AS storage_class,\n  price_per_unit,\n  unit\nFROM `\u003cproject\u003e.aws_pricing.aws_pricing`\nWHERE service_code = 'AmazonS3'\n  AND term_type    = 'OnDemand'\nORDER BY region_code, storage_class;\n```\n\nPrice-over-time (history table; partition filter required):\n\n```sql\nSELECT\n  ingestion_date,\n  region_code,\n  price_per_unit\nFROM `\u003cproject\u003e.aws_pricing.aws_pricing_history`\nWHERE ingestion_date BETWEEN DATE '2026-05-01' AND DATE '2026-05-27'\n  AND service_code = 'AmazonEC2'\n  AND JSON_VALUE(attributes, '$.instanceType') = 'm5.large'\n  AND region_code = 'us-east-1'\n  AND term_type = 'OnDemand'\nORDER BY ingestion_date;\n```\n\n## Cross-team access\n\nGranting another team read access:\n\n- `roles/bigquery.dataViewer` on the dataset (or per-table on `aws_pricing` only for tighter scope).\n- They also need `roles/bigquery.jobUser` in **their own project** to run queries — they pay their own query cost (standard BigQuery billing pattern).\n\n## Configuration\n\n| Var | Default | Purpose |\n|---|---|---|\n| `GCP_PROJECT` | — | GCP project ID (required) |\n| `BQ_DATASET` | `aws_pricing` | dataset name |\n| `BQ_LOCATION` | `US` | dataset region; must match staging bucket region |\n| `GCS_STAGING_BUCKET` | — | bucket for intermediate JSONL files (required) |\n| `GCS_STAGING_PREFIX` | `ingestion/` | object key prefix |\n| `AWS_PRICING_BASE_URL` | `https://pricing.us-east-1.amazonaws.com` | |\n| `AWS_REQUEST_TIMEOUT_S` | `60` | per-request timeout (full EC2 offer ~200 MB) |\n| `AWS_MAX_RETRIES` | `5` | tenacity retries on 429 / 5xx |\n| `AWS_MAX_WORKERS` | `10` | parallel offer downloads |\n| `AWS_SERVICE_FILTER` | `` | comma-sep offer codes, e.g. `AmazonEC2,AmazonRDS,AmazonS3`. Empty = all. |\n| `AWS_INCLUDE_SAVINGS_PLANS` | `true` | also load Compute / Database / ML Savings Plans |\n| `AWS_INCLUDE_RESERVED` | `true` | also flatten Reserved Instance terms |\n| `HTTP_PROXY` | `` | proxy for outbound HTTP |\n| `HTTPS_PROXY` | `` | proxy for outbound HTTPS |\n| `NO_PROXY` | `` | proxy bypass list |\n| `JSONL_BATCH_SIZE` | `10000` | items per uploaded JSONL file |\n| `LOG_LEVEL` | `INFO` | |\n\n## Memory budget on Cloud Run\n\nCloud Run's `/tmp` is **tmpfs (RAM-backed)** — anything we write to disk counts\nagainst the instance's memory limit. The loader compensates by gzipping the\ndownloaded offer JSON on tmpfs (transparently re-opened by the parser via\nmagic-byte detection), which cuts that 6-8x. The dominant per-worker costs\nduring an offer download:\n\n| Per worker (large offer in flight) | Cost |\n|---|---|\n| Downloaded offer JSON on `/tmp`, gzipped (EC2 us-east-1) | ~30 MB |\n| `products` lookup as raw JSON bytes (50K-100K SKUs, no Python object overhead) | ~310 MB |\n| In-flight NDJSON temp file on `/tmp`, gzipped (BQ LOAD reads `.jsonl.gz` natively) | ~120 MB |\n| **Subtotal per worker** | **~460 MB** |\n\nWith `AWS_MAX_WORKERS=3` and Python runtime + libraries (~300 MB), total peak\nis ~1.7 GB — fits in a 4 GiB instance with margin. Sizing reference:\n\n| Cloud Run memory | Recommended `AWS_MAX_WORKERS` |\n|---|---|\n| 4 GiB | 3 (default) |\n| 8 GiB | 5–6 |\n| 16 GiB | 8–10 |\n\n`AWS_DISCOVER_WORKERS` controls concurrent `region_index.json` fetches and\nthose responses are tiny — leave it at 10 regardless of instance size; it\ndoesn't affect memory.\n\nOOM symptom in Cloud Logging looks like: container killed exit code 137,\nwith the last `mem.snapshot` log showing peak_rss climbing past the limit\nduring the parallel-download phase. If that happens, halve `AWS_MAX_WORKERS`.\n\n## Observability\n\n- **Container-level memory**: `run.googleapis.com/container/memory/utilizations` in Cloud Monitoring — the source of truth for \"did this job OOM\". Cloud Run publishes this automatically, no instrumentation needed.\n- **Per-stage RSS in logs**: every run emits structured `mem.snapshot` log lines at start, after discovery, after the parallel download pool, after the LOAD JOB, after the live-table swap, and per `(service, region)` target. Useful when you need to figure out *which stage* of a load drove the peak — the container metric tells you peak but not why.\n  - Sample log line: `mem.snapshot label=download.done rss_mb=412.3 peak_rss_mb=415.7 service=AmazonEC2 region=us-east-1 rows=1234567`\n  - Grep `mem.snapshot` in Cloud Logging Explorer to see the full profile of a run.\n- **Runtime banner**: first log line of every job records Python version and the active `ijson.backend`. If a future image change loses the C wheel and falls back to the pure-Python parser (3-10x slower on EC2 us-east-1), you'll see it at a glance.\n\n## Design notes\n\n- **Per-(service, region) version diff**: the heart of the incremental story. The full AWS dataset is ~5–10 GB but most files are unchanged on any given day.\n- **JSON column for service-specific attributes**: AWS publishes incompatible attribute keys per service. A unified table with `attributes JSON` keeps the query interface uniform (`JSON_VALUE(...)`) while absorbing all variability.\n- **BIGNUMERIC for prices**: AWS routinely publishes 10+ decimal places. `NUMERIC` would round them.\n- **Partition decorator + WRITE_TRUNCATE + carryforward INSERT**: today's partition is composed of (a) fresh rows from changed `(service, region)` files via LOAD JOB and (b) carried-forward rows from yesterday's partition for unchanged pairs.\n- **`CREATE OR REPLACE TABLE` for the live table**: single atomic statement; no rename gymnastics.\n- **`require_partition_filter = TRUE` on history**: protects every consumer from accidental full-table scans without anyone having to think about it.\n- **JSONL over Parquet**: native JSON columns serialize cleanly to NEWLINE_DELIMITED_JSON; price strings survive into `BIGNUMERIC` without float round-trips.\n- **ADC instead of service-account key files**: Cloud Run's identity is the auth.\n- **UUID `run_id`**: BigQuery has no auto-increment; UUID also matches the GCS staging-prefix layout.\n- **Empty-discovery safeguard**: if AWS returns zero targets, the loader refuses to swap the live table (`RuntimeError`) and the failure is recorded in `pricing_runs`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimfanda35%2Faws-pricing-to-bq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimfanda35%2Faws-pricing-to-bq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimfanda35%2Faws-pricing-to-bq/lists"}