{"id":52049266,"url":"https://github.com/jonasbrami/iceberg-ivm","last_synced_at":"2026-08-02T11:30:24.064Z","repository":{"id":357502923,"uuid":"1207139831","full_name":"jonasbrami/iceberg-ivm","owner":"jonasbrami","description":"Metadata-driven incremental view maintenance for Iceberg tables on Trino — bucket-aware aggregating MVs, no full rescans","archived":false,"fork":false,"pushed_at":"2026-06-04T06:05:21.000Z","size":1839,"stargazers_count":2,"open_issues_count":5,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-04T07:22:57.795Z","etag":null,"topics":["analytics","apache-iceberg","data-engineering","database","datalake","iceberg","incremental-view-maintenance","ivm","lakehouse","materialized-view","materialized-views","olap","python","sql","trino"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonasbrami.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-10T16:07:15.000Z","updated_at":"2026-05-20T03:44:18.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jonasbrami/iceberg-ivm","commit_stats":null,"previous_names":["jonasbrami/iceberg-ivm"],"tags_count":20,"template":false,"template_full_name":null,"purl":"pkg:github/jonasbrami/iceberg-ivm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonasbrami%2Ficeberg-ivm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonasbrami%2Ficeberg-ivm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonasbrami%2Ficeberg-ivm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonasbrami%2Ficeberg-ivm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonasbrami","download_url":"https://codeload.github.com/jonasbrami/iceberg-ivm/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonasbrami%2Ficeberg-ivm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36192499,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-08-02T02:00:06.915Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","apache-iceberg","data-engineering","database","datalake","iceberg","incremental-view-maintenance","ivm","lakehouse","materialized-view","materialized-views","olap","python","sql","trino"],"created_at":"2026-08-02T11:30:23.207Z","updated_at":"2026-08-02T11:30:24.050Z","avatar_url":"https://github.com/jonasbrami.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# iceberg-ivm\n\nA simple **Iceberg IVM (incremental view maintenance) controller** that delegates\nall computation to Trino.\n\n\u003e **Want to try it in one command?** See [`./quickstart`](./quickstart) — a\n\u003e fully self-contained Trino + MinIO + Postgres + iceberg-ivm stack with\n\u003e sample data and 8 pre-loaded materialized views.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"docs/screenshots/ui-overview.png\" alt=\"iceberg-ivm UI\" width=\"720\"\u003e\u003c/p\u003e\n\nThe service owns the control plane — view definitions, change detection,\nrefresh scheduling, watermarks — while Trino executes the actual SQL against\nIceberg tables. Change detection runs on Iceberg file-level metadata only;\nwhen source data changes, only the affected time range is recomputed from\ncomplete source data, guaranteeing correct aggregations. Refreshes are atomic\nvia `MERGE INTO`.\n\n**Lightweight by design.** The service itself moves no data and holds no\ntable contents — it only reads Iceberg metadata (`$snapshots`, `$all_entries`)\nand issues SQL to Trino. All data lives in Iceberg; all scans and writes\nhappen inside Trino. Its resident state is a handful of per-view status\ncounters.\n\n**Materialized views are Iceberg tables, which makes them chainable.** Because\neach target is a first-class Iceberg table, it can be the *source* of another\nview. Example: a `trades → 1-minute bars` MV feeding a `1-minute → 1-hour\nbars` MV. The second view only needs metadata from the first's target table,\nso the chain stays cheap: each hop is still a bounded `MERGE` on a snapped\ntime range.\n\n## Why this project exists\n\nThere's a specific, unfilled niche in the Iceberg + Trino ecosystem: **metadata-driven, bucket-aware incremental refresh of aggregating materialized views**. Each existing option falls short in a different way:\n\n| Option | Gap |\n|---|---|\n| **Trino native Iceberg MVs** ([REFRESH MATERIALIZED VIEW](https://trino.io/docs/current/sql/refresh-materialized-view.html)) | Incremental path is append-style — the planner falls back to full refresh when the query aggregates. The open issue [trinodb/trino#18673](https://github.com/trinodb/trino/issues/18673) tracks exactly this gap. |\n| **Hive Iceberg MVs** ([Cloudera blog](https://www.cloudera.com/blog/technical/accelerating-queries-on-iceberg-tables-with-materialized-views.html)) | Incremental rebuild of aggregations exists, but only for **decomposable aggregates** (SUM/MIN/MAX/COUNT/AVG) via additive delta merge — and it **fails on compaction** because it can't tell the new snapshot's intent. |\n| **SQLMesh `INCREMENTAL_BY_TIME_RANGE`** | Bucket-aware by design, but the change signal is a **cron interval**, not Iceberg metadata. Late data needs a manually widened `lookback`. Doesn't read `$snapshots` or file-level statistics. |\n| **dbt-trino incremental models** | The `is_incremental()` predicate is whatever SQL the user writes. No Iceberg snapshot or file-stats integration; no bucket-snapping; no concept of \"compaction is a no-op\". |\n\nThis project combines three things no other option does together:\n\n1. **Read Iceberg file-level min/max stats** (`$all_entries.readable_metrics`) to find the exact time range of new data — never scans the data itself for change detection.\n2. **Snap that range outward to complete GROUP BY bucket boundaries** so re-aggregation produces correct results, regardless of how the source is partitioned.\n3. **Recompute the affected buckets from full source data via `MERGE`** — works for any aggregate (including non-decomposable ones like `min_by`, `max_by`, percentiles), and tolerates compaction by treating `replace` snapshots as no-ops.\n\nThe result is a small, sharply-scoped service: Iceberg sources, `date_trunc`-bucketed aggregations, Trino as the executor. No DAG, no second CLI, no fork of the engine.\n\n## Run it (TL;DR)\n\nAgainst an existing Trino cluster, drop `config.yaml` + `views.yaml` (see\n[Quick start](#quick-start) below) into `./data/` and bring the container up:\n\n```yaml\n# docker-compose.yml\nservices:\n  iceberg-ivm:\n    image: jonasbrami/iceberg-ivm:0.6.0\n    environment:\n      TRINO_URL: http://trino:8080\n      TRINO_USER: iceberg-ivm\n      # TRINO_PASSWORD: …            # only if your Trino requires it\n    volumes:\n      - ./data:/data            # config.yaml, views.yaml, and state.db all live here\n    command: [\"-c\", \"/data/config.yaml\", \"--views\", \"/data/views.yaml\"]\n    ports:\n      - \"8000:8000\"\n```\n\n```bash\ndocker compose up -d \u0026\u0026 open http://localhost:8000\n```\n\nNo Trino cluster handy? **[`./quickstart`](./quickstart)** brings up Trino +\nMinIO + Postgres + iceberg-ivm with sample data and 8 pre-loaded views in one\n`docker compose up`.\n\n## How it works\n\n```mermaid\nsequenceDiagram\n    participant O as iceberg-ivm\n    participant T as Trino\n    participant SRC as Source Table\u003cbr/\u003e(Iceberg metadata)\n    participant TGT as Target MV\u003cbr/\u003e(Iceberg table)\n\n    O-\u003e\u003eT: SELECT snapshot_id FROM source.$snapshots\n    T--\u003e\u003eO: current_snapshot = 200\n\n    Note over O: last_processed = 150 → changed\n\n    O-\u003e\u003eT: SELECT readable_metrics\u003cbr/\u003eFROM source.$all_entries\u003cbr/\u003eWHERE snapshot_id IN (new)\n    T-\u003e\u003eSRC: read manifest files (no data scan)\n    T--\u003e\u003eO: ts ∈ [Apr 9 10:00, Apr 9 15:30]\n\n    Note over O: expand_to_bucket_bounds(\"day\")\u003cbr/\u003e→ [Apr 9, Apr 10)\n\n    O-\u003e\u003eT: MERGE INTO target\u003cbr/\u003eUSING (SELECT ... WHERE ts \u003e= Apr 9 AND ts \u003c Apr 10)\u003cbr/\u003eON keys WHEN MATCHED UPDATE / NOT MATCHED INSERT\n    T-\u003e\u003eSRC: read data files (partition-pruned)\n    T-\u003e\u003eTGT: atomic Iceberg commit\n    T--\u003e\u003eO: done\n\n    O-\u003e\u003eO: state.db: UPDATE view_status SET last_source_snapshot=200\n```\n\nThe five phases — **Detect** (`$snapshots`, ~50 ms), **Measure**\n(`$all_entries` file stats, metadata only), **Snap** (range expansion in\npure Python), **Refresh** (`MERGE INTO` whose range filter Trino pushes\ndown to partition pruning), **Persist** (bookmark to SQLite) — recur on\nevery cycle. On a quiet source the cycle is dominated by the `$snapshots`\npoll; on a busy one, by the `MERGE` itself.\n\n### What a first run looks like\n\nOn the first refresh of a brand-new view you'll see:\n\n1. `discover_columns` (a `PREPARE` + `DESCRIBE OUTPUT`, no data scan) resolves the target column types.\n2. `CREATE TABLE IF NOT EXISTS target (…)` creates the Iceberg table. The target is **unpartitioned by default**; set `target_partitioning` on the view to partition it (e.g. `\"ARRAY['day(minute)']\"`).\n3. The detector sees no `last_source_snapshot` recorded in iceberg-ivm's `state.db` and returns `FULL_REFRESH`.\n4. `execute_full_refresh` runs `DELETE FROM target WHERE true` then `INSERT INTO target SELECT …` (your original query, verbatim — no WHERE injected).\n5. The source snapshot ID is written to `state.db` (`view_status.last_source_snapshot`).\n\nEvery subsequent cycle is incremental unless you delete the bookmark.\n\n### Iceberg metadata: what we read and why\n\nThe service treats Iceberg's metadata tables as a small, structured\nlog it can poll cheaply. It never scans source *data* for change\ndetection — just the metadata describing what data exists.\n\n| Table | What we read | What it tells us |\n|---|---|---|\n| `source.$snapshots` | `(snapshot_id, operation, committed_at)` | Did the source change since our last refresh? And what kind of change? (See operation classification below.) |\n| `source.$all_entries` | `readable_metrics` JSON on files added in new snapshots (filtered by `snapshot_id IN (new)` and `status = 1`) | Per-file column-level min/max bounds. Gives us the tightest `[min_ts, max_ts]` bracket over the new data without reading a single data file. |\n\n**`$snapshots.operation` classification:**\n\n- `append` / `overwrite` — real data changes; drive incremental refresh. `overwrite` is what `MERGE INTO` writes, so this also catches upstream chained MVs whose own refresh is a `MERGE`.\n- `replace` — compaction (files rewritten, no data change). Skipped.\n- `delete` or anything else — rejected loudly. Sources are not allowed to lose data without replacement.\n\n**Why `$all_entries` and not `$partitions`?** An earlier prototype diffed\n`$partitions` between snapshots — but when the GROUP BY granularity is\ncoarser than the source partitioning (e.g. weekly bars from a daily-\npartitioned source), a per-partition filter reads only the changed\npartition and miscomputes the aggregate. File-level min/max is\ngranularity-independent: the detector always gets the real time range\nof the new data regardless of how the source is partitioned.\n\n**Sample `readable_metrics`** (one row per file in `$all_entries`):\n\n```json\n{\n  \"ts\":     {\"lower_bound\": \"2026-04-08T10:00:41+00:00\",\n             \"upper_bound\": \"2026-04-08T15:59:50+00:00\"},\n  \"price\":  {\"lower_bound\": 138.5,  \"upper_bound\": 260.0},\n  \"symbol\": {\"lower_bound\": \"AAPL\", \"upper_bound\": \"TSLA\"}\n}\n```\n\nThe detector pulls the `filter_column`'s `lower_bound` / `upper_bound`\nacross every new file, takes the chronological min of the lowers and\nmax of the uppers, then hands those two timestamps to\n`expand_to_bucket_bounds(…)` (the `date_trunc` inverse — see next\nsection) to produce a bucket-aligned filter range. The range becomes\nliteral `TIMESTAMP` values in the `MERGE`'s WHERE, which Trino's\nplanner pushes straight down to Iceberg partition pruning.\n\n### `date_trunc` and `expand_to_bucket_bounds`: forward and inverse\n\nThe entire incremental refresh correctness depends on one thing: given the\nmin/max timestamps from new files, compute a filter range that covers **every\ncomplete GROUP BY bucket** touched by that data. This is done by inverting\nthe `date_trunc` function used in the query's GROUP BY.\n\n`date_trunc` is a **many-to-one** function: it maps every timestamp within a\nbucket to the same boundary value. `expand_to_bucket_bounds` is its inverse:\nit expands a raw timestamp range outward to the nearest bucket boundaries so\nthe filter captures all rows that belong to any touched bucket.\n\n```mermaid\nflowchart LR\n    subgraph timeline[\"Timeline\"]\n        direction LR\n        h08[\"08:00\"]\n        h09[\"09:00\"]\n        h10[\"10:00\"]\n        h11[\"11:00\"]\n        h12[\"12:00\"]\n    end\n\n    subgraph filestats[\"New files\"]\n        fs[\"min = 09:15\u003cbr/\u003emax = 10:55\"]\n    end\n\n    subgraph snapresult[\"expand_to_bucket_bounds result\"]\n        direction TB\n        floor[\"\u003cb\u003efloor\u003c/b\u003e(09:15) = 09:00\"]\n        ceil[\"\u003cb\u003eceil\u003c/b\u003e(10:55) = 11:00\"]\n    end\n\n    subgraph filter[\"WHERE ts \u003e= 09:00 AND ts \u003c 11:00\"]\n        direction TB\n        covered[\"Covers complete buckets\u003cbr/\u003e\u003cb\u003e[09:00, 10:00)\u003c/b\u003e and \u003cb\u003e[10:00, 11:00)\u003c/b\u003e\u003cbr/\u003e\u003cbr/\u003eAll rows in both buckets are\u003cbr/\u003eincluded in the GROUP BY →\u003cbr/\u003eMERGE produces correct aggregates\"]\n    end\n\n    fs --\u003e floor \u0026 ceil\n    floor --\u003e h09\n    ceil --\u003e h11\n    h09 \u0026 h10 \u0026 h11 --\u003e covered\n\n    style filestats fill:#4a1942,stroke:#6b2d5b,color:#e0e0e0\n    style snapresult fill:#0f3460,stroke:#16213e,color:#e0e0e0\n    style filter fill:#2d6a4f,stroke:#40916c,color:#e0e0e0\n```\n\nThe two operations mirror each other exactly:\n\n| | `date_trunc('hour', ts)` | `expand_to_bucket_bounds('hour')` |\n|---|---|---|\n| **Direction** | timestamp → bucket start | timestamp range → bucket-aligned range |\n| **Operation** | floor to `:00:00` | floor min to `:00:00`, ceil max to next `:00:00` |\n| **Used in** | `GROUP BY` (query) | `WHERE` filter (iceberg-ivm) |\n| **Guarantees** | rows are grouped by hour | filter covers complete hours |\n\nThis is why only simple `date_trunc` is allowed: for any `date_trunc('X', col)`,\nthe inverse is trivially computable by `expand_to_bucket_bounds('X')`. Complex expressions\n(e.g. 5-minute bars via arithmetic) break this — the bucket width can't be\nreliably inferred, and the inverse would produce a too-narrow filter that\ncorrupts aggregates.\n\n## Requirements\n\n- **Trino** with the Iceberg connector enabled — recent versions are recommended for `$all_entries` / `readable_metrics`.\n- **Iceberg catalog supporting writes from Trino** — Hive Metastore, REST, Nessie, Polaris, Snowflake Open Catalog, or JDBC.\n- **Trino user** with the following privileges:\n  - `SELECT` on every source table (plus its `$snapshots`, `$all_entries` metadata tables — these inherit from the base-table grant in Trino's Iceberg connector).\n  - `CREATE TABLE` on the target schema — iceberg-ivm creates the target on first refresh.\n  - `SELECT`, `INSERT`, `DELETE`, `UPDATE` on every target table — needed for `MERGE` (incremental) and `DELETE + INSERT` (full refresh).\n- **Iceberg column statistics** on the source `filter_column` — the detector reads `readable_metrics` from `$all_entries`. Statistics are on by default for Trino and Spark ≥ 3.5 writers.\n- **Python ≥ 3.12** — only when running from source (the Docker image does not require this on the host).\n\n## Quick start\n\n### 1. Set Trino credentials (environment variables)\n\nTrino URL, user, and password are read **only** from the environment. They\nnever appear in `config.yaml`, so secrets stay out of the repo and\nper-environment deployments only need to set a few env vars.\n\n| Variable | Required | Notes |\n|---|---|---|\n| `TRINO_URL` | yes | Coordinator URL the orchestrator uses to reach Trino, e.g. `http://trino:8080` or `https://trino.prod:8443`. Scheme determines `http` vs `https`. |\n| `TRINO_USER` | yes | Trino username. |\n| `TRINO_PASSWORD` | no | BasicAuth password. Omit (or leave empty) for anonymous access — e.g. the local dev compose stack. |\n| `TRINO_PUBLIC_URL` | no | Browser-reachable Trino URL used to rewrite the per-query `info_uri` deep-links in the UI. Defaults to `TRINO_URL`. Set when the orchestrator and the user's browser reach Trino over different hostnames (e.g. `http://trino:8080` inside docker vs `http://localhost:28080` from the host). |\n\nThe service refuses to start if `TRINO_URL` or `TRINO_USER` is missing.\n\n### 2. Create two YAML files\n\n**`config.yaml`** — server settings + per-deployment Trino *catalog* and\n*schema* (not secrets; see [`config.yaml.example`](config.yaml.example)):\n\n```yaml\nserver:\n  port: 8000\n  config_reload_interval_seconds: 30\ntrino:\n  catalog: iceberg\n  schema: analytics\n```\n\n**`views.yaml`** — the views to maintain. Write each `query` *exactly* as you\nwould after `CREATE MATERIALIZED VIEW … AS`:\n\n```yaml\nviews:\n  - name: ohlcv_1m\n    query: |\n      SELECT\n        symbol,\n        date_trunc('minute', ts) AS minute_ts,\n        min_by(price, ts) AS open, max(price) AS high,\n        min(price) AS low,        max_by(price, ts) AS close,\n        sum(quantity) AS volume,  count(*) AS trade_count\n      FROM iceberg.market_data.trades\n      GROUP BY symbol, date_trunc('minute', ts)\n    refresh_interval_seconds: 30\n```\n\nThe service parses each query and derives `source_table`, `filter_column`,\n`filter_granularity`, and `merge_keys` from it. At refresh time the time-range\nWHERE predicate is AST-injected automatically — there is no `{range_filter}`\nplaceholder. Column types are auto-discovered via `DESCRIBE OUTPUT` and the\ntarget table is created on first run.\n\nViews can also be managed interactively from the web UI.\n\n### 3. Run it\n\nInstall dependencies, export the credential env vars, and start the service:\n\n```bash\nuv sync\nexport TRINO_URL=http://localhost:8080\nexport TRINO_USER=iceberg-ivm\n# export TRINO_PASSWORD=…            # only if your Trino requires it\nuv run iceberg-ivm -c config.yaml --views views.yaml\n# Web UI:  http://localhost:8000\n# Metrics: http://localhost:8000/metrics\n```\n\n**CLI flags:**\n\n| Flag | Default | Description |\n|---|---|---|\n| `-c`, `--config` | `config.yaml` | Path to the server + trino catalog/schema config |\n| `--views` | `views.yaml` | Path to the views file (empty if absent) |\n| `-v`, `--verbose` | off | Enable DEBUG logging |\n\nThe server port comes from `server.port` in `config.yaml`. Both files are\nhot-reloaded on mtime change at `server.config_reload_interval_seconds` (default\n30s) — no restart needed when adding/editing views.\n\n### Running against a local Trino stack\n\nFor a turnkey demo, see **[`./quickstart`](./quickstart)** — one\n`docker compose up`, sample data seeded, 8 views pre-loaded.\n\nFor development against the codebase, `tests/docker-compose.yml` brings up\njust Trino + MinIO + Postgres:\n\n```bash\ncd tests \u0026\u0026 docker compose up -d trino\n# Trino UI: http://localhost:18080\n# Seed sample data + generate config/views:\ncd ..\nuv run --with trino python tests/seed_data.py\n\n# The compose Trino accepts any user without a password, so TRINO_PASSWORD\n# stays unset:\nexport TRINO_URL=http://localhost:18080\nexport TRINO_USER=demo\nuv run iceberg-ivm -c config.yaml --views views.yaml\n```\n\n## Using it\n\n### The web UI\n\nOpen http://localhost:8000 after starting the service. You get a single-page app listing every configured view:\n\n- **Header** — view count, poll cadence, a **New View** button that opens the create-view modal.\n- **View cards** — one per view, showing:\n  - name and a status badge (`pending`, `idle`, `incremental`, `full`, `error`)\n  - `source → target` qualified table names\n  - stats row: total `Refreshes`, last `Duration`, polled `Interval`, `Last` refresh timestamp, cumulative `Errors`\n  - **Refresh / Edit / Delete** buttons\n  - a collapsible **query** panel with the raw SQL\n  - a collapsible **recent queries (N)** panel with one row per refresh-time query (`merge`, `chunk_merge`, or `maintenance_\u003cop\u003e`) — click any row to open that query in the Trino UI\n- **Live query parsing** — while you type in the create/edit modal, a debounced panel under the query textarea shows the parser's derived `source`, `filter col`, `granularity`, and `merge keys`, or a red parse error.\n- **Refresh toast** — clicking Refresh pops a small bottom-right toast reporting the action (`no changes` / `incremental refresh` / `full refresh` / error message).\n\n### Managing views — two paths\n\nBoth write to the same `views.yaml`, so you can mix freely:\n\n| Path | When | Notes |\n|---|---|---|\n| Edit `views.yaml` on disk | Version-controlled deploys; bulk changes | Hot-reloaded on mtime change every `config_reload_interval_seconds` (default 30s) — no restart needed. |\n| Web UI / REST API | Exploratory / ops workflows | `POST /api/views`, `DELETE /api/views/{name}`, `POST /api/views/{name}/refresh`. Writes to `views.yaml` atomically. |\n\n### How the refresh loop actually runs\n\nOne asyncio task ticks every **1 second**. On each tick:\n\n- If `config_reload_interval_seconds` has elapsed since the last reload, re-read `config.yaml` + `views.yaml`.\n- For each configured view, if `refresh_interval_seconds` has elapsed since its last refresh, run one refresh cycle.\n\nRefreshes are **sequential** — one view at a time, no pool, no queue. A single slow refresh delays the next tick's scheduling; plan `refresh_interval_seconds` accordingly.\n\n(For a step-by-step trace of a brand-new view's first refresh, see [What a first run looks like](#what-a-first-run-looks-like) above.)\n\n### Day-2 operations\n\n| Task | How |\n|---|---|\n| Add / edit a view | Edit `views.yaml` (or use UI / API). Takes effect at next config reload. |\n| Delete a view | UI → Delete, or `DELETE /api/views/{name}`. Removes it from the schedule **only** — the target Iceberg table is untouched. Drop it separately if you want the data gone. |\n| Force a full refresh | Delete the view via the UI / `DELETE /api/views/{name}` and recreate it (this clears the SQLite bookmark), then `DROP TABLE \u003ctarget\u003e` if you also want the data gone. Just dropping the target by itself will not full-refresh — the next tick will incremental-refresh against an empty table. |\n| Restart iceberg-ivm | Safe. The bookmark lives in `state.db` (mounted alongside `views.yaml`); restart picks up where it left off. |\n| Manual refresh | UI → Refresh, or `POST /api/views/{name}/refresh`. Runs one cycle synchronously and returns the action taken. |\n| Health check | `curl http://localhost:8000/health` → `{\"status\":\"ok\",\"views\":N}` |\n\n### Deployment notes\n\n- **Single-instance.** iceberg-ivm is not HA. One process per target catalog/schema. Two instances against the same targets would race on `ALTER TABLE SET PROPERTIES` and produce conflicting MERGEs.\n- **Crash-safe.** All correctness state is the SQLite bookmark on disk plus committed Iceberg snapshots. Kill/restart picks up exactly where it left off — provided `state.db` is on a persistent volume (next bullet).\n- **Persist the SQLite state DB.** Backed by a SQLite file (`server.state_db_path`, default `state.db` next to `views.yaml`). It also holds the per-view source-snapshot bookmark (`view_status.last_source_snapshot`), so losing it forces every view to full-refresh on next start (and the UI's \"recent queries\" history disappears too). If you run in docker, mount a volume at the directory containing it — e.g. `-v ivm-state:/data` to match the compose example above.\n- **What to monitor** (all labels `view`):\n  - `rate(mv_refresh_errors_total[5m]) \u003e 0` — refresh is failing.\n  - `time() - mv_refresh_last_success_timestamp \u003e 3 * refresh_interval_seconds` — view is stalled.\n  - `histogram_quantile(0.95, rate(mv_refresh_duration_seconds_bucket[10m]))` — refresh tail latency; unexpected growth usually means a partition scan you didn't plan for.\n\n### Configuration reference\n\n| Field | Required | Description |\n|---|---|---|\n| `name` | yes | Unique view name |\n| `query` | yes | The full SELECT — exactly what you would write after `CREATE MATERIALIZED VIEW … AS`. `source_table`, `filter_column`, `filter_granularity`, and `merge_keys` are derived from this |\n| `target_table` | no | Defaults to `{catalog}.{schema}.{name}` |\n| `target_partitioning` | no | Iceberg `ARRAY[...]` string for the target table's partitioning (e.g. `\"ARRAY['day(minute)']\"`). Unpartitioned if omitted — iceberg-ivm does not auto-inherit from the source, because the source's partition column is typically consumed by `date_trunc(...)` and no longer exists on the target (see issue #22). |\n| `refresh_interval_seconds` | no | Defaults to 60 |\n\n### API\n\n| Endpoint | Method | Description |\n|---|---|---|\n| `/` | GET | Web UI |\n| `/api/views` | GET | List all views with status |\n| `/api/views` | POST | Create a new view |\n| `/api/views/{name}` | DELETE | Remove a view |\n| `/api/views/{name}/refresh` | POST | Trigger manual refresh |\n| `/metrics` | GET | Prometheus metrics |\n| `/health` | GET | Health check |\n\n### Prometheus metrics\n\n| Metric | Type | Labels |\n|---|---|---|\n| `mv_refresh_total` | counter | view, type(full/incremental/skip) |\n| `mv_refresh_duration_seconds` | histogram | view |\n| `mv_refresh_last_success_timestamp` | gauge | view |\n| `mv_refresh_errors_total` | counter | view |\n| `mv_config_reload_total` | counter | |\n| `mv_views_configured` | gauge | |\n\n## Cross-partition GROUP BY\n\nThe tool correctly handles GROUP BY expressions coarser than the source\npartition granularity (e.g. weekly bars from a daily-partitioned table).\n\n```mermaid\ngraph LR\n    subgraph \"Source: trades (partitioned by day)\"\n        D1[\"day=Apr 6 (Mon)\"]\n        D2[\"day=Apr 7 (Tue)\"]\n        D3[\"day=Apr 8 (Wed) ← NEW\"]\n    end\n\n    subgraph IVM[\"iceberg-ivm\"]\n        FS[\"File stats: ts ∈ [Apr 8 10:00, Apr 8 15:30]\"]\n        SR[\"expand_to_bucket_bounds('week'): [Apr 6, Apr 13)\"]\n        FS --\u003e SR\n    end\n\n    subgraph \"MERGE reads ALL 3 days\"\n        Q[\"WHERE ts \u003e= Apr 6\u003cbr/\u003eAND ts \u003c Apr 13\"]\n    end\n\n    subgraph \"Target: weekly bars\"\n        W[\"week=Apr 6\u003cbr/\u003evol=350 (Mon+Tue+Wed)\"]\n    end\n\n    D3 --\u003e|\"readable_metrics\"| FS\n    SR --\u003e Q\n    D1 --\u003e Q\n    D2 --\u003e Q\n    D3 --\u003e Q\n    Q --\u003e W\n\n    style D3 fill:#2d6a4f,stroke:#40916c\n    style W fill:#2d6a4f,stroke:#40916c\n```\n\nThe inferred `filter_granularity` (`week`) snaps the file-stats range to complete\nweek boundaries, so the MERGE query reads Mon+Tue+Wed and produces a correct\nweekly bar.\n\n## Query parsing\n\nAt config-load time iceberg-ivm parses every view query with an AST-based\nparser (`sqlparse`). It derives:\n\n- `source_table` — from the FROM clause\n- `filter_column` — the bare column inside `date_trunc('X', col)`\n- `filter_granularity` — the `'X'` literal (one of `minute`, `hour`, `day`,\n  `week`, `month`, `quarter`, `year`)\n- `merge_keys` — resolved from the GROUP BY list against the projection\n  (positional `GROUP BY 1, 2` refs are handled too)\n\nAt refresh time iceberg-ivm AST-injects the time-range `WHERE` predicate\ndirectly into the query:\n\n```sql\n-- operator writes:\nSELECT symbol, date_trunc('week', ts) AS week, sum(qty) AS volume\nFROM iceberg.md.trades\nWHERE color = 'red'\nGROUP BY 1, 2\n\n-- iceberg-ivm runs:\nMERGE INTO iceberg.md.trades_weekly AS t USING (\n  SELECT symbol, date_trunc('week', ts) AS week, sum(qty) AS volume\n  FROM iceberg.md.trades\n  WHERE color = 'red'\n    AND ts \u003e= TIMESTAMP '2026-04-06 00:00:00.000000 UTC'\n    AND ts \u003c TIMESTAMP '2026-04-13 00:00:00.000000 UTC'\n  GROUP BY 1, 2\n) AS s ON t.symbol = s.symbol AND t.week = s.week …\n```\n\n## Example queries\n\n### ✅ Supported\n\n**Minute-level OHLCV bars:**\n```sql\nSELECT\n  symbol,\n  date_trunc('minute', ts) AS minute_ts,\n  min_by(price, ts) AS open, max(price) AS high,\n  min(price) AS low,         max_by(price, ts) AS close,\n  sum(quantity) AS volume,   count(*) AS trade_count\nFROM iceberg.market_data.trades\nGROUP BY symbol, date_trunc('minute', ts)\n```\n\n**Weekly bars from a daily-partitioned source** — the detector expands the\nfile-stats range to full week boundaries:\n```sql\nSELECT symbol, date_trunc('week', ts) AS week, sum(quantity) AS volume\nFROM iceberg.market_data.trades\nGROUP BY 1, 2\n```\n\n**Pre-filtered view** (your `WHERE` is preserved; iceberg-ivm `AND`s the\ntime-range predicate onto it):\n```sql\nSELECT symbol, date_trunc('hour', ts) AS hour, count(*) AS c\nFROM iceberg.md.trades\nWHERE status = 'settled'\nGROUP BY 1, 2\n```\n\n**Positional GROUP BY** (resolves to projection aliases — merge keys become\n`['symbol', 'day']`):\n```sql\nSELECT symbol, date_trunc('day', ts) AS day, sum(qty) AS v\nFROM iceberg.md.trades\nGROUP BY 1, 2\n```\n\n**Chained MV** (a view whose source is another view's target — since every\ntarget is an Iceberg table, it's a normal source for the next hop). The\nupstream `ohlcv_1m` example above projects its bucket column as `minute_ts`,\nso the chained query reads `minute_ts`:\n```sql\nSELECT symbol, date_trunc('hour', minute_ts) AS hour_bucket,\n       min_by(open, minute_ts) AS open, max(high) AS high,\n       min(low) AS low,       max_by(close, minute_ts) AS close,\n       sum(volume) AS volume, sum(trade_count) AS trade_count\nFROM iceberg.analytics.ohlcv_1m\nGROUP BY 1, 2\n```\nNote: avoid naming the upstream MV's time column with a SQL reserved word\n(`minute`, `hour`, `day`, `week`, `month`, `quarter`, `year`) — the parser\nrejects bare reserved words as the second arg of `date_trunc(...)`. Use a\nnon-reserved alias like `minute_ts` or `bucket` in the upstream view.\n\n### ❌ Not supported — rejected at config load\n\n**Joins** (change detection only watches one source — other tables' changes\nwould be silently missed):\n```sql\nSELECT t.symbol, date_trunc('day', t.ts) AS day, count(*)\nFROM trades t JOIN symbols s ON t.symbol = s.ticker\nGROUP BY 1, 2\n```\n\n**CTEs / `WITH` clauses:**\n```sql\nWITH filtered AS (SELECT * FROM trades WHERE price \u003e 0)\nSELECT date_trunc('day', ts) AS day, count(*)\nFROM filtered\nGROUP BY 1\n```\n\n**Subquery in FROM:**\n```sql\nSELECT date_trunc('day', ts) AS day, count(*)\nFROM (SELECT ts FROM trades WHERE price \u003e 0) s\nGROUP BY 1\n```\n\n**Set operations** (`UNION` / `INTERSECT` / `EXCEPT`):\n```sql\nSELECT date_trunc('day', ts) AS day FROM trades_us\nUNION ALL\nSELECT date_trunc('day', ts) AS day FROM trades_eu\n```\n\n**`date_trunc` wrapped in arithmetic** — the canonical 5-minute-bars mistake.\nThe bucket width can't be reliably inferred, so the inverse would produce a\ntoo-narrow filter that corrupts aggregates:\n```sql\nSELECT date_trunc('minute', ts) - (extract(minute FROM ts) % 5) * INTERVAL '1' MINUTE AS bar\nFROM trades\nGROUP BY 1\n```\n\n**Multiple distinct granularities in one view** — a view has a single\ngranularity:\n```sql\nSELECT date_trunc('day', ts) AS d, date_trunc('hour', ts) AS h, count(*)\nFROM trades\nGROUP BY 1, 2\n```\n\n**Computed projection without an alias** — the target-table column would get\nan auto-generated name like `_col0`:\n```sql\nSELECT date_trunc('day', ts), count(*)\nFROM trades\nGROUP BY 1\n```\n\n**No `date_trunc` / no `GROUP BY`** — the correctness model requires both:\n```sql\nSELECT symbol, price FROM trades\n```\n\n## Limitations\n\n### Query shape\n\nThe query must be a `SELECT … GROUP BY` over a **single source table**. The\nparser enforces this at load time and rejects anything else with a clear error.\n\n### Rejected at parse time\n\nJoins, CTEs (`WITH …`), subqueries in FROM, set operations (`UNION` /\n`INTERSECT` / `EXCEPT`), queries without `GROUP BY`, `date_trunc` wrapped in\narithmetic (5-minute-bar style), and computed projections without an alias\nare all rejected at config load with a clear error. See\n[❌ Not supported](#-not-supported--rejected-at-config-load) above for the\nSQL shapes and the reasoning behind each.\n\n### Rejected at refresh time\n\n- **Source pure-delete snapshots** — detected via `$snapshots`, raise\n  `UnexpectedOperationError`. Data may not disappear without replacement.\n  (`MERGE INTO`, which Iceberg labels as an `overwrite` snapshot, *is*\n  supported — that's how chained MVs work.)\n- **Missing column stats** — if the source writer disables Iceberg column\n  statistics, the detector can't determine the affected range and raises\n  `MissingFilterColumnError`.\n\n### Assumptions\n\n- **Sources don't lose data without replacement.** The detector accepts\n  three Iceberg snapshot operations: `append` and `overwrite` are treated\n  as real data changes and drive incremental refresh (`overwrite` is what\n  `MERGE INTO` writes — both for raw sources mutated by external MERGEs\n  and, importantly, for upstream chained MVs whose own refresh is a\n  MERGE); `replace` (compaction) is skipped — files were rewritten but no\n  data changed. Any other operation (`delete`, or some unknown new op)\n  fails loudly.\n- **UTC session timezone.** iceberg-ivm pins every Trino session to\n  `UTC` so that `date_trunc('day' | 'week' | …, ts)` on `TIMESTAMP WITH\n  TIME ZONE` columns aligns with the Python-side `expand_to_bucket_bounds` bucket math.\n  Without this pin, a non-UTC session would produce bucket boundaries\n  that disagree with the computed filter range and silently corrupt\n  incremental aggregates. See [DESIGN.md](DESIGN.md#timezone-assumption).\n- **Iceberg v2** (required for MERGE)\n- Source files have column-level min/max statistics (default in Parquet)\n\n## Tests\n\n```bash\n# Unit tests only\nuv run pytest tests/unit/ -v\n\n# Full suite (requires docker compose)\ncd tests \u0026\u0026 docker compose up -d\ncd .. \u0026\u0026 uv run pytest tests/ -v\ncd tests \u0026\u0026 docker compose down -v\n```\n\n## Contributing\n\n```bash\nuv sync\nuv run pre-commit install --install-hooks\nuv run pre-commit install --hook-type pre-push\n```\n\nRuff lint + format runs on `git commit`; unit tests run on `git push`. CI enforces the same lint/format checks.\n\n## Project structure\n\n```\nsrc/iceberg_ivm/\n    config.py        -- YAML config loading, saving, validation\n    detector.py      -- $snapshots + $all_entries file stats + expand_to_bucket_bounds()\n    executor.py      -- MERGE SQL generation + execution\n    introspect.py    -- DESCRIBE OUTPUT, EXPLAIN IO, SHOW CREATE TABLE\n    query_parser.py  -- AST-based parser for view queries (source/filter/granularity/keys)\n    query_history.py -- SQLite-backed view_status / query history / source-snapshot bookmark\n    server.py        -- FastAPI: web UI, REST API, Prometheus, refresh loop\n    cli.py           -- Entry point, starts uvicorn\n    static/\n        index.html   -- Web UI (Tailwind CSS + Alpine.js)\ntests/\n    unit/            -- 262 tests (mock cursors, FastAPI test client)\n    integration/     -- 17 e2e tests (Trino + Iceberg + MinIO via docker compose)\n```\n\n---\n\n*Designed and implemented through a conversation between a human prompter\nand Claude Code. See [DESIGN.md](DESIGN.md) for the full design rationale\nand conversation context.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonasbrami%2Ficeberg-ivm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonasbrami%2Ficeberg-ivm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonasbrami%2Ficeberg-ivm/lists"}