{"id":51700500,"url":"https://github.com/ccyrene/trace-weaver","last_synced_at":"2026-07-16T10:05:35.667Z","repository":{"id":364847582,"uuid":"1269363289","full_name":"ccyrene/trace-weaver","owner":"ccyrene","description":"Static column-level data-lineage compiler for annotated Airflow DAGs — declared vs inferred provenance; exports to OpenMetadata / OpenLineage / Graphviz.","archived":false,"fork":false,"pushed_at":"2026-07-13T11:51:06.000Z","size":161,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-13T13:19:35.563Z","etag":null,"topics":["airflow","column-level-lineage","data-engineering","data-lineage","metadata","openlineage","openmetadata","rust"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ccyrene.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-14T16:10:53.000Z","updated_at":"2026-07-13T11:51:05.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ccyrene/trace-weaver","commit_stats":null,"previous_names":["ccyrene/trace-weaver"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/ccyrene/trace-weaver","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccyrene%2Ftrace-weaver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccyrene%2Ftrace-weaver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccyrene%2Ftrace-weaver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccyrene%2Ftrace-weaver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ccyrene","download_url":"https://codeload.github.com/ccyrene/trace-weaver/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccyrene%2Ftrace-weaver/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35539570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-16T02:00:06.687Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","column-level-lineage","data-engineering","data-lineage","metadata","openlineage","openmetadata","rust"],"created_at":"2026-07-16T10:05:34.608Z","updated_at":"2026-07-16T10:05:35.658Z","avatar_url":"https://github.com/ccyrene.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# trace-weaver\n\n**A static, column-level data-lineage compiler for Apache Airflow DAGs.**\n\ntrace-weaver reads your DAG code **without executing it**, recovers\ncolumn-level lineage, and compiles it into a single intermediate document\n(`*.weave.json`) that it can export to **OpenMetadata**, **OpenLineage** (Marquez\n\u0026 friends), or a **Graphviz DOT** graph.\n\n```\nDAG code (.py)  ──scan──▶  weave IR (.weave.json)  ──export──▶  OpenMetadata\n                  │                                              OpenLineage\n              (no exec)                                          Graphviz DOT\n```\n\nThe compiler is written in Rust (a Cargo workspace); a tiny, dependency-free\n**Python authoring SDK** (`trace_weaver`) is included for the cases where you\nwant to *declare* lineage by hand. The SDK is optional — see below.\n\n---\n\n## Highlights\n\n- **Works on plain Airflow DAGs with zero annotation.** trace-weaver discovers\n  tasks from raw operators — `PythonOperator(python_callable=…)` and SQL\n  operators (`sql=`/`query=`) — then reads the embedded SQL and the pandas/Spark\n  function body to derive column lineage on its own.\n- **Auto-infer first, declare only the tail.** Column lineage is extracted from\n  SQL (parsed) and from pandas/Spark dataflow (statically traced). You annotate\n  only the spots the analyzer genuinely cannot read.\n- **Provenance is first-class.** Every dataset, job, edge and column mapping\n  records whether it was **declared** by a human or **inferred** by the compiler\n  (and how). Exporters visibly tag inferred lineage so a guess is never mistaken\n  for a hand-declared fact.\n- **Nothing is silently dropped.** A pattern the analyzer can't trace becomes a\n  precise `W_OPAQUE_COLUMN` diagnostic pointing at the line — not a missing edge.\n- **Static \u0026 reproducible.** No DAG is ever run; no network call is made during a\n  scan. The compiled document is deterministic.\n\n---\n\n## Quick start\n\n```bash\n# Build the compiler (produces target/release/trace-weaver).\ncargo build --release\n\n# Scan a plain Airflow DAG (no @tw needed). --service/--database/--schema expand\n# bare table names into OpenMetadata FQNs (service.database.schema.table).\ntrace-weaver scan examples/dags \\\n  --service \"Test Database\" --database poc_db --schema public \\\n  -o build/lineage.weave.json --strict\n\n# Inspect / validate the compiled document.\ntrace-weaver validate build/lineage.weave.json\n\n# Export.\ntrace-weaver export --to openmetadata --dry-run build/lineage.weave.json\ntrace-weaver export --to openlineage  -o build/events.json build/lineage.weave.json\ntrace-weaver graph  build/lineage.weave.json -o build/lineage.dot   # → DOT\n```\n\n[`examples/dags/medallion.py`](examples/dags/medallion.py) is a fully\nun-annotated `landing → bronze → silver → gold` DAG. Scanning it yields\n**4 datasets · 3 edges · 17 column mappings · 0 diagnostics** — bronze \u0026 gold\ninferred from SQL, silver inferred from the pandas body.\n\n---\n\n## How it works\n\nA Cargo workspace, one concern per crate:\n\n| Crate                  | Responsibility |\n|------------------------|----------------|\n| `trace-weaver-core`    | The weave IR (`WeaveDocument` / `Dataset` / `Job` / `Edge` / `ColumnEdge`), the `Origin` provenance type, graph helpers, and structural validation. Aligned with the OpenLineage spec. |\n| `trace-weaver-scan`    | DAG code → IR. AST parsing (`python.rs`), SQL column lineage (`sql.rs`), pandas/Spark dataflow analysis (`dataflow.rs`), same-name gap-fill (`infer.rs`). |\n| `trace-weaver-export`  | IR → catalogues: `openmetadata.rs`, `openlineage.rs`, `dot.rs`. One `export(target, doc, \u0026cfg)` entry point. |\n| `trace-weaver-cli`     | The `trace-weaver` binary: `scan` / `validate` / `export` / `graph`. |\n\n**Scan pipeline** (per file, literals only — no code is executed):\n\n1. Parse the source AST (`rustpython-parser`).\n2. **Discover tasks.** (A) `@tw.task` / `@tw.sql` decorated functions, and\n   (B) decorator-free raw Airflow operators. A `@tw` decorator on a function\n   *overrides* its decorator-free discovery (deduped by name).\n3. For SQL steps, parse the query for column lineage (`inferred from SQL`).\n4. For pandas/Spark steps, statically trace the function body\n   (`inferred from code`); fill remaining same-name gaps by identity.\n5. Finalize: back-fill dataset schemas from observed columns, flag any\n   data-producing edge that ended up with no column lineage\n   (`W_NO_COLUMN_LINEAGE`), and run structural validation.\n\n---\n\n## Provenance: declared vs. inferred\n\nThe compiler treats your declarations as the source of truth and only *fills\ngaps*. Every element carries an `Origin`:\n\n| Tier            | Source | ~confidence | When |\n|-----------------|--------|:-----------:|------|\n| **Declared**    | a human, in a `@tw.task(...)` `column_map` | — (authoritative) | never overwritten |\n| **Inferred from SQL**  | parsing an embedded SQL query  | `0.85` | `engine=\"sql\"` + a query is present |\n| **Inferred from code** | statically tracing a pandas/Spark body | `0.70` | `df[\"c\"]=…`, `withColumn`, `select`, `groupby/agg`, `rename`, `expr(\"…\")`, … |\n| _(identity gap-fill)_  | conservative same-name passthrough     | `0.40` | a target column sharing a name with an undeclared input column |\n\nExporters append `(inferred from SQL)` / `(inferred from code)` to inferred\nlabels, so a human reading the lineage can always tell declared truth from a\nmachine guess.\n\nOn a **plain (un-annotated) DAG**, the lineage is inferred top to bottom: not\nonly the columns but the **datasets, jobs and edges themselves** are tagged\ninferred — because their very existence was recovered by static analysis, not\nhand-declared. A `@tw`-declared task, by contrast, keeps a `declared` origin and\nonly its undeclared columns are inferred.\n\n---\n\n## What the scanner reads automatically (and what it can't)\n\n**Auto (no annotation):** SQL operators; `read_sql`/`to_sql`/`spark.read.table`/\n`saveAsTable`; `df[\"c\"]=expr`, arithmetic / cast / `.map` / comparisons; fan-in;\n`rename` / `assign` / `df[[...]]`; `withColumn` / `withColumnRenamed` / `select`\n/ `selectExpr` / `expr()`; `groupby`/`agg`; literal-list loops; inline\n`apply(lambda …)`; and column names from a **local string constant**\n(`col = \"amount_usd\"; out[col] = …`).\n\n**Auto (v0.5):** **f-string SQL** — `spark.sql(f\"INSERT INTO {table} SELECT\nCAST(\\`X\\` AS STRING), … FROM staging\")` folds its literal / resolvable\ninterpolations and substitutes a placeholder for genuinely runtime ones (a\nfunction param like `{table}`), so the static `SELECT`/`CAST` column list still\nmaps (`INSERT OVERWRITE` and column-list-less `INSERT` included);\n`createOrReplaceTempView(\"v\")` binds `v` to its frame so a later\n`spark.sql(\"… FROM v\")` chains through it; **undecorated transform functions**\n(`def run(spark, src_path, …)` dispatched via `importlib`) are traced for\n**column lineage only** — they contribute column-carrying edges but **no job**,\nso they never enter the lineage gate's task count; and a `@lineage`-declared\nfunction's body is now also traced, so inferable column mappings attach beneath\nits declared (HIGH-confidence) datasets.\n\n**Opaque → `W_OPAQUE_COLUMN` (declare or refactor):** column names from a\n**runtime** value (`out[x]=…` where `x` isn't a compile-time constant;\n`df.columns=[...]`; `pivot`/`melt`/`explode`); named UDFs / `.rdd` / `.pipe`;\njoin columns that aren't keys; chain-reassignment loops\n(`for c in cols: df = df.withColumn(c, …)`).\n\nThe do/don't guide with per-case rewrites is in\n[`TRACEABLE_PIPELINES.md`](TRACEABLE_PIPELINES.md).\n\n---\n\n## CLI reference\n\n```text\ntrace-weaver scan \u003cpath\u003e [-o out.weave.json] [--namespace NS]\n                         [--service S --database D --schema SC]\n                         [--no-sql-infer] [--no-code-infer] [--strict]\ntrace-weaver validate \u003cdoc.weave.json\u003e [--strict]\ntrace-weaver export --to \u003copenmetadata|openlineage|dot\u003e \u003cdoc.weave.json\u003e\n                    [--dry-run] [-o out] [--om-host URL]\n                    [--om-token T | --om-token-file PATH] [--om-service S]\n                    [--ol-producer URI] [--timeout S] [--retries N]\n                    [--fail-on-partial]\ntrace-weaver graph \u003cdoc.weave.json\u003e [-o out.dot]      # shortcut for export --to dot\ntrace-weaver gate --repo-path P [--git-ref R]\n                  [--min-task-coverage F] [--min-high-confidence F]\n                  [--min-annotation-coverage F] [--format text|json]\n```\n\n- **`scan`** walks a file or directory of `.py` DAGs and writes the weave IR.\n  `--service/--database/--schema` supply OpenMetadata FQN parts for DAGs without\n  a `tw.configure(...)`. `--no-sql-infer` / `--no-code-infer` disable a tier.\n- **`validate`** runs structural checks (unknown datasets, duplicate names,\n  off-endpoint columns, …).\n- **`export`** sends the document to a catalogue. `--dry-run` builds the request\n  bodies / artifact and performs **no** network I/O.\n- **`gate`** scans a repo and fails CI when lineage coverage/confidence falls\n  below a threshold (see [CI lineage gate](#ci-lineage-gate) below).\n- **Exit codes:** `0` success; `1` on error or a tripped `--strict` /\n  `--fail-on-partial` gate.\n- **OpenMetadata auth:** the ingestion-bot JWT is read from `--om-token-file`,\n  then `--om-token`, then the `OPENMETADATA_BOT_TOKEN` environment variable.\n  Prefer the file or env var — a raw `--om-token` is visible in your shell\n  history.\n\n### Export targets\n\n- **`openmetadata`** (`om`) — PUTs `add_lineage` edges (with per-column\n  `columnsLineage` and `function` labels) to `{host}/v1/lineage`; prints the\n  request bodies under `--dry-run`.\n- **`openlineage`** (`ol`) — emits `COMPLETE` `RunEvent`s carrying the\n  `columnLineage` dataset facet, as a JSON array (file-only).\n- **`dot`** (`graphviz`) — a Graphviz DOT graph; edges with any inferred lineage\n  are drawn dashed/orange.\n\n### CI lineage gate\n\n`trace-weaver gate` turns a scan into a pass/fail check so a PR can be blocked when\nlineage regresses. It scans `--repo-path` (optionally as of `--git-ref`) and\ncompares three metrics against thresholds:\n\n- **`task_coverage`** — fraction of tasks (jobs) that carry at least one lineage\n  edge. A declared self-loop (`input == output`) is a real edge and counts here.\n- **`annotation_coverage`** — fraction of tasks that carry an explicit\n  trace-weaver decorator (`@tw.task` / `@tw.sql` / `@lineage`, **bare or\n  called**), i.e. `tasks_annotated / tasks_total`. A bare `@lineage` marker or\n  an inputs-only declaration is annotated even though it yields no edge.\n- **`high_confidence_fraction`** — fraction of edges that are **declared**\n  (high confidence) rather than inferred from SQL/code or reconstructed from a\n  non-literal `@lineage` dataset.\n\nThe `tasks_total` denominator counts **every** task, including Pass-B tasks the\nscanner discovers decorator-free from raw Airflow operators. Those cannot carry\na decorator, so they can never be \"annotated\" and intentionally keep\n`annotation_coverage` below 1.0 — surfacing exactly the un-reviewed surface a\nhuman still owes an annotation.\n\n**Column-discovery edges are measured separately.** Edges recovered by the\ncolumn-discovery pass (Pass C — inferred, job-less hops that reconstruct\ncolumn lineage from a library call) do **not** enter `edges_total`,\n`high_confidence_edges` or `high_confidence_fraction`; those stay pure\ntask/declared-scope metrics. Instead the gate reports two **report-only**\ncounters (with per-DAG equivalents in the text and JSON output):\n`column_edges` (how many such edges) and `column_mappings` (the total\ncolumn-level mappings they carry). This keeps a large, entirely-inferred body\nof discovered column lineage from diluting the task confidence ratio a CI gate\nis built on — the same dimensional separation the annotation split introduced.\n\n\u003e **Known limitation (targeted for v0.6).** When a column-discovery function\n\u003e fans one dataset→dataset edge across many output branches, the analyzer can\n\u003e attach an inflated column-mapping set to that edge (order of ~1000 mappings\n\u003e per edge on some modules). It never affects the task/declared gate metrics —\n\u003e the count lands only in the report-only `column_mappings` — but it inflates\n\u003e that number and downstream column-lineage exports. Precise per-branch\n\u003e attribution is deferred to v0.6.\n\n```bash\n# Fail the build if fewer than 80% of tasks have lineage, under 90% of tasks are\n# annotated, or under 50% of edges are declared. The JSON format adds a per-DAG\n# breakdown (with the annotation fields on each DAG too).\ntrace-weaver gate --repo-path dags \\\n  --min-task-coverage 0.8 --min-annotation-coverage 0.9 --min-high-confidence 0.5\ntrace-weaver gate --repo-path dags --format json\n```\n\n**Which metric to use when.** They measure different things and are best gated\n**together**:\n\n- **`task_coverage` = data-flow completeness** — of the tasks that exist, how\n  many actually resolve to a lineage edge. Use it to guarantee the graph is\n  connected.\n- **`annotation_coverage` = review completeness** — how much of the pipeline a\n  human has explicitly marked, regardless of whether that mark produced a full\n  edge. Use it to guarantee humans have looked at every task. It is the honest\n  companion to `task_coverage`: annotating a task truthfully (a bare marker, or\n  an inputs-only declaration for a sink whose output you can't name) raises\n  `annotation_coverage` even when it can't raise `task_coverage`, so gating on\n  annotation never punishes honest, single-sided declarations.\n\nThresholds may also come from the environment — `TRACEWEAVER_MIN_TASK_COVERAGE`,\n`TRACEWEAVER_MIN_ANNOTATION_COVERAGE`, and `TRACEWEAVER_MIN_HIGH_CONFIDENCE` —\nand an explicit flag always wins over the env var. **Exit codes:** `0` pass, `1`\na threshold failed (the failing metric is printed), `2` usage error (bad path,\nunparseable threshold, invalid `--format`).\n\n#### Running `gate` in CI\n\nOnce a release is published (see [Publishing](#publishing-docker-hub) below),\npull the image instead of building from source. `gate` never writes a file, so\na plain bind mount at the image's `WORKDIR` (`/work`) is enough — no volume\npermission juggling needed (contrast `scan -o`, which writes into the mount\nand does need it — see [Running via Docker](#running-via-docker)):\n\n```bash\ndocker run --rm -v \"$PWD:/work\" \u003cdockerhub-namespace\u003e/trace-weaver:0.4.0 \\\n  gate --repo-path dags --min-task-coverage 0.8 --min-high-confidence 0.5\n```\n\n**GitHub Actions:**\n\n```yaml\nname: lineage-gate\non: [pull_request]\njobs:\n  gate:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - name: trace-weaver lineage gate\n        run: |\n          docker run --rm -v \"$PWD:/work\" \u003cdockerhub-namespace\u003e/trace-weaver:0.4.0 \\\n            gate --repo-path dags --min-task-coverage 0.8 --min-high-confidence 0.5\n```\n\n**Bitbucket Pipelines** (needs the `docker` service to run `docker run` inside\na step):\n\n```yaml\ndefinitions:\n  services:\n    docker:\n      memory: 2048\n\npipelines:\n  pull-requests:\n    '**':\n      - step:\n          name: trace-weaver lineage gate\n          services:\n            - docker\n          script:\n            - docker run --rm -v \"$BITBUCKET_CLONE_DIR:/work\" \u003cdockerhub-namespace\u003e/trace-weaver:0.4.0\n                gate --repo-path dags --min-task-coverage 0.8 --min-high-confidence 0.5\n```\n\nSwap `\u003cdockerhub-namespace\u003e` for the `DOCKERHUB_USERNAME` the image was\npublished under, and pin to a release tag (`:0.4.0`) rather than `:latest` for\na reproducible gate.\n\n---\n\n## Lineage decorator (`@lineage`)\n\nFor **dataset-level** lineage that the code analyzer can't see (an external API,\nan opaque UDF, a step that shells out), declare the datasets a task reads and\nwrites with the `@lineage` decorator. Like the rest of the SDK it is a **runtime\nno-op** — it returns your function unchanged and only attaches\n`__traceweaver_lineage__` — and the scanner reads it statically without importing\nyour module.\n\n```python\nfrom trace_weaver import lineage\n\n@lineage(\n    inputs=[\"s3://acme-raw/sales/{ds}/events.parquet\"],   # {ds} template is fine\n    outputs=[\"iceberg://warehouse.sales.bronze_events\"],\n    name=\"ingest_sales_events\",          # optional: overrides the task name\n    description=\"Land raw S3 events into bronze.\",\n)\ndef build_bronze():\n    ...\n\n@lineage            # bare form: marks the function, declares no datasets\ndef touch():\n    ...\n```\n\n- `inputs` / `outputs` are lists of dataset **URI strings** (`s3://`, `iceberg://`,\n  `postgresql://`, `mongodb://`, `file://`, or an Airflow conn-id ref). A string\n  may contain `{placeholders}` — it is still treated as one declared dataset.\n- The scanner recognises every import form: `from trace_weaver import lineage`,\n  `... import lineage as X`, `import trace_weaver` + `@trace_weaver.lineage`, and\n  `import trace_weaver as tw` + `@tw.lineage`. It also works **stacked with\n  Airflow's `@task`** in any order.\n- **Confidence:** a string-literal dataset is **declared / high confidence**; a\n  non-literal entry (an f-string, an arbitrary call, a subscript) is kept as a\n  best-effort textual representation and marked **medium confidence** (inferred)\n  so the gate and exporters can tell it apart.\n\n### URIs may live in a shared constants module\n\nYou don't have to inline the URI string. A dataset entry may reference a\n**module-level string constant** — locally or imported from another file — and\nit still resolves to **declared / high confidence**. This lets a team centralize\nits dataset URIs in one place (e.g. `config/datasets.py`) and reference them by\nname everywhere:\n\n```python\n# config/datasets.py\nRAW_SALES    = \"s3://acme-raw/sales/events.parquet\"\nBRONZE_SALES = \"iceberg://warehouse.sales.bronze\"\n\n# services/ingest.py\nfrom trace_weaver import lineage\nfrom config.datasets import RAW_SALES, BRONZE_SALES\n\n@lineage(inputs=[RAW_SALES], outputs=[BRONZE_SALES])   # still declared / HIGH\ndef ingest_sales():\n    ...\n```\n\nThe scanner builds a repo-wide table of module-level constants before scanning\n(keyed by the dotted module path a file imports under — `config/datasets.py` →\n`config.datasets`), so a constant defined in any scanned file resolves. The same\nresolution applies to `@tw.task` / `@tw.sql` `inputs=`/`outputs=`.\n\n**Supported reference forms** (each resolves to declared / HIGH):\n\n| Form                                             | Example                                                        |\n| ------------------------------------------------ | -------------------------------------------------------------- |\n| Bare name, same module                           | `RAW = \"s3://…\"` then `inputs=[RAW]`                           |\n| One-level alias, same module                     | `RAW = BASE` (chained, cycle-guarded) then `inputs=[RAW]`     |\n| `from pkg.mod import NAME`                        | `from config.datasets import RAW_SALES` then `inputs=[RAW_SALES]` |\n| `from pkg.mod import NAME as ALIAS`              | `from config.datasets import RAW_SALES as RAW` then `inputs=[RAW]` |\n| `import pkg.mod as m` + attribute               | `import config.datasets as ds` then `inputs=[ds.RAW_SALES]`   |\n| `import pkg.mod` + dotted attribute             | `import config.datasets` then `inputs=[config.datasets.RAW_SALES]` |\n\n**NOT resolved** (kept as best-effort text, marked medium / inferred — never\nguessed): a missing/undefined name, a function call (`make_uri()`), an f-string\nwith placeholders (`f\"s3://…/{day}\"`), a subscript (`CFG[\"raw\"]`), or a value\nthat is not a plain string literal.\n\nSee [`examples/sample_dags/declared_lineage.py`](examples/sample_dags/declared_lineage.py)\nfor a full DAG.\n\n---\n\n## Optional: the Python authoring SDK (`@tw`)\n\nYou rarely need it — the compiler infers lineage from SQL and pandas/Spark code.\nReach for the SDK only to **declare** a column the analyzer flagged opaque, or to\nattach a description/transform label. The decorator is a **runtime no-op**: it\nreturns your function unchanged, so your DAGs run exactly as before.\n\n```bash\ncd python \u0026\u0026 pip install -e .        # stdlib only, Python 3.9+\n```\n\n### Installing in an Airflow image\n\nDAG code runs inside your Airflow image, not this repo, so install the SDK\nstraight from GitHub — no local checkout needed:\n\n```bash\npip install \"trace-weaver @ git+https://github.com/ccyrene/trace-weaver@main#subdirectory=python\"\n```\n\nThis tracks the `main` branch. Once a versioned tag is released, pin to it\ninstead (e.g. `@v0.2.0#subdirectory=python`) — a tag will be preferred over\n`@main` as soon as one exists, for reproducible image builds.\n\n```python\nimport trace_weaver as tw\n\n# Per-file defaults: FQN parts + dag, so you can use bare table names below.\ntw.configure(service=\"Test Database\", database=\"poc_db\", schema=\"public\",\n             dag=\"medallion_lineage\")\n\nBRONZE_SQL = \"SELECT CAST(raw_event_id AS BIGINT) AS event_id FROM landing_sales\"\n\n# SQL step — column lineage auto-derived from the query (no column_map needed).\n@tw.sql(BRONZE_SQL, inputs=[\"landing_sales\"], outputs=[\"bronze_sales\"],\n        transform=\"CAST / DEDUPE\")\ndef build_bronze():\n    ...\n\n# pandas/Spark step — declare only the columns the analyzer can't trace.\n@tw.task(\n    inputs=[\"bronze_sales\"],\n    outputs=[\"silver_sales\"],\n    column_map=[\n        # (sources, target, function); sources may be a bare string for one source\n        ([\"amount\", \"currency\"], \"amount_usd\", \"ROUND(amount * fx[currency], 2)\"),\n    ],\n    copy=[\"event_id\", \"customer_name\"],   # same-name passthrough → declared identity\n)\ndef build_silver():\n    ...\n```\n\n**Decorator reference** — `@task(dag=None, inputs=None, outputs=None,\nengine=None, sql=None, description=None, transform=None, column_map=None,\ncopy=None, **kwargs)`:\n\n| arg           | type                              | notes |\n|---------------|-----------------------------------|-------|\n| `dag`         | `str`                             | DAG / pipeline id (falls back to `configure(dag=)` or a `with DAG(...)` block) |\n| `inputs`      | `list[str \\| Dataset]`            | source dataset FQNs (bare names expand via `configure`) |\n| `outputs`     | `list[str \\| Dataset]`            | output dataset FQNs |\n| `engine`      | `\"sql\"\\|\"pandas\"\\|\"spark\"\\|\"python\"\\|\"bash\"` | optional; inferred (`sql` if a query is present, else `python`). Unknown values tolerated |\n| `sql`         | `str` or module-level const name  | parsed when `engine=\"sql\"` |\n| `description` | `str` or const name               | rich (markdown) edge description |\n| `transform`   | `str`                             | short transform-kind label |\n| `column_map`  | `list[(sources, target, function)]` | declared, authoritative lineage; `sources` may be a list **or a bare string** |\n| `copy`        | `list[str]`                       | same-name passthrough columns — each a declared identity; `column_map` wins on conflict |\n| `**kwargs`    | anything                          | tolerated \u0026 recorded for forward-compat |\n\n`@tw.sql(QUERY, ...)` is sugar for `@task(engine=\"sql\", sql=QUERY, ...)`.\n`inputs` + `outputs` are the meaningful minimum; everything else is optional.\nDatasets use OpenMetadata style `service.database.schema.table` (the *service*\nsegment may contain spaces). Every declaration is also appended to `tw.registry`\n(a `list[trace_weaver.Declaration]`) for optional runtime introspection — the\ndecorator's only side effect.\n\n---\n\n## Build, test \u0026 lint\n\n```bash\ncargo build --release                                            # → target/release/trace-weaver\ncargo test --workspace                                           # Rust test suite\ncargo clippy --workspace --all-targets --all-features -- -D warnings\ncargo fmt --all --check\nruff check python examples                                       # Python SDK + examples\n```\n\nRust 2021 edition, MSRV 1.80.\n\n## Running via Docker\n\nThe `Dockerfile` is a two-stage build: a `builder` stage compiles a stripped\nrelease binary, and a `debian:bookworm-slim` `runtime` stage carries just the\nbinary + CA roots, running as a fixed non-root `traceweaver` user (uid/gid\n`10001`). `WORKDIR` is `/work` and `ENTRYPOINT` is `[\"trace-weaver\"]`, so any\nCLI subcommand is just appended as `docker run` arguments:\n\n```bash\ndocker build -t trace-weaver:0.4.0 .\n\n# Read-only commands (gate, validate, export --dry-run): bind-mount the repo\n# at /work — no special permissions needed since nothing is written back.\ndocker run --rm -v \"$PWD:/work\" trace-weaver:0.4.0 \\\n  gate --repo-path examples/sample_dags --min-task-coverage 0.5\n\n# Commands that write into the mount (scan -o, export -o, graph -o): pass\n# --user \"$(id -u):$(id -g)\" so the container writes as YOUR uid, not the\n# image's fixed uid 10001 (which the bind-mounted host directory doesn't\n# grant write access to).\ndocker run --rm --user \"$(id -u):$(id -g)\" -v \"$PWD:/work\" trace-weaver:0.4.0 \\\n  scan examples/sample_dags -o /work/out.weave.json\n```\n\nOnce an image is published (below), swap the locally-built `trace-weaver:0.4.0`\ntag for `\u003cdockerhub-namespace\u003e/trace-weaver:0.4.0`.\n\n## Publishing (Docker Hub)\n\n`.github/workflows/publish.yml` builds the production image and pushes it to\nDocker Hub as `docker.io/$DOCKERHUB_USERNAME/trace-weaver` (tags `{version}` and\n`latest`). It runs on a pushed `v*` tag and via manual `workflow_dispatch`. It\nrequires two **repository secrets**:\n\n| secret               | purpose                                            |\n|----------------------|----------------------------------------------------|\n| `DOCKERHUB_USERNAME` | Docker Hub account/namespace to push under         |\n| `DOCKERHUB_TOKEN`    | Docker Hub access token (used by `docker/login-action`) |\n\nThe existing `ci.yml` gates are unchanged; publishing is a separate workflow.\n\n**Make the Docker Hub repository public.** The workflow only pushes; it never\nflips repo visibility. A private repo means every `docker pull` — including\nthe [CI gate examples](#running-gate-in-ci) above and anyone else's `docker\nrun` — needs Docker Hub credentials. Set the repository (create it once,\nmanually, under the `DOCKERHUB_USERNAME` namespace, or via Docker Hub's own\nsettings after the first push) to **public** so it can be pulled anonymously.\n\n## Changelog\n\nSee [`CHANGELOG.md`](CHANGELOG.md).\n\n## License\n\nApache-2.0. See [`LICENSE`](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fccyrene%2Ftrace-weaver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fccyrene%2Ftrace-weaver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fccyrene%2Ftrace-weaver/lists"}