{"id":48421281,"url":"https://github.com/sidequery/sqlmesh-openlineage","last_synced_at":"2026-04-06T08:30:41.454Z","repository":{"id":331918627,"uuid":"1132248810","full_name":"sidequery/sqlmesh-openlineage","owner":"sidequery","description":null,"archived":false,"fork":false,"pushed_at":"2026-02-16T15:03:14.000Z","size":185,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-16T22:30:58.201Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sidequery.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-11T15:59:21.000Z","updated_at":"2026-02-16T15:03:18.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/sidequery/sqlmesh-openlineage","commit_stats":null,"previous_names":["sidequery/sqlmesh-openlineage"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/sidequery/sqlmesh-openlineage","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sidequery%2Fsqlmesh-openlineage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sidequery%2Fsqlmesh-openlineage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sidequery%2Fsqlmesh-openlineage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sidequery%2Fsqlmesh-openlineage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sidequery","download_url":"https://codeload.github.com/sidequery/sqlmesh-openlineage/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sidequery%2Fsqlmesh-openlineage/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31464604,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T21:22:52.476Z","status":"online","status_checked_at":"2026-04-06T02:00:07.287Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-06T08:30:40.593Z","updated_at":"2026-04-06T08:30:41.447Z","avatar_url":"https://github.com/sidequery.png","language":"Python","readme":"# sqlmesh-openlineage\n\nOpenLineage integration for SQLMesh. Automatically emits lineage events to Marquez or any OpenLineage-compatible backend.\n\n## Features\n\n- **Table-level lineage**: Track which models depend on which upstream models\n- **Column-level lineage**: Track which columns flow from source to destination\n- **Schema capture**: Column names and types for each model\n- **Execution stats**: Duration, rows processed, bytes processed\n- **Per-model events**: START/COMPLETE/FAIL events for each model evaluation\n\n## Installation\n\n```bash\npip install sqlmesh-openlineage\n```\n\nOr with uv:\n\n```bash\nuv add sqlmesh-openlineage\n```\n\n## Quick Start (CLI Users)\n\n**Note:** This package requires Python-based SQLMesh configuration (`config.py`), not YAML configuration.\n\nAdd this to your `config.py`:\n\n```python\nimport sqlmesh_openlineage\n\nsqlmesh_openlineage.install(\n    url=\"http://localhost:5000\",\n    namespace=\"my_project\",\n    # api_key=\"...\",  # optional\n)\n\nfrom sqlmesh.core.config import Config\n\nconfig = Config(\n    # ... your existing config\n)\n```\n\nThen run `sqlmesh run` as normal. OpenLineage events will be emitted for each model evaluation.\n\n## Environment Variables\n\nYou can also configure via environment variables:\n\n```bash\nexport OPENLINEAGE_URL=http://localhost:5000\nexport OPENLINEAGE_NAMESPACE=my_project\nexport OPENLINEAGE_API_KEY=...  # optional\n```\n\nThen in `config.py`:\n\n```python\nimport sqlmesh_openlineage\nsqlmesh_openlineage.install()  # reads from env vars\n```\n\n## How It Works\n\nThis package uses SQLMesh's `set_console()` API to inject a custom Console wrapper. The wrapper intercepts per-snapshot lifecycle events and emits corresponding OpenLineage events:\n\n- `START` event when a model evaluation begins\n- `COMPLETE` event when evaluation succeeds (includes execution stats)\n- `FAIL` event when evaluation fails or audits fail\n\n## Events Emitted\n\n| SQLMesh Event | OpenLineage Event | Data Included |\n|---------------|-------------------|---------------|\n| Model evaluation start | RunEvent(START) | Input datasets, output dataset with schema, column lineage |\n| Model evaluation success | RunEvent(COMPLETE) | Execution stats (rows, bytes, duration) |\n| Model evaluation failure | RunEvent(FAIL) | Error message |\n| Audit failure | RunEvent(FAIL) | Audit failure details |\n\n## Column-Level Lineage\n\nThe integration automatically extracts column-level lineage using SQLMesh's built-in lineage analysis. For example, if you have:\n\n```sql\n-- customers.sql\nSELECT customer_id, name, email FROM raw_customers\n\n-- customer_summary.sql\nSELECT\n    c.customer_id,\n    c.name as customer_name,\n    COUNT(o.order_id) as total_orders\nFROM customers c\nLEFT JOIN orders o ON c.customer_id = o.customer_id\nGROUP BY c.customer_id, c.name\n```\n\nThe lineage will show that `customer_summary.customer_name` traces back to `customers.name`.\n\n## Testing with Marquez\n\n```bash\n# Start Marquez (requires Docker)\ndocker compose up -d\n\n# Configure and run SQLMesh\nexport OPENLINEAGE_URL=http://localhost:5001\nsqlmesh run\n\n# View lineage at http://localhost:3000\n```\n\n## Development\n\n```bash\n# Install dependencies\nuv sync --dev\n\n# Run tests (unit + integration)\nuv run pytest tests/ -v\n\n# Run Marquez integration test (requires Docker)\ndocker compose up -d\nuv run pytest tests/test_marquez_integration.py -v -s\ndocker compose down\n```\n\n## License\n\nMIT\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsidequery%2Fsqlmesh-openlineage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsidequery%2Fsqlmesh-openlineage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsidequery%2Fsqlmesh-openlineage/lists"}