{"id":19963238,"url":"https://github.com/bilbottom/sql-lineage","last_synced_at":"2025-05-03T22:31:58.167Z","repository":{"id":229286068,"uuid":"776081300","full_name":"Bilbottom/sql-lineage","owner":"Bilbottom","description":"Personal SQL lineage generator.","archived":false,"fork":false,"pushed_at":"2024-10-08T07:21:30.000Z","size":132,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-29T15:41:01.984Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bilbottom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-22T16:36:40.000Z","updated_at":"2024-10-08T07:21:32.000Z","dependencies_parsed_at":null,"dependency_job_id":"7ab217a4-a730-48dc-a580-da81a18a81dc","html_url":"https://github.com/Bilbottom/sql-lineage","commit_stats":null,"previous_names":["bilbottom/sql-lineage"],"tags_count":3,"template":false,"template_full_name":"Bilbottom/python-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bilbottom%2Fsql-lineage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bilbottom%2Fsql-lineage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bilbottom%2Fsql-lineage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bilbottom%2Fsql-lineage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bilbottom","download_url":"https://codeload.github.com/Bilbottom/sql-lineage/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224374655,"owners_count":17300691,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T02:15:20.935Z","updated_at":"2025-05-03T22:31:58.149Z","avatar_url":"https://github.com/Bilbottom.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n[![Python](https://img.shields.io/badge/Python-3.11+-blue.svg)](https://www.python.org/downloads/)\n[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)\n[![tests](https://github.com/Bilbottom/sql-lineage/actions/workflows/tests.yaml/badge.svg)](https://github.com/Bilbottom/sql-lineage/actions/workflows/tests.yaml)\n[![coverage](coverage.svg)](https://github.com/dbrgn/coverage-badge)\n[![GitHub last commit](https://img.shields.io/github/last-commit/Bilbottom/sql-lineage)](https://shields.io/badges/git-hub-last-commit)\n\n[![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-square)](https://github.com/prettier/prettier)\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/Bilbottom/sql-lineage/main.svg)](https://results.pre-commit.ci/latest/github/Bilbottom/sql-lineage/main)\n\n\u003c/div\u003e\n\n---\n\n# SQL Lineage 🔀\n\nPersonal SQL lineage generator.\n\nBuilt on the following awesome libraries:\n\n- [SQLGlot](https://github.com/tobymao/sqlglot) for SQL parsing\n- [arguably](https://github.com/treykeown/arguably) for the CLI\n\n## So... what is this? 🤔\n\n\u003e [!NOTE]\n\u003e\n\u003e This project is still in development and currently only parses the CTEs of an SQL query since that's all I need for now.\n\nI write _a lot_ of SQL, and I often need to understand \"lineage\" in lots of different ways.\n\nThings like [dbt](https://www.getdbt.com/) and [SQLMesh](https://sqlmesh.com/) are great for object lineage (tables, columns, etc.), but I often need to understand \"lineage\" like the CTE lineage in a query, among other things.\n\nThis project is just a personal tool to help me generate lineage diagrams (in [Mermaid](https://mermaid.js.org/) syntax) for SQL queries to fit that requirement.\n\nUpcoming improvements will (hopefully) include:\n\n- Semantic edges (e.g. distinguish between `JOIN`, `WHERE`, `UNION`, etc.)\n- Support for parameterised queries (e.g. with Jinja blocks, like [dbt](https://www.getdbt.com/))\n- Parsing only the `SELECT` part of a SQL file that includes DDL/DML commands\n- Lineage for multiple files in a single diagram\n- Column lineage to Mermaid\n\n## Installation ⬇️\n\nGrab a copy from PyPI like usual (note the `bills-` prefix):\n\n```\npip install bills-sql-lineage\n```\n\n## Usage 📖\n\n\u003e [!WARNING]\n\u003e\n\u003e This is likely to change significantly as the project evolves.\n\nPass the path to a SQL file to the `lineage` command to generate the lineage as a [Mermaid](https://mermaid.js.org/) diagram:\n\n```\nlineage path/to/file.sql\n```\n\nThis will write a Mermaid diagram to `path/to/file.mermaid`. You can control the target path with the `--target` argument:\n\n```\nlineage path/to/file.sql --target path/to/output.mermaid\n```\n\nBy default, the SQL dialect will be inferred by SQLGlot, but you can specify a dialect with the `--dialect` argument:\n\n```\nlineage path/to/file.sql --dialect snowflake\n```\n\n## Example 📝\n\nGiven the following SQL query:\n\n```sql\nwith\n\naaa as (select 1 as aa),\nbbb as (select 2 as bb),\nccc as (select 3 as cc, aa from aaa),\nddd as (select 4 as dd, aa from aaa where aa not in (select bb from bbb))\n\nselect *\nfrom ccc\n    inner join ddd using (aa)\n```\n\n...the following Mermaid diagram will be generated:\n\n```mermaid\ngraph TD\n    aaa\n    bbb\n    ccc\n    ddd\n    final\n    aaa --\u003e ccc\n    aaa --\u003e ddd\n    bbb --\u003e ddd\n    ccc --\u003e final\n    ddd --\u003e final\n```\n\nNote that the `final` node is an alias for the final `SELECT` statement since the final `SELECT` statement is not a CTE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbilbottom%2Fsql-lineage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbilbottom%2Fsql-lineage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbilbottom%2Fsql-lineage/lists"}