{"id":51577195,"url":"https://github.com/royashbrook/sql-spider","last_synced_at":"2026-07-11T01:32:07.095Z","repository":{"id":364399066,"uuid":"1264336064","full_name":"royashbrook/sql-spider","owner":"royashbrook","description":"Deterministic SQL dependency-graph extractor and spider-to-closure mapper. Pluggable multi-dialect, optional graphify export. Drop-in agent skill.","archived":false,"fork":false,"pushed_at":"2026-06-22T05:22:03.000Z","size":466,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-22T07:21:26.699Z","etag":null,"topics":["agent-skill","claude-code","dependency-graph","knowledge-graph","scriptdom","sql","sqlite","tsql"],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/royashbrook.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-09T19:41:26.000Z","updated_at":"2026-06-22T05:22:07.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/royashbrook/sql-spider","commit_stats":null,"previous_names":["royashbrook/sql-spider"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/royashbrook/sql-spider","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/royashbrook%2Fsql-spider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/royashbrook%2Fsql-spider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/royashbrook%2Fsql-spider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/royashbrook%2Fsql-spider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/royashbrook","download_url":"https://codeload.github.com/royashbrook/sql-spider/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/royashbrook%2Fsql-spider/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35348376,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-skill","claude-code","dependency-graph","knowledge-graph","scriptdom","sql","sqlite","tsql"],"created_at":"2026-07-11T01:32:06.790Z","updated_at":"2026-07-11T01:32:07.045Z","avatar_url":"https://github.com/royashbrook.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"assets/logo-dark.svg\"\u003e\n    \u003cimg src=\"assets/logo.svg\" width=\"260\" alt=\"sql-spider\"\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003esql-spider\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\u003cem\u003eseed one object. it crawls the whole dependency web, one connected component, zero orphans.\u003c/em\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/royashbrook/sql-spider/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/royashbrook/sql-spider/actions/workflows/ci.yml/badge.svg\" alt=\"ci\"\u003e\u003c/a\u003e\n  \u003cimg src=\"https://img.shields.io/badge/.NET-10-111111?style=flat-square\" alt=\".NET 10\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/deterministic-yes-111111?style=flat-square\" alt=\"deterministic\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/license-MIT-111111?style=flat-square\" alt=\"MIT\"\u003e\n\u003c/p\u003e\n\n\u003e **sql-spider is a drop-in agent skill: you point an agent at it and it maps your database.**\n\u003e\n\u003e **Fastest path (anyone):** paste this one line to your agent:\n\u003e\n\u003e \u003e `Fetch https://raw.githubusercontent.com/royashbrook/sql-spider/main/SKILL.md and follow it to install and use sql-spider.`\n\u003e\n\u003e The agent fetches the instructions, installs the skill so it's there next time, builds the tool,\n\u003e and drives the read-only spider loop to a closed graph. No git knowledge needed on your end.\n\u003e (Equivalent in plain words: *\"install this skill from https://github.com/royashbrook/sql-spider,\n\u003e then map my database starting from `Orders`.\"*) That's the whole intended use.\n\u003e\n\u003e **If you are an AGENT reading this README:** your entry point is [`SKILL.md`](SKILL.md), go\n\u003e there. First clone the repo into your skills directory so the capability persists\n\u003e (`git clone https://github.com/royashbrook/sql-spider ~/.claude/skills/sql-spider` for Claude\n\u003e Code; wherever your harness auto-discovers skills otherwise). Run-in-place works once and is\n\u003e forgotten; installed, every future session discovers it on its own, so *\"use this tool\"*\n\u003e includes installing it. For the graphify pairing, install graphify's *skill* too, not just its\n\u003e binary (two commands, see [Pairing with graphify](#pairing-with-graphify-and-a-quick-look); the\n\u003e second is the one everyone skips).\n\u003e\n\u003e **If you are a HUMAN:** you do not need to read the rest of this README. It is reference for\n\u003e when you (or your agent) want the details. The tool *can* be run by hand (it is a normal .NET\n\u003e CLI), but it was built to be DRIVEN by an agent: it emits read-only queries, something runs\n\u003e them against your database, the results feed back, round by round, until the graph closes. That\n\u003e loop is an agent's job, not a fun manual one. The tool never connects to a database itself; the\n\u003e agent (or you) is the adapter. So the move is the same one above: point your agent at it.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"380\" alt=\"the raw dependency graph sql-spider produces\" src=\"https://github.com/user-attachments/assets/01d2211a-5e06-4e22-9443-f633f7e276fb\"\u003e\n  \u003cimg width=\"380\" alt=\"the same graph after graphify clusters it\" src=\"https://github.com/user-attachments/assets/4fb2d594-673c-4587-9931-428b2748bd54\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\u003csub\u003ethe closed graph (left), and the same graph after graphify clusters it (right), \"equally beautiful and useless\" to a human, and exactly the structured input an agent wants.\u003c/sub\u003e\u003c/p\u003e\n\nA deterministic SQL dependency-graph extractor and spider. The graph engine is\n**dialect-agnostic**: only the \"parse SQL into dependency facts\" step is dialect-specific,\nbehind a one-method `IDialectExtractor` interface. T-SQL is parsed with\n[Microsoft.SqlServer.TransactSql.ScriptDom](https://www.nuget.org/packages/Microsoft.SqlServer.TransactSql.ScriptDom)\n(the same parser SQL Server itself uses) for full fidelity; other dialects (SQLite, and the\nMySQL/Postgres/etc. that [SqlParserCS](https://www.nuget.org/packages/SqlParserCS/) supports) go\nthrough a generic SqlParserCS-backed extractor. Either way it is fully deterministic, with no\nheuristics-that-drift and no live database connection needed to parse: it turns the actual SQL\ninto an AST and reads the dependencies straight off the tree, so the same input always produces\nthe same graph.\n\nIt does two things:\n\n1. **extract**: parse a directory of `.sql` files into a node-link dependency graph\n   (`graph.json`), a frontier of referenced-but-undefined objects (`frontier.json`), and a set\n   of self-audits. A single `.sql` file may define many objects (a full schema/install script,\n   a migration); each defined object becomes its own node with its own correctly-scoped edges.\n2. **spider**: seed one object, emit the read-only queries needed to pull its definition,\n   schema, and referencers; you run those against your database and feed the results back. Each\n   pass shrinks the frontier. You are done when the frontier is empty **and** the graph is a\n   single connected component.\n\n## The success condition\n\n\u003e Seed an object, iterate read-only pulls until the graph is **closed**: one connected\n\u003e component, **zero orphans**, no phantom edges.\n\nThe connected-components audit is the gate. `extract` returns a non-zero exit code while the\ngraph is unclosed (more than one component, or any degree-0 node), so you can wire it into a\nbuild or CI step. When it prints `OK: single connected component, no orphans` you have a\ncomplete, self-consistent dependency closure around your seed.\n\n## Dialects\n\nThe headline design: **sql-spider is a dialect-agnostic dependency-graph engine.** Everything is\ndialect-**neutral**: graph assembly, the frontier, the spider loop (seed/generate/reverse/absorb),\nthe connected-components closure audit, lineage roll-up, and `viz`. It works\non dependency *facts*, never on SQL text. The only dialect-specific step is turning `.sql` into\nthose facts, which lives behind one small interface:\n\n```csharp\ninterface IDialectExtractor\n{\n    CorpusFacts Extract(IReadOnlyList\u003cstring\u003e files);   // SQL text -\u003e dependency facts\n}\n```\n\nTwo implementations ship:\n\n- **`TSqlExtractor`**: T-SQL via Microsoft ScriptDom. Full fidelity: tables, views, procs,\n  functions, triggers, declared foreign keys, alias-resolved column join-keys, and `UPDATE`-set\n  column lineage. This is the default (`--dialect tsql`).\n- **`GenericSqlExtractor`**: any dialect [SqlParserCS](https://www.nuget.org/packages/SqlParserCS/)\n  understands. Wired here for **SQLite** (`--dialect sqlite`): it walks the parsed AST and emits\n  facts for `CREATE TABLE` (+ its `FOREIGN KEY` constraints, table-level and inline), `CREATE VIEW`\n  (+ the tables its query selects from), `CREATE TRIGGER` (+ the tables it reads/writes), and\n  `CREATE INDEX` (the table it is on). SQLite has no stored procedures or SQL functions, so the\n  call/function facts simply never appear, which is correct for the dialect, not a gap.\n\n### Adding a dialect\n\nImplement `IDialectExtractor` (or, for any dialect SqlParserCS already supports, reuse\n`GenericSqlExtractor` with a different `Dialect`) and register it in `Extractor.Pick`. The rest of\nthe pipeline (graph, closure, frontier, viz) is untouched. The interface returns exactly the\nfacts the engine needs per file: the defined containers (name + kind), and per container its table\nreads/writes, called procs/functions, FK edges, and (where the dialect supports it) column\njoin-keys and lineage.\n\n\u003e SqlParserCS API note: its SQLite dialect does not parse SQLite's `CREATE TRIGGER` syntax (it\n\u003e routes to a constraint-trigger parser and throws). `GenericSqlExtractor` isolates that: it\n\u003e re-parses statement-by-statement so one unparseable statement never sinks the file, and falls\n\u003e back to a small regex to still pull a trigger's name and the tables it touches.\n\n## Project layout\n\nThe code is split into separate assemblies so the pluggable-parser strategy is structural, not\njust a convention: **the parser is a separate module, and the engine never depends on a specific\nSQL dialect.**\n\n```\nsql-spider.sln\nsql-spider.csproj            CLI / executable: arg routing + the dialect pick (--dialect -\u003e a\n                             concrete parser). The ONE place allowed to name both parsers.\nsrc/core/   SqlSpider.Core     the dialect-NEUTRAL engine: the IDialectExtractor interface +\n                             CorpusFacts/FileFacts/Container fact types, graph assembly, the\n                             frontier, join-key/lineage layers, the connected-components closure\n                             audit, the seed/generate/reverse/absorb orchestrator, and viz.\n                             References NO SQL parser at all.\nsrc/tsql/   SqlSpider.TSql     the T-SQL parser module: TSqlExtractor + its ScriptDom visitors.\n                             References Core + Microsoft.SqlServer.TransactSql.ScriptDom.\nsrc/generic/SqlSpider.Generic  the generic/SQLite parser module: GenericSqlExtractor.\n                             References Core + SqlParserCS.\n```\n\n`SqlSpider.Core` has **zero** parser package references. That is the load-bearing invariant. The\nengine consumes the dependency *facts* an `IDialectExtractor` returns and never sees SQL text, so\nit cannot depend on how that text was parsed. The CLI injects the picked parser into the engine as\na delegate; swap or add a parser module without recompiling the engine. (`dotnet run` from the\nrepo root still resolves to the root `sql-spider.csproj`, so every command below is unchanged.)\n\n## Quick start (Northwind)\n\nThe repo ships with Microsoft's public Northwind sample database script\n(`examples/northwind/northwind.sql`, MIT-licensed): one file that defines 13 tables, 16 views,\nand 7 stored procedures. It is exactly the multi-object-per-file shape the extractor is built for.\n\nRun it one step at a time:\n\n**1. Build** (once).\n```sh\ndotnet build\n```\n\n**2. Extract**: parse the script into a dependency graph. The outputs land right in the example\nfolder, so it stays self-contained (and they're gitignored: they regenerate every run, the repo\nonly ships the `.sql`).\n```sh\ndotnet run -- extract examples/northwind          # writes graph.json + frontier.json into examples/northwind/\n```\n\n**3. Visualize**: render that graph to a standalone HTML and open it.\n```sh\ndotnet run -- viz examples/northwind/graph.json   # writes examples/northwind/graph.html\nopen examples/northwind/graph.html                # macOS; or open it in any browser\n```\n\n(`viz` reads the `graph.json` from any `extract`, so in the spider loop you can re-`viz` after each\nround to watch the graph grow.)\n\nYou get 36 object nodes (13 tables, 16 views, 7 procs), 66 edges, and the connected-components\naudit reports **one component, zero orphans: closure holds.** That includes the lookup/junction\ntables (region, territories, the demographics tables) whose only links are declared foreign keys:\nNorthwind declares all of its FKs via `ALTER TABLE ... ADD CONSTRAINT`, and the extractor parses\nthose (and inline FK constraints) straight from the DDL, so a pure-static parse stays connected\nwith no live database needed.\n\n## Quick start, second dialect (Chinook / SQLite)\n\n`examples/sqlite/` ships [Chinook](https://github.com/lerocha/chinook-database), the SQLite\nworld's Northwind, as a schema-only DDL file (`chinook.sql`: 11 FK-connected `CREATE TABLE` +\n11 `CREATE INDEX`; the original script's ~15k rows of `INSERT` data were stripped so the example\nis fast and purely about the dependency graph). Point `extract` at it with `--dialect sqlite`:\n\n```sh\ndotnet run -- extract examples/sqlite --dialect sqlite\n```\n\nYou get 11 table nodes and 10 foreign-key edges (the `Employee.ReportsTo` self-FK is dropped as a\nself-edge), and the closure audit reports **one component, zero orphans**: every Chinook table is\nFK-reachable. No procs/functions/columns appear, which is correct for SQLite. Same engine, same\naudit, same `viz`, only the parser changed.\n\n## See the loop (AdventureWorks, stepped)\n\nNorthwind closes in one pass; it doesn't show the *loop*. `examples/adventureworks/` does: a\nstepped walk-through of the spider with **no live database**. It seeds on one table\n(`Sales.SalesOrderHeader`) and stages each pull by hand, so you watch the frontier shrink ring\nby ring (`7 tables → 6 tables + 1 function → 0`) until it closes:\n\n```sh\nbash examples/adventureworks/demo.sh\n```\n\nSee `examples/adventureworks/README.md` for the prose walk-through, including how genuinely\nstandalone objects are reported with a reason rather than as an error.\n\n## Subcommands\n\nInvoke any subcommand with `dotnet run -- \u003csubcommand\u003e [args]`, no separate binary to run, `dotnet run`\ncompiles if needed then runs. (For a standalone `sql-spider` command, `dotnet publish -c Release` or\n`dotnet tool install`.)\n\n```\ndotnet run -- extract  \u003ccorpus-dir\u003e [graph.json] [frontier.json] [--dialect tsql|sqlite] [--graphify[=out.json]]\ndotnet run -- seed     \u003croot-object\u003e \u003coutdir\u003e\ndotnet run -- generate \u003cfrontier.json\u003e \u003coutdir\u003e\ndotnet run -- reverse  \u003creferencers.csv\u003e \u003coutdir\u003e [--roots a b ...]\ndotnet run -- absorb   \u003ccsv\u003e [csv ...] --corpus \u003cdir\u003e\ndotnet run -- viz      \u003cgraph.json\u003e [out.html]\n```\n\n- **extract**: parse `*.sql` in `\u003ccorpus-dir\u003e` into `graph.json` + `frontier.json` + audits.\n  `--dialect` selects the parser (`tsql` default, or `sqlite`); see [Dialects](#dialects).\n- **seed**: cold start from a single root on an empty corpus: emit the pull queries for the\n  root's own definition, its schema (if a table), and everything that references it.\n- **generate**: emit forward-pull queries for the current frontier (undefined procs/tables).\n- **reverse**: from a referencers CSV, pull the module definitions of objects that reference\n  the given roots (one level of reverse dependency).\n- **absorb**: fold pulled CSVs back into the corpus as `.sql` (module defs are reassembled\n  from chunked columns; table schemas are synthesized into `create table` DDL).\n- **viz**: render an extracted `graph.json` to one self-contained interactive HTML file\n  (force-directed, draggable, nodes colored by kind, edge relations on hover). See below.\n\n## The loop (one pass)\n\n```sh\ndotnet run -- extract  corpus/ graph.json frontier.json\ndotnet run -- generate frontier.json stage/\n# run each emitted query against your database, save results as CSV into stage/\ndotnet run -- absorb   stage/*.csv --corpus corpus/\n# repeat: the frontier shrinks each pass, done when it is empty and the graph is one component\n```\n\n## Bring-your-own-adapter\n\nsql-spider **never opens a database connection.** It is a pure\nemit-queries / ingest-CSVs tool, which keeps it driver-agnostic and safe to run anywhere:\n\n- `generate` / `seed` / `reverse` write **read-only** `.sql` query files (each carries a\n  no-lock, deadlock-victim, `set nocount on` safety preamble) into an output directory.\n- You run those queries against your database with whatever adapter you like and save each\n  result as a CSV.\n- `absorb` reads those CSVs back into the corpus.\n\nThe CSV column contract is just the column names the emitted queries produce (`object_name` +\n`d00..dNN` chunks for module defs, `table_name`/`column_name`/... for schema, `fk_table`/... for\nFK edges, `referencing`/`referenced` for reverse deps). Any adapter that produces those columns\nworks. Two common ones:\n\n```powershell\n# PowerShell / Invoke-Sqlcmd (works everywhere PowerShell does, including Linux/macOS)\nInvoke-Sqlcmd -ServerInstance \u003cserver\u003e -Database \u003cdb\u003e -InputFile stage\\20240101-ProcDefs.sql |\n  Export-Csv -NoTypeInformation -Encoding utf8 -Path stage\\20240101-ProcDefs.csv\n```\n\n\u003e Your adapter must emit **real (RFC-4180-style) CSV**: quoted fields, so embedded commas,\n\u003e quotes, and newlines survive. `Export-Csv` does this. Plain `sqlcmd -s \",\"` does **not** (it\n\u003e never quotes, prints a dashed separator row, and renders NULL as the literal word): module\n\u003e definitions round-tripped through it come back corrupted. If you use `sqlcmd`, post-process its\n\u003e output into proper CSV, or use any client library that writes quoted CSV.\n\n## Pairing with graphify (and a quick look)\n\nsql-spider stands on its own: it builds and closes the dependency graph, and `viz` (below) shows\nit with zero install. We built it that way first. The graphify pairing came second, as a way to\nplug the closed graph into a richer knowledge-graph tool, so it is an **opt-in switch**, not a\ndependency.\n\nPass `--graphify` to *also* write a graphify-format graph alongside the native one. Use the flag,\ndo not copy the native `graph.json` into graphify (it is a directed multigraph and gets under-read\nraw):\n\n```sh\ndotnet run -- extract corpus/ --graphify     # writes corpus/graph.json AND corpus/graphify-out/graph.json\ngraphify cluster-only corpus/                 # community detection + report   (graphify, installed separately)\n```\n\nThe native `graph.json` is always written and unchanged. The `--graphify` file is a separate,\ngraphify-shaped copy of the same graph (top-level `nodes`/`edges`/`hyperedges`, `confidence_score`\nadded) that drops straight into [graphify](https://github.com/safishamsi/graphify)'s `cluster-only`\n/ `query` / `path` / `explain` / `merge-graphs`. By default it **keeps sql-spider's own relation\nvocabulary** (`fk` / `references` / `writes` / `calls` / `join_key`); graphify ingests arbitrary\nrelations fine, and the read-vs-write split is the most useful signal for \"what actually writes this\ntable.\" Add `--graphify-standard` to collapse onto graphify's blessed enum instead\n(`references` / `calls` / `shares_data_with`), which is handy when you are merging many databases and\nwant one uniform vocabulary. If you don't use graphify, ignore the flag.\n\ngraphify is installed separately, and **it is two commands, not one**:\n\n```sh\nuv tool install graphifyy            # the graphify binary\ngraphify install --platform claude   # registers graphify's own skill with your agent\n```\n\nThe second command is the one everyone (humans and agents alike) skips. Without it your agent can\nrun graphify when told to, but will never reach for it on its own, because the skill registration\nis what makes graphify a discoverable capability rather than just a binary on PATH. Swap\n`--platform claude` for your harness if it isn't Claude Code.\n\n\u003e Tip: to ask graphify \"how is table X used\", use `graphify explain X`, not `query X`. A referenced\n\u003e table is a sink (its edges point inward), so an outward `query` returns just the table itself,\n\u003e while `explain` shows the inbound edges that answer the question.\n\nIf you just want a quick look without installing anything, `viz` renders the native `graph.json`\nto a standalone HTML file:\n\n```sh\ndotnet run -- viz graph.json          # writes graph.html\ndotnet run -- viz graph.json out.html  # or name the output\n```\n\nThe output is one self-contained HTML file. Open it in any browser. It draws an interactive\nforce-directed graph (vis-network from a CDN, nothing to install): nodes colored by kind\n(table / view / proc / function / trigger / column / script), draggable, with each edge's\nrelation shown on hover. The graph data is embedded directly in the file, so there's no server\nand no separate data file to serve.\n\n## Scope\n\nThis is the whole thing: a self-contained skill plus the .NET tool it drives. It is not a\ncomponent of any other project; you clone it and use it.\n\ngraphify interoperability is one-directional and opt-in: `extract --graphify` writes a\ngraphify-format projection alongside the native graph, so a closed graph drops straight into\ngraphify for clustering and query if you want to go further than the built-in `viz` (see\n[Pairing with graphify](#pairing-with-graphify-and-a-quick-look)). There is no graphify-native port\nof this tool and none is planned. The value is the pluggable-parser engine and the\nspider-to-closure loop, delivered as a drop-in skill, not a contribution to another codebase.\n\n**Names are database-scoped, not schema-scoped (a deliberate choice).** The extractor identifies\nobjects by bare name: `[dbo].[Foo]`, `dbo.Foo`, and `FOO` all resolve to one node, which is what\nyou want inside one database. The flip side: two same-named objects in *different* schemas\n(`sales.orders` vs `archive.orders`) merge into one node. We kept it that way because the spider's\nunit of work is one database reached through one adapter, and single-name identity keeps every\njoin (frontier, absorb, reverse) simple and predictable. If your map genuinely spans schemas or\ndatabases, run the loop once per scope and combine afterward: the boundary edges of one closed\ngraph (references that leave the scope) are the *seeds* for the next scan, and the resulting\n`graph.json` files merge on node id (`graphify merge-graphs a.json b.json` does exactly this, or\na few lines of json union). Not turnkey, but the bones are all here, and an agent driving the\nskill can do it without new tooling.\n\n## Tests\n\n```sh\nbash tests/run-tests.sh\n```\n\nEnd-to-end regression tests that drive the real CLI against small fixture corpora: the shipped\nexamples must close with their documented numbers, and each fixture encodes a specific fixed bug\n(if one fails, that bug is back). CI runs the same script on every push.\n\n## A real run (how we actually used it)\n\nThis tool started as a way to map a database we **couldn't connect to directly**: a\n[TMW](https://transportation.trimble.com/) system (Trimble's transportation-management platform),\nreachable only through a job that had read access. So the tool never connected: it emitted the\nread-only queries, that job ran them and committed back CSVs, and we fed those in. The whole\nbring-your-own-adapter design above is exactly that experience, generalized.\n\nThe run went in rounds. We seeded from a handful of core tables (the order, payment, and invoice\nheaders). Each round was the same: `extract` → read the frontier → `generate`/`reverse` the pull\nqueries → run them through the job → `absorb` the CSVs → `extract` again.\n\n- Forward-closure from a curated set of procedures converged fast: undefined procs `90 → 23 → 8`\n  over two passes.\n- Reverse-sweeping the three root tables pulled in everything that referenced them; one of those\n  tables alone is referenced by **~1,500 objects**.\n- The full order/pay/invoice subsystem closed at roughly **3,000 objects / 17,000 edges in one\n  connected component.**\n\nEvery gap the closure hit showed up as a degree-0 node or a disconnected component, and the\nself-audit pointed straight at it. The ones we closed are each now baked into the tool:\n\n- **Truncation**: the largest procedures blew past the query driver's character cap and came back\n  clipped. Fix: pull each definition in `substring` chunks and reassemble; re-pull anything that hit\n  the cap at a higher chunk count.\n- **Legacy syntax**: procedures written in the 1990s (double-quoted string literals, old `*=`\n  outer joins) that the modern parser rejects. Fix: fall back to the older T-SQL parser dialect when\n  the current one errors.\n- **FK-only relationships**: lookup tables whose only link is a declared foreign key. Fix: parse\n  FK constraints straight from the DDL, including `ALTER TABLE ... ADD CONSTRAINT`.\n- **Function calls**: scalar UDFs in expressions and table-valued functions in `FROM` weren't being\n  recorded as edges. Fix: extract them too.\n\nAs a quick sanity check on the finished tool: seeding from a single table and pulling just 25 of its\n~1,500 referencers produced a **94-node, single-connected-component** graph in two rounds. The loop\nholds.\n\nAnd the payoff, measured rather than guessed: building the structural layer originally pulled\n**2,308 KB of raw SQL across 121 read-only queries (~580k tokens of material)**. Re-verifying every\nstructural claim in our internal documentation against the finished graph (62 claims) consumed\n**~15 KB (~4k tokens) and zero new database queries, about 1% of the original cost.** That is the\nworking model: the graph is the *finder*. The pull cost is not an extra cost, you pay it anyway the\nfirst time you analyze the database at all; the graph is what keeps you from paying it again on\nevery question. Dependency questions are answered from the graph, and you only open the underlying\nSQL when the graph points you at the one object that matters.\n\nBy the time I created this for use with graphify on something else, I had already done all the work with this raw data just as text and sql code and had generated a significant corpus of data that an agent could consume. But I was able to validate the output of that using graphify as a skill (at greatly reduced token cost) and also do additional research which helped in a different way using this skill.\n\n\n## Requirements\n\n- .NET 10 SDK\n- `Microsoft.SqlServer.TransactSql.ScriptDom` (T-SQL parser) and `SqlParserCS` (generic /\n  SQLite parser), both restored automatically by `dotnet build`.\n\n## License\n\nMIT.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froyashbrook%2Fsql-spider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Froyashbrook%2Fsql-spider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froyashbrook%2Fsql-spider/lists"}