{"id":50915485,"url":"https://github.com/paulushcgcj/sqlfy","last_synced_at":"2026-06-16T14:30:59.870Z","repository":{"id":364699898,"uuid":"1249407458","full_name":"paulushcgcj/sqlfy","owner":"paulushcgcj","description":"Schema Graph Engine — Parse Flyway migrations into an AST, reconstruct your database schema state, and export LLM-ready vector context.","archived":false,"fork":false,"pushed_at":"2026-06-14T05:51:02.000Z","size":1436,"stargazers_count":1,"open_issues_count":61,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-14T06:26:10.494Z","etag":null,"topics":["llm","migration","sql"],"latest_commit_sha":null,"homepage":"https://paulushcgcj/github.io/sqlfy","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paulushcgcj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-25T17:05:28.000Z","updated_at":"2026-06-14T05:51:04.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/paulushcgcj/sqlfy","commit_stats":null,"previous_names":["paulushcgcj/sqlfy"],"tags_count":39,"template":false,"template_full_name":null,"purl":"pkg:github/paulushcgcj/sqlfy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulushcgcj%2Fsqlfy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulushcgcj%2Fsqlfy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulushcgcj%2Fsqlfy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulushcgcj%2Fsqlfy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paulushcgcj","download_url":"https://codeload.github.com/paulushcgcj/sqlfy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulushcgcj%2Fsqlfy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34410778,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm","migration","sql"],"created_at":"2026-06-16T14:30:55.678Z","updated_at":"2026-06-16T14:30:59.864Z","avatar_url":"https://github.com/paulushcgcj.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SQLfy\n\n**Schema Graph Engine** — Parse Flyway migrations into an AST, reconstruct your database schema state, and export LLM-ready vector context.\n\n```\nFlyway SQL files  →  sqlglot AST  →  Reconstructor  →  Schema Graph / SchemaState  →  LLM Chunks\n```\n\n---\n\n## Overview\n\nSQLfy reads a set of Flyway migration files in version order, parses each DDL statement into an abstract syntax tree, and reconstructs the **final state** of your database schema. From that state it produces:\n\n- An interactive **ERD** showing tables and foreign-key relationships\n- A structured **table explorer** with columns, types, constraints, indexes, and comments\n- Pre-formatted **LLM context chunks** ready to be embedded into a RAG pipeline or pasted into a prompt\n\nPrimary target dialect is **OracleDB**. **PostgreSQL**, **MySQL**, and **SQLite** are also supported via the `--dialect` flag.\n\n### Multi-Dialect Support\n\nSQLfy supports multiple SQL dialects with automatic type normalization:\n\n| Dialect | Invoke with | Type Normalization Examples |\n|---|---|---|\n| **Oracle** _(default)_ | `--dialect oracle` | `VARCHAR2` → `VARCHAR`, `NUMBER` → `NUMERIC` |\n| **PostgreSQL** | `--dialect postgres` | `SERIAL` → `INTEGER`, `TEXT` → `VARCHAR` |\n| **MySQL** | `--dialect mysql` | `TINYINT` → `SMALLINT`, `DATETIME` → `TIMESTAMP` |\n| **SQLite** | `--dialect sqlite` | `TEXT` → `VARCHAR`, `REAL` → `FLOAT` |\n\n**Usage:**\n```bash\nsqlfy dump ./postgres-migrations --dialect postgres\nsqlfy graph ./mysql-migrations --dialect mysql --format mermaid\nsqlfy insights ./sqlite-migrations --dialect sqlite\n```\n\n**How it works:**\n- The `--dialect` flag is passed to [sqlglot](https://github.com/tobymao/sqlglot) for parsing\n- Types are normalized to canonical forms (e.g., `SERIAL` → `INTEGER`, `VARCHAR2` → `VARCHAR`)\n- Auto-increment columns are detected per-dialect (`SERIAL`, `AUTO_INCREMENT`, `IDENTITY`)\n- Output formats work consistently across all dialects\n\n---\n\n## Repository Structure\n\n```\nsqlfy/\n├── app/          React + Vite + Tauri desktop UI\n├── cli/          Python CLI (pip-installable)\n│   ├── src/sqlfy/\n│   │   ├── core.py          Schema graph engine (data types, chunk builder, layout)\n│   │   ├── reconstructor.py Stateful migration processor (incremental, point-in-time)\n│   │   ├── schema_state.py  SchemaState dictionary — serialisable, LLM-ready snapshot\n│   │   └── main.py          argparse CLI entry point\n│   ├── tests/               pytest suite (140+ tests)\n│   └── pyproject.toml\n└── samples/      Shared Flyway .sql fixtures (Oracle DDL — used by app and test suite)\n```\n\n---\n\n## Quick Start\n\n### Desktop app\n\n```bash\ncd app\nnpm install\nnpm run dev          # Vite dev server (browser)\nnpx tauri dev        # Tauri desktop window\n```\n\nThe app is pre-loaded with the sample Oracle schema from `samples/`. Replace the SQL with your own Flyway files, or add files with **+ Add Migration File**, then click **▶ Parse →**.\n\n### CLI\n\n```bash\ncd cli\npip install .        # install\nsqlfy ./samples      # human-readable schema summary\n```\n\n---\n\n## Distribution\n\n### Automated releases (Recommended)\n\nEvery time you create a new tag, GitHub Actions automatically builds binaries for all platforms:\n\n```bash\n# Create and push a tag\ngit tag v0.20.0\ngit push origin v0.20.0\n```\n\nThis triggers the build workflow which creates:\n- `sqlfy-macos-arm64.zip` (macOS Apple Silicon)\n- `sqlfy-linux-amd64.zip` (Linux x86_64)\n- `sqlfy-windows-amd64.zip` (Windows x86_64)\n\nEach zip contains the binary + README.md. The workflow automatically creates a GitHub Release with all files attached.\n\n**Users download from:** `https://github.com/paulushcgcj/sqlfy/releases`\n\n### Building a standalone binary locally\n\nTo build manually for your current platform:\n\n```bash\ncd cli\nbash build-binary.sh\n```\n\nThis creates `cli/dist/sqlfy-binary/sqlfy` (~35 MB) — a self-contained executable with zero dependencies.\n\n**To share:**\n1. Zip the binary: `tar -czf sqlfy-macos.tar.gz -C dist/sqlfy-binary sqlfy`\n2. Send `sqlfy-macos.tar.gz` to your user\n3. They extract and run: `tar -xzf sqlfy-macos.tar.gz \u0026\u0026 chmod +x sqlfy \u0026\u0026 ./sqlfy --help`\n\n**Cross-platform:** Build on macOS → works on macOS. Build on Linux → works on Linux.\n\n**Alternative (requires Python 3.11+):**\n```bash\ncd cli\npython -m build                    # creates wheel\npip install dist/sqlfy-*.whl       # install from wheel\n```\n\n---\n\n## CLI Reference\n\nSQLfy has 31 CLI subcommands covering schema reconstruction, graph visualization, impact analysis, linting, drift detection, domain analysis, RAG Q\u0026A, and more.\n\nSee the [full command reference on the wiki](https://github.com/paulushcgcj/sqlfy.wiki/wiki/commands/) for documentation on every command, including usage, flags, and examples.\n\n### Quick reference\n\n| Subcommand | Description |\n|---|---|\n| `dump` | Output the Schema State Dictionary |\n| `manifest` | Output graph manifest/metadata with high-level summary |\n| `chunks` | Output LLM vector chunks |\n| `diff` | Compare two Schema State Dictionaries or migration directories |\n| `diff-versions` | Compare two version snapshots from the same migration set |\n| `graph` | Graph representation (DOT, Mermaid, Excalidraw, Draw.io, JSON, HTML, report) |\n| `graph-migrations` | Visualize migration timeline and dependency graph |\n| `build-graph` | Build complete graphify-out/ directory (unified all-in-one) |\n| `rollback-analysis` | Analyze migration rollback feasibility and generate rollback scripts |\n| `lint` | Lint migration SQL for quality and style using sqlfluff |\n| `insights` | Analyse schema and report findings (orphan tables, missing PKs, etc.) |\n| `health` | Generate migration folder health report with quality score |\n| `simulate` | Simulate schema evolution with hypothetical migrations |\n| `integrity` | Check migration file integrity using SHA256 hashes |\n| `provenance` | Collect git provenance for migration files |\n| `cache` | Manage file-based caching system |\n| `ask` | Ask a natural language question about the schema (RAG) |\n| `chat` | Interactive multi-turn schema chat session |\n| `export` | Export schema as self-contained HTML documentation |\n| `query` | Deterministic graph queries (no LLM) |\n| `impact` | Analyze impact of schema object changes using graph traversal |\n| `lineage` | Column-level lineage and data flow analysis |\n| `domains` | Detect semantic business domains using community detection |\n| `stability` | Calculate schema stability metrics and churn rates |\n| `validate` | Validate migration ordering and detect issues |\n| `deps` | Analyze migration dependencies and detect circular dependencies |\n| `drift` | Detect schema drift between migration folders and generate repair SQL |\n| `classify` | Classify migrations by semantic category (table creation, data migration, cleanup, etc.) |\n| `naming` | Enforce migration filename naming conventions (Flyway pattern, description format) |\n| `cost` | Estimate migration execution cost (score, category, estimated_seconds) |\n| `safety` | Score migrations by safety level (SAFE / MEDIUM_RISK / HIGH_RISK / DANGEROUS) |\n\n**Common flags available on most commands:**\n- `--dialect oracle|postgres|mysql|sqlite` — SQL dialect (default: `oracle`)\n- `--at VERSION` — Point-in-time snapshot at a specific Flyway version\n- `--out FILE` — Write output to file instead of stdout\n- `--format` — Output format (varies by command)\n\n\n\u003e Use `sqlfy \u003csubcommand\u003e --help` for detailed usage.\n\n---\n\n\n## Development\n\n### App\n\n```bash\ncd app\nnpm install\nnpm run dev          # Vite dev server (browser, no Tauri)\nnpm run build        # production Vite build\nnpm run lint         # ESLint\nnpx tauri dev        # Tauri desktop window (requires Rust + cargo)\nnpx tauri build      # Tauri production bundle (.app / .exe / .deb)\n```\n\n### CLI\n\n```bash\ncd cli\npip install -e \".[dev]\"   # editable install + pytest\npython -m pytest -v       # run all tests\npython -m sqlfy ./samples # run directly without installing\n```\n\nTests read real `.sql` files from `samples/` and validate the parser, Reconstructor, and SchemaState builder end-to-end.\n\n### PyInstaller binary (for bundling with Tauri)\n\n```bash\ncd cli\npip install pyinstaller\npyinstaller --onefile src/sqlfy/main.py --name sqlfy\n# Output: dist/sqlfy  — copy to app/src-tauri/binaries/sqlfy-\u003ctarget-triple\u003e\n```\n\n---\n\n## How the App Uses the CLI\n\nThe desktop app (Tauri) and the browser dev mode use the CLI differently:\n\n```\nBrowser dev mode:\n  App (TypeScript) ──▶ app/src/core/core.ts  (in-process parser, no CLI)\n\nTauri desktop:\n  App (TypeScript) ──▶ app/src/bridge/cli.ts\n       │\n       ├─ writes migrations to a temp JSON file: [{ filename, sql }]\n       ├─ spawns CLI sidecar:  sqlfy --json-input \u003ctmp\u003e --all\n       └─ parses response JSON: { graph: {...}, chunks: [...] }\n```\n\n**Detection** — `app/src/bridge/cli.ts` checks `'__TAURI_INTERNALS__' in window` to decide which path to use.\n\n**CLI sidecar** — configured in `app/src-tauri/tauri.conf.json` under `externalBin`. The binary must be placed at `app/src-tauri/binaries/sqlfy-\u003ctarget-triple\u003e` before `npx tauri build`.\n\n**Output contract** — the CLI's `--all` flag produces:\n\n```json\n{\n  \"graph\":  { \"tables\": {}, \"sequences\": {}, \"edges\": [], \"migration_history\": [] },\n  \"chunks\": [{ \"id\": \"\", \"type\": \"\", \"title\": \"\", \"content\": \"\", \"metadata\": {}, \"hint\": \"\" }]\n}\n```\n\nThe TypeScript deserialiser in `cli.ts` maps `snake_case` keys to `camelCase` for the React component layer.\n\n---\n\n## Features\n\n### ① Migrations tab\n- Add, edit, or remove SQL migration files directly in the browser\n- Files are parsed in Flyway version order (`V1__`, `V2__`, …)\n- Supports multi-file sequences with incremental schema changes\n\n### ② Schema Graph tab\n- **ERD canvas** — topology-aware layout showing table nodes and FK edges\n- **Table detail panel** — per-table view of:\n  - Columns with data type, precision/scale, nullability, default value, and inline comment\n  - Constraint badges: `PK`, `NOT NULL`, `UNIQUE`, `FK`\n  - Outgoing and incoming FK relationships with `ON DELETE` action\n  - Indexes (including unique indexes) with version provenance\n  - Check constraints\n  - Migration action history per table (CREATE, ADD_COLUMN, MODIFY_COLUMN, …)\n- **Sequence list** — `START WITH` / `INCREMENT BY` metadata per sequence\n\n### ③ LLM Chunks tab\n- **Schema Summary** chunk — table count, column count, FK edge count, migration history, table role classification (root / junction / leaf / standalone)\n- **Per-table** chunks — full column inventory + constraint + relationship text in a structured, embedding-friendly format\n- **Relationship Graph** chunk — adjacency list of all FK edges for JOIN-path planning\n- One-click copy per chunk\n### ④ Ask tab\n- Natural-language Q\u0026A against your schema using RAG (Retrieval-Augmented Generation)\n- Choose retrieval strategy: local BM25 (no keys) or dense embeddings (requires API key)\n- Shows source chunks and provenance for transparency and reproducibility\n- Useful for quick schema discovery: \"Which tables lack a PRIMARY KEY?\", \"How do orders join to customers?\"\n\n### ⑤ Schema tab\n- Table explorer and compact schema panel with per-table details:\n  - Columns with data type, nullability, defaults, and inline comments\n  - Constraint, index, and FK badges with provenance\n  - Migration history for the selected table (CREATE / ALTER operations)\n- Includes a lightweight \"Run insights\" action to analyse the current schema from this panel\n\n### ⑥ Insights tab\n- Dedicated schema quality analysis panel powered by the `sqlfy insights` engine:\n  - Health score (0–100) and grade (A–D)\n  - Severity filter (Error / Warning / Info), category dropdown, and keyword search\n  - Expandable finding cards with full detail and suggested fix or SQL\n  - CLI-required: runs the Python CLI (`sqlfy insights --format json`) via Tauri or the dev-server proxy\n  - Browser-only mode shows a clear \"CLI required\" message and documentation on how to enable the CLI\n\n### ⑦ Graph Export tab\n- Export the schema graph to multiple formats: Mermaid, DOT, Excalidraw, Draw.io, JSON, HTML, or a human-readable summary\n- Advanced options: diagram title, layout resolution, `--no-split` subgraph behavior, and point-in-time `--at` version\n- Uses the CLI sidecar in Tauri or the dev-server proxy; browser fallback produces a limited in-process Mermaid/DOT rendering\n\n### ⑧ Simulate tab\n- Test hypothetical DDL changes against the current schema without modifying any files\n- Enter any SQL statement (DDL), optionally specify a base migration version (`--at`), and run a sandboxed simulation\n- Results show: safety badge (✓ Safe / ✕ Unsafe), breaking-change flag, health score, schema diff stats (tables/columns/sequences/relationships added or removed), and collapsible warnings list\n- Requires CLI (Tauri or Vite dev server) — not available in pure-browser mode\n\n---\n---\n\n## Supported DDL\n\n| Statement | Support |\n|---|---|\n| `CREATE TABLE` | ✅ columns, PK, FK, UNIQUE, CHECK |\n| `ALTER TABLE … ADD COLUMN` | ✅ |\n| `ALTER TABLE … ADD CONSTRAINT` | ✅ |\n| `ALTER TABLE … DROP COLUMN` | ✅ |\n| `ALTER TABLE … DROP CONSTRAINT` | ✅ |\n| `ALTER TABLE … MODIFY` | ✅ type, precision/scale, default, nullability |\n| `ALTER TABLE … RENAME COLUMN` | ✅ |\n| `CREATE [UNIQUE] INDEX` | ✅ |\n| `DROP TABLE` | ✅ |\n| `DROP INDEX` | ✅ |\n| `CREATE SEQUENCE` | ✅ |\n| `DROP SEQUENCE` | ✅ |\n| `COMMENT ON TABLE / COLUMN` | ✅ |\n\n---\n\n## LLM Usage\n\n\u003e [!IMPORTANT]\n\u003e **Vector embeddings require an API key.**\n\u003e The `ask` and `chat` subcommands support a `--embed` flag that switches from\n\u003e BM25 retrieval to dense vector search using [Voyage AI](https://voyageai.com)\n\u003e (model `voyage-3`, accessed via the Anthropic API).\n\u003e Set `ANTHROPIC_API_KEY` in your environment before using `--embed`.\n\u003e Without the flag, all retrieval is local BM25 — no key needed.\n\u003e\n\u003e TODO: evaluate whether to replace with a local embedding model (e.g. `nomic-embed-text`\n\u003e via Ollama) to remove the external dependency entirely.\n\nEach chunk is self-contained and human-readable. Example table chunk:\n\n```\nTABLE: APP.ORDERS\nSchema: APP | Created: V2\n\nCOLUMNS:\n  ORDER_ID: NUMBER(10) [PK, NOT NULL]\n  USER_ID: NUMBER(10) [NOT NULL, FK]\n  TOTAL_AMOUNT: NUMBER(12,2) [NOT NULL]\n  STATUS: VARCHAR2(20) [NOT NULL, DEFAULT PENDING]\n  CREATED_AT: TIMESTAMP [NOT NULL, DEFAULT SYSTIMESTAMP]\n\nREFERENCES (outgoing FK):\n  USER_ID) → APP.USERS(USER_ID) ON DELETE CASCADE [FK_ORDERS_USER]\n\nREFERENCED BY:\n  APP.ORDER_ITEMS.ORDER_ID → ORDER_ID\n\nINDEXES:\n  IDX_ORDERS_USER: (USER_ID) [V2]\n  IDX_ORDERS_STATUS: (STATUS, CREATED_AT) [V2]\n\nMIGRATION ACTIONS:\n  V2: CREATE TABLE APP.ORDERS\n```\n\nPaste the **Schema Summary** chunk as system context and individual **table chunks** as retrieval results for precise, grounded SQL generation.\n\n---\n\n## Tech Stack\n\n| Layer | Technology |\n|---|---|\n| Desktop UI | React 19 + Vite + Tauri 2 |\n| CLI | Python 3.11+ with sqlglot ≥25 (Oracle AST) |\n| Distribution | PyInstaller binary + Tauri desktop bundle |\n| Tests | pytest 9 |\n\n---\n\n## Roadmap\n\n- [x] Split into `app/` (React/Vite/Tauri) and `cli/` (Python)\n- [x] Shared `samples/` fixtures used by both the app and the test suite\n- [x] Migrate parser to **sqlglot** for full Oracle AST fidelity\n- [x] `DROP TABLE`, `DROP COLUMN`, `DROP CONSTRAINT`, `MODIFY COLUMN`, `RENAME COLUMN` support\n- [x] `SchemaState` dictionary — versioned, serialisable, fingerprinted snapshot\n- [x] YAML export of SchemaState (`sqlfy dump --format yaml`)\n- [x] Point-in-time reconstruction via `--at`\n- [x] Schema diff command (`sqlfy diff`)\n- [x] Graph output command (`sqlfy graph` — DOT, Mermaid, Excalidraw, Draw.io, JSON, HTML, report)\n- [x] Schema insights (`sqlfy insights` — orphan tables, missing PKs, FK candidates, circular refs, islands)\n- [x] Health report (`sqlfy health` — migration quality score)\n- [x] Schema simulator (`sqlfy simulate` — test what-if migrations)\n- [x] Migration integrity checks (`sqlfy integrity` — SHA256 hashing)\n- [x] File-based caching (`sqlfy cache`)\n- [x] Natural language queries (`sqlfy ask` — single-shot RAG)\n- [x] Interactive chat (`sqlfy chat` — multi-turn conversations)\n- [x] HTML documentation export (`sqlfy export`)\n- [x] Deterministic graph queries (`sqlfy query` — tables, columns, fk-path, refs, orphans, islands, cycles, missing-pk, indexes)\n- [x] Impact analysis (`sqlfy impact` — graph traversal for change impact)\n- [x] Manifest generation (`sqlfy manifest` — metadata summary)\n- [x] Community detection in graph exports (NetworkX + Louvain algorithm)\n- [ ] PostgreSQL dialect parity\n- [ ] Vector embeddings: evaluate replacing Voyage AI (`ANTHROPIC_API_KEY`) with a local model (e.g. Ollama `nomic-embed-text`) — see LLM Usage note above\n\n---\n\n## License\n\nMIT\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulushcgcj%2Fsqlfy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaulushcgcj%2Fsqlfy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulushcgcj%2Fsqlfy/lists"}