{"id":51763101,"url":"https://github.com/rafaelpierre/agentic-text2sql","last_synced_at":"2026-07-19T16:04:16.347Z","repository":{"id":347819387,"uuid":"1195331874","full_name":"rafaelpierre/agentic-text2sql","owner":"rafaelpierre","description":"Sample repo showcasing agentic capabilities for Text2SQL use case","archived":false,"fork":false,"pushed_at":"2026-03-29T16:01:19.000Z","size":98,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-29T18:45:02.059Z","etag":null,"topics":["agents","bm25","pydantic-ai","pydantic-ai-agents","text2sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rafaelpierre.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-29T14:42:35.000Z","updated_at":"2026-03-29T16:01:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/rafaelpierre/agentic-text2sql","commit_stats":null,"previous_names":["rafaelpierre/agentic-text2sql"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/rafaelpierre/agentic-text2sql","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelpierre%2Fagentic-text2sql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelpierre%2Fagentic-text2sql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelpierre%2Fagentic-text2sql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelpierre%2Fagentic-text2sql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rafaelpierre","download_url":"https://codeload.github.com/rafaelpierre/agentic-text2sql/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelpierre%2Fagentic-text2sql/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35657587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-19T02:00:06.923Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","bm25","pydantic-ai","pydantic-ai-agents","text2sql"],"created_at":"2026-07-19T16:04:11.001Z","updated_at":"2026-07-19T16:04:16.331Z","avatar_url":"https://github.com/rafaelpierre.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# agentic-text2sql\n\nA fast, self-correcting Natural Language → SQL pipeline built with **[Pydantic AI](https://ai.pydantic.dev/)** as the agent orchestration framework and **Azure OpenAI** as the model backend.\n\n- **~8 seconds** end-to-end latency on a 10-table e-commerce schema\n- **Self-correcting SQL** via `ModelRetry` on `OperationalError`\n- **Zero LLM cost** for schema lookup — pure-Python FTS replaces the schema agent\n- **Two-model routing**: a nano model for cheap tasks, a full model only for SQL generation\n\n---\n\n## Architecture\n\n```mermaid\nflowchart TD\n    Q([\"User question\"]) --\u003e S1\n\n    subgraph S1 [\"Stage 1 — LLM Query Expansion  ~1s\"]\n        direction TB\n        E1[\"gpt-5.4-nano · max_tokens=150\"]\n        E2[\"Extract noun phrases\"]\n        E3[\"Fallback: NLTK stopword removal\"]\n        E1 --\u003e E2 --\u003e E3\n    end\n\n    S1 --\u003e|\"noun terms\"| S2\n\n    subgraph S2 [\"Stage 2 — Fast Schema Lookup  \u003c50ms  ·  no LLM\"]\n        direction TB\n        F1[\"asyncio.gather → parallel FTS per term\"]\n        F2[\"BM25 score merge · top-3 tables\"]\n        F3[\"FK neighbour expansion\"]\n        F4[\"DBML generation\"]\n        F1 --\u003e F2 --\u003e F3 --\u003e F4\n    end\n\n    S2 --\u003e|\"DBML schema snippet\"| S3\n\n    subgraph S3 [\"Stage 3 — SQL Agent  ~6-8s  ·  Pydantic AI\"]\n        direction TB\n        A1[\"gpt-5.3-chat · REASONING_EFFORT=low\"]\n        A2[\"Tool: execute_sql\"]\n        A3[\"ModelRetry on OperationalError\"]\n        A1 --\u003e A2 --\u003e A3\n        A3 --\u003e|\"retry with error context\"| A1\n    end\n\n    S3 --\u003e|\"SQL + result rows\"| S4\n\n    subgraph S4 [\"Stage 4 — Answer Narration  ~1s  optional\"]\n        direction TB\n        N1[\"gpt-5.4-nano · max_tokens=300\"]\n        N2[\"Plain-English answer\"]\n        N1 --\u003e N2\n    end\n\n    S4 --\u003e R([\"Answer + SQL + rows\"])\n```\n\n---\n\n## Quick Start\n\n### Prerequisites\n\n- Python ≥ 3.11\n- [uv](https://github.com/astral-sh/uv)\n- Azure OpenAI access with `gpt-5.4-nano` and `gpt-5.3-chat` deployed\n\n### Install\n\n```bash\ngit clone https://github.com/your-org/agentic-text2sql\ncd agentic-text2sql\nuv sync\n```\n\n### Configure\n\n```bash\ncp .env.example .env\n```\n\nEdit `.env` with your Azure credentials:\n\n```env\nAZURE_OPENAI_ENDPOINT=https://\u003cyour-resource\u003e.cognitiveservices.azure.com/\nAZURE_OPENAI_API_KEY=\u003cyour-key\u003e\nOPENAI_API_VERSION=2024-12-01-preview\n```\n\n### Seed and Index\n\n```bash\n# Populate meta-schema (table + column descriptions)\nuv run python cli.py seed-meta\n\n# Insert dummy e-commerce rows\nuv run python cli.py seed-data\n\n# Build FTS5 search index\nuv run python cli.py build-fts\n```\n\n### Ask a Question\n\n```bash\nuv run python cli.py ask \"Which products are in the Electronics category?\"\n\n# With verbose pipeline output\nuv run python cli.py ask \"Top 5 customers by lifetime spend\" --verbose\n\n# SQL only, no narration\nuv run python cli.py ask \"How many orders were placed last month?\" --no-narrate\n```\n\n---\n\n## CLI Reference\n\n| Command | Description |\n|---|---|\n| `seed-meta` | Populate meta-schema with e-commerce table/column descriptions |\n| `seed-data` | Insert dummy e-commerce rows into the data schema |\n| `build-fts` | Create / refresh FTS5 virtual tables |\n| `search \u003cquery\u003e` | Run BM25 FTS and print matching tables/columns with scores |\n| `ask \u003cquestion\u003e` | Run the full Text2SQL pipeline |\n| `dbml export` | Print DBML schema to stdout |\n\n---\n\n## Environment Variables\n\n| Variable | Default | Description |\n|---|---|---|\n| `AZURE_OPENAI_ENDPOINT` | — | Azure OpenAI resource endpoint |\n| `AZURE_OPENAI_API_KEY` | — | API key |\n| `OPENAI_API_VERSION` | — | API version (e.g. `2024-12-01-preview`) |\n| `FAST_MODEL` | `gpt-5.4-nano` | Model for query expansion and narration |\n| `SQL_MODEL` | `gpt-5.3-chat` | Model for SQL generation |\n| `ANSWER_MODEL` | `$FAST_MODEL` | Model for answer narration (defaults to FAST_MODEL) |\n| `REASONING_EFFORT` | `low` | `low` / `medium` / `high` — controls chain-of-thought |\n| `MAX_TOKENS_SQL` | `1024` | Token cap for SQL agent output |\n| `MAX_TOKENS_ANSWER` | `300` | Token cap for narration output |\n| `DB_URL` | `sqlite:///./text2sql.db` | SQLAlchemy database URL |\n\n---\n\n## Model Routing\n\n| Task | Model | Why |\n|---|---|---|\n| Query expansion (noun extraction) | `gpt-5.4-nano` | Simple extraction, ~150 tokens |\n| Answer narration | `gpt-5.4-nano` | Templated prose, low stakes |\n| SQL generation + execution | `gpt-5.3-chat` | Needs full reasoning, self-correction |\n\nThe schema lookup stage uses **no LLM** — it runs pure-Python BM25 FTS via SQLite FTS5.\n\n---\n\n## Eval Results\n\nTested on 10 Natural Language → SQL examples covering:\n`simple_filter`, `aggregation`, `multi-table join`, `subquery`, `window function`, `CTE`\n\n| Metric | Value |\n|---|---|\n| Average score | 92.7 / 100 |\n| Average latency | ~8s |\n| Self-correcting retries | ModelRetry on OperationalError |\n\nRun the eval yourself:\n\n```bash\nuv run python eval/run_eval.py\n# Results written to eval/eval_results.jsonl\n```\n\n---\n\n## Project Structure\n\n```\nagentic-text2sql/\n├── cli.py                          # Typer CLI entry point\n├── pyproject.toml\n├── .env                            # Azure credentials (not committed)\n├── eval/\n│   ├── eval_examples.jsonl         # 10 NL→SQL test cases\n│   └── run_eval.py                 # LLM-as-judge eval runner\n├── tests/\n│   ├── conftest.py\n│   ├── test_dbml_gen.py\n│   ├── test_fts.py\n│   └── test_retrieval.py\n└── text2sql_mvp/app/\n    ├── text2sql.py                 # Pipeline — main entry point\n    ├── fts.py                      # FTS5 index build + BM25 search\n    ├── dbml_gen.py                 # DBML schema generation\n    ├── meta_schema.py              # FK neighbour expansion, table registry\n    ├── retrieval.py                # Legacy LLM retrieval (unused in pipeline)\n    ├── seed_meta.py                # E-commerce schema definitions\n    ├── seed_data.py                # Dummy data rows\n    ├── db.py                       # SQLAlchemy engine factory\n    ├── data_schema.py              # SQLAlchemy table models\n    └── log.py                      # ANSI colour logging\n```\n\n---\n\n## Running Tests\n\n```bash\nuv run pytest\n```\n\nAll 29 tests run against an in-memory SQLite database and do not require Azure credentials.\n\n---\n\n## How Self-Correction Works\n\nThe SQL agent is given an `execute_sql` tool. When the generated SQL raises a `sqlalchemy.exc.OperationalError` (e.g. wrong column name, bad syntax), the tool raises `ModelRetry` with the error message. pydantic-ai automatically re-prompts the model with the full error context, allowing it to fix the SQL without any manual retry logic.\n\n```python\n@sql_agent.tool\nasync def execute_sql(ctx: RunContext[PipelineDeps], sql: str) -\u003e str:\n    try:\n        result = conn.execute(text(sql))\n        ctx.deps.rows = [dict(row) for row in result]\n        return json.dumps(ctx.deps.rows[:5], default=str)\n    except OperationalError as e:\n        raise ModelRetry(f\"SQL error: {e}\\nSchema:\\n{ctx.deps.dbml}\") from e\n```\n\n---\n\n## Appendix — Research Directions for Improving Agentic Text2SQL\n\nThe current pipeline scores ~92.7/100 on 10 eval queries with ~8s latency. The main bottleneck is the **one-shot BM25 retrieval** in Stage 2 — it's fast (\u003c50ms) but fragile: keyword mismatches (\"revenue\" won't match `total_amount`), no semantic understanding, and fixed top-3 table selection. Below are 5 research directions, prioritized by expected impact.\n\n### Current Pipeline Weaknesses (from code analysis)\n\n1. **BM25 keyword mismatch** — FTS5 matches literal tokens. Lemmatization helps but doesn't bridge synonyms.\n2. **Fixed top-k=3 tables** — Simple queries may need 1 table; complex queries may need 5+.\n3. **No column-level precision** — DBML dumps ALL columns for matched tables, increasing prompt noise.\n4. **No retrieval feedback loop** — On SQL failure, `ModelRetry` re-prompts the agent with the *same* schema. If a table/column is missing from the DBML, it can never self-correct.\n5. **Flat query expansion** — LLM extracts nouns but doesn't reason about relationships or domain semantics.\n\n### Direction 1: Hierarchical Retrieval\n\nReplace the single-pass BM25 with a multi-level cascade:\n\n- **Level 1 — Coarse**: BM25 over `table_registry` → top-5 candidate tables (broad recall)\n- **Level 2 — Fine-grained**: For each candidate, BM25/semantic search over its `column_registry`. Prune tables whose best column score is below a threshold.\n- **Level 3 — Relationship expansion**: FK-walk from surviving tables, but only add neighbors if they contribute relevant columns (not blind expansion like today)\n- **Level 4 — Schema assembly**: Generate DBML with relevant columns annotated or irrelevant ones pruned\n\n**Files**: `retrieval.py`, `fts.py`, `dbml_gen.py`\n\n### Direction 2: Hybrid Retrieval (BM25 + Embeddings)\n\nAdd a dense embedding index alongside BM25 for semantic matching.\n\n- Embed `search_doc` for each table/column at seed time (e.g. `text-embedding-3-small` or local `sentence-transformers`)\n- At query time, fuse BM25 scores with cosine similarity via Reciprocal Rank Fusion (RRF)\n- Directly fixes the synonym problem: \"revenue\" will be semantically close to \"total_amount\"\n- Can use `sqlite-vec` extension for in-process vector search (no new infra)\n\n**Files**: New `embedding.py`, modify `fts.py`, `retrieval.py`, `meta_schema.py`\n\n### Direction 3: Retrieval-Augmented Self-Correction (Closed-Loop Retrieval)\n\nWhen the SQL agent fails with \"no such table/column\", feed the error back to **retrieval**, not just the SQL agent.\n\n- Parse `OperationalError` messages for missing entities\n- Trigger a **targeted re-retrieval** for that specific table/column\n- Expand the DBML and re-run the SQL agent with enriched context\n\n**Files**: `text2sql.py`, `retrieval.py`\n\n### Direction 4: Schema-Aware Query Decomposition\n\nFor complex questions, decompose into sub-questions, retrieve schema per sub-question, then compose.\n\nExample: *\"Average order value for customers in New York who bought Electronics\"*\n- Sub-Q1: \"customers in New York\" → `users` (city)\n- Sub-Q2: \"Electronics products\" → `products` (category)\n- Sub-Q3: \"average order value\" → `orders` (total_amount)\n- Compose: verify JOIN paths exist via FK graph\n\n**Files**: `retrieval.py`, `text2sql.py`\n\n### Direction 5: Few-Shot Example Retrieval\n\nRetrieve similar previously-successful NL→SQL pairs as few-shot examples for the SQL agent.\n\n- Build an example bank from eval/production runs: (question, SQL, score)\n- At query time, embed the question → find top-2 nearest examples → inject into system prompt\n- Proven technique in text2sql literature; especially helps with CTEs, window functions\n\n**Files**: New `example_store.py`, modify `text2sql.py`\n\n### Recommended Priority\n\n| Priority | Direction | Expected Gain | Complexity | Latency Impact |\n|----------|-----------|---------------|------------|----------------|\n| 1 | **Hierarchical Retrieval** | Medium-high | Medium | +10-20ms |\n| 2 | **Hybrid BM25+Embeddings** | High | Medium | +50-100ms |\n| 3 | **Few-Shot Example Retrieval** | Medium-high | Low-medium | +100-200ms |\n| 4 | **Closed-Loop Retrieval** | Medium | Low | +0ms (only on retry) |\n| 5 | **Query Decomposition** | High (complex Qs) | High | +1-2s |\n\n**Recommendation**: Start with **Direction 1** (hierarchical retrieval) — it improves the existing BM25 pipeline with no new dependencies. Then layer **Direction 2** (embeddings) on top for semantic matching. **Direction 4** (closed-loop retrieval) is a quick parallel win.\n\n### Verification\n\n1. **Expand eval set** from 10 → 25-30 examples: add synonym-heavy queries, ambiguous column references, 4+ table joins, date arithmetic, negation queries\n2. **Add retrieval-specific metrics**: Table Recall@k, Column Precision — measure retrieval quality independently from SQL generation\n3. **A/B comparison**: Run old vs. new retrieval on same eval set, compare scores and latency\n4. **Track retrieval latency** separately from end-to-end latency\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frafaelpierre%2Fagentic-text2sql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frafaelpierre%2Fagentic-text2sql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frafaelpierre%2Fagentic-text2sql/lists"}