{"id":51483540,"url":"https://github.com/zay-m3/naturalsql","last_synced_at":"2026-07-07T04:01:11.979Z","repository":{"id":345342295,"uuid":"1184816408","full_name":"Zay-M3/NaturalSQL","owner":"Zay-M3","description":"NaturalSQL: Lightweight Python library that converts SQL schema into vector context so LLMs can generate more accurate SQL from natural language.","archived":false,"fork":false,"pushed_at":"2026-04-19T18:08:19.000Z","size":1715,"stargazers_count":19,"open_issues_count":0,"forks_count":5,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-23T16:18:41.949Z","etag":null,"topics":["chromadb","mysql","postgresql","rag","sql","sqlite","sqlserver"],"latest_commit_sha":null,"homepage":"https://naturalsql.online/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Zay-M3.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-18T00:48:17.000Z","updated_at":"2026-04-20T00:58:13.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Zay-M3/NaturalSQL","commit_stats":null,"previous_names":["zay-m3/naturalsql"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/Zay-M3/NaturalSQL","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zay-M3%2FNaturalSQL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zay-M3%2FNaturalSQL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zay-M3%2FNaturalSQL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zay-M3%2FNaturalSQL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Zay-M3","download_url":"https://codeload.github.com/Zay-M3/NaturalSQL/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zay-M3%2FNaturalSQL/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35213821,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chromadb","mysql","postgresql","rag","sql","sqlite","sqlserver"],"created_at":"2026-07-07T04:01:11.145Z","updated_at":"2026-07-07T04:01:11.962Z","avatar_url":"https://github.com/Zay-M3.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NaturalSQL\n\nDocumentación en español: [README_SPANISH.md](README_SPANISH.md)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/naturalsql_logo.png\" alt=\"NaturalSQL Logo\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pypi.org/project/naturalsql/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/naturalsql?color=blue\" alt=\"PyPI version\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/naturalsql/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/pyversions/naturalsql\" alt=\"Python versions\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/naturalsql/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/dm/naturalsql\" alt=\"Downloads\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/Zay-M3/NaturalSQL/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/pypi/l/naturalsql\" alt=\"License\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nLightweight library to convert your SQL database schema into a vector database, so LLMs can use real table/column context to generate more accurate SQL from natural language.\n\n## Problem\n\nI wanted to interact with SQL databases through an LLM without pulling in large frameworks with many unrelated features.\n\n## Solution\n\nNaturalSQL extracts your database schema, vectorizes it using a configurable backend (`chroma` or `sqlite`) and configurable embedding provider (`local` or `gemini`), then performs semantic retrieval to return relevant schema context for your LLM.\n\n## Installation\n\n```bash\n# Base installation\npip install naturalsql\n\n# Chroma + local embeddings\npip install \"naturalsql[chroma-local]\"\n\n# SQLite + local embeddings\npip install \"naturalsql[sqlite-local]\"\n\n# SQLite + Gemini embeddings\npip install \"naturalsql[sqlite-gemini]\"\n\n# Chroma + Gemini embeddings\npip install \"naturalsql[chroma-gemini]\"\n\n# PostgreSQL driver support\npip install naturalsql[postgresql]\n\n# MySQL driver support\npip install naturalsql[mysql]\n\n# SQL Server driver support\npip install naturalsql[sqlserver]\n\n# All DB drivers\npip install naturalsql[all-db]\n```\n\n\u003e SQLite is included in the Python standard library.\n\u003e For Gemini, the current SDK is `google-genai` (import path: `google.genai`).\n\u003e Use environment variables for API keys (e.g. `GEMINI_API_KEY`), never hardcode secrets.\n\n## Supported SQL engines\n\n- PostgreSQL\n- MySQL\n- SQL Server\n- SQLite\n\n## Quick start\n\n```python\nfrom naturalsql import NaturalSQL\n\n# 1) Create an instance\nnsql = NaturalSQL(\n    db_url=\"postgresql://user:password@localhost:5432/mydb\",\n    db_type=\"postgresql\",\n)\n\n# 2) Build vector DB from schema\nresult = nsql.build_vector_db()\nprint(f\"Indexed tables: {result['indexed_documents']}\")\nprint(f\"From cache: {result['from_cache']}\")\n\n# 3) Retrieve relevant tables for a question\ntables = nsql.search(\"Show me sales from last month\")\n\n# 4) Use returned tables as LLM context\nfor table in tables:\n    print(table)\n```\n\n### Automatic cache\n\nThe first `search()` call loads the embedding model (~2-5s). Subsequent calls reuse the in-memory instance and typically run in ~10-15ms.\n\nSimilarly, `build_vector_db()` reuses existing vector storage when available (`forced_reset=False`) to avoid unnecessary reindexing.\n\n## E2E examples\n\n```python\nfrom naturalsql import NaturalSQL\n\n# A) Chroma + local (practical tuning for Chroma 1.5.x)\nnsql = NaturalSQL(\n    db_url=\"postgresql://user:pass@localhost:5432/mydb\",\n    db_type=\"postgresql\",\n    vector_backend=\"chroma\",\n    embedding_provider=\"local\",\n    vector_distance_threshold=1.6,  # If search() returns [], try 1.4-1.6\n)\n\nnsql.build_vector_db(storage_path=\"./metadata_vdb\", forced_reset=False)\ntables = nsql.search(\"sales for the last month\", limit=3)\nprint(tables)\n```\n\n```python\nimport os\nfrom naturalsql import NaturalSQL\n\n# B) SQLite + Gemini\nnsql = NaturalSQL(\n    db_url=\"sqlite:///./app.db\",\n    db_type=\"sqlite\",\n    vector_backend=\"sqlite\",\n    embedding_provider=\"gemini\",\n    gemini_api_key=os.environ[\"GEMINI_API_KEY\"],\n    gemini_embedding_model=\"gemini-embedding-2-preview\",\n)\n\nnsql.build_vector_db(storage_path=\"./metadata_vdb_sqlite\", forced_reset=False)\ntables = nsql.search(\"users with recent purchases\", limit=3)\nprint(tables)\n```\n\n## API\n\n### `NaturalSQL(**kwargs)`\n\nCreates an instance with DB and embedding configuration.\n\n| Parameter | Type | Default | Description |\n|---|---|---|---|\n| `db_url` | `str \\| None` | `None` | Database connection URL |\n| `db_type` | `str` | `\"\"` | Engine: `postgresql`, `mysql`, `sqlite`, `sqlserver` |\n| `db_normalize_embeddings` | `bool` | `True` | Normalize embedding vectors |\n| `device` | `str` | `\"cpu\"` | Embedding device: `cpu` or `cuda` |\n| `vector_backend` | `Literal[\"chroma\", \"sqlite\"]` | `\"chroma\"` | Vector backend |\n| `embedding_provider` | `Literal[\"local\", \"gemini\"]` | `\"local\"` | Embedding provider |\n| `gemini_api_key` | `str \\| None` | `None` | Required when `embedding_provider=\"gemini\"` |\n| `gemini_embedding_model` | `str` | `\"gemini-embedding-2-preview\"` | Gemini embedding model |\n| `vector_distance_threshold` | `float` | `1.0` | Max distance threshold used by `search()` filtering. For Chroma 1.5.x, if you get `[]`, try `1.4-1.6` (validated with `1.6`). |\n\n### `nsql.build_vector_db(...) -\u003e dict`\n\nConnects to the DB, extracts schema, and indexes it in the configured vector backend (`chroma` or `sqlite`).\n\n| Parameter | Type | Default | Description |\n|---|---|---|---|\n| `storage_path` | `str` | `\"./metadata_vdb\"` | Vector storage path |\n| `forced_reset` | `bool` | `False` | Rebuild vector collection from scratch |\n\nReturns:\n\n| Key | Type | Description |\n|---|---|---|\n| `storage_path` | `str` | Storage path used |\n| `indexed_documents` | `int` | Number of indexed tables |\n| `from_cache` | `bool` | `True` if existing vector store was reused |\n\n### `nsql.search(...) -\u003e list`\n\nRetrieves semantically relevant tables for a natural-language question.\n\n| Parameter | Type | Default | Description |\n|---|---|---|---|\n| `request` | `str` | required | Natural-language question |\n| `storage_path` | `str` | `\"./metadata_vdb\"` | Vector storage path |\n| `limit` | `int` | `3` | Maximum number of tables to return |\n\n### `build_prompt(relevant_tables, user_question) -\u003e str`\n\nHelper function in `naturalsql.utils.prompt` to create an LLM prompt using relevant tables + user question.\n\n```python\nfrom naturalsql.utils.prompt import build_prompt\n\nprompt = build_prompt(tables, \"Show me sales from last month\")\n```\n\n## How It Works Internally \n\nThink of NaturalSQL as a 5-step pipeline:\n\n1. Read DB structure with simple SQL/system queries.\n2. Normalize that structure into one common Python dictionary.\n3. Convert dictionary entries into semantic text documents.\n4. Turn those texts into embedding vectors.\n5. Store vectors in Chroma or SQLite and retrieve best matches for RAG.\n\n### 1) Schema extraction strategy by DB engine\n\nNaturalSQL does not parse SQL files. It asks the live database for metadata.\n\n- PostgreSQL: `information_schema.columns` for columns + joins over `information_schema.table_constraints`, `key_column_usage`, `constraint_column_usage` for foreign keys.\n- MySQL: `information_schema.columns` and `information_schema.key_column_usage` (scoped by `DATABASE()`).\n- SQL Server: `INFORMATION_SCHEMA.COLUMNS` for columns + `sys.foreign_key_columns` with `sys.tables`, `sys.schemas`, `sys.columns` for FK relationships.\n- SQLite: `sqlite_master` (table list), `PRAGMA table_info('\u003ctable\u003e')` for columns, `PRAGMA foreign_key_list('\u003ctable\u003e')` for relationships.\n\nEssential idea: most engines use `information_schema`; SQLite uses `PRAGMA`.\n\n### 2) Unified schema dictionary (same format for all engines)\n\nAfter extraction, data is normalized to one shape:\n\n```python\n{\n  \"tables\": {\n    \"public.users\": {\n      \"schema\": \"public\",\n      \"table\": \"users\",\n      \"columns\": [(\"id\", \"integer\"), (\"email\", \"text\")]\n    }\n  },\n  \"relationships\": [\n    {\n      \"from_schema\": \"public\",\n      \"from_table\": \"orders\",\n      \"from_column\": \"user_id\",\n      \"to_schema\": \"public\",\n      \"to_table\": \"users\",\n      \"to_column\": \"id\"\n    }\n  ]\n}\n```\n\nThis is what makes the next steps backend-agnostic.\n\n### 3) Dictionary to semantic documents (for embedding)\n\nThe extractor then generates documents for two kinds of knowledge:\n\n- `kind=table`: table + columns + data types\n- `kind=relationship`: FK direction between tables\n\nExample payload item:\n\n```python\n{\n  \"id\": \"table::public.users\",\n  \"content\": \"Schema: public. Table name: users. It has the following columns: id (integer), email (text)\",\n  \"metadata\": {\n    \"kind\": \"table\",\n    \"schema\": \"public\",\n    \"table\": \"users\"\n  }\n}\n```\n\nRelationship example:\n\n```python\n{\n  \"id\": \"rel::public.orders.user_id-\u003epublic.users.id\",\n  \"content\": \"Relationship: public.orders.user_id -\u003e public.users.id\",\n  \"metadata\": {\n    \"kind\": \"relationship\"\n  }\n}\n```\n\n### 4) Embedding generation\n\nNaturalSQL supports two embedding providers:\n\n- `local`: `sentence-transformers` (`all-MiniLM-L6-v2`)\n- `gemini`: `google-genai` (`gemini-embedding-2-preview` by default)\n\nThe system embeds:\n\n- documents with retrieval-document mode\n- user query with retrieval-query mode\n\nThis gives a vector for each schema document and one vector for each user question.\n\n### 5) Vector storage strategy (Chroma vs SQLite)\n\n#### Chroma backend\n\n- Uses `chromadb.PersistentClient(path=storage_path)`.\n- Stores `ids`, `documents`, `embeddings`, `metadatas` in collection `db_schema`.\n- Retrieval uses native vector query + metadata filter, for example `where={\"kind\": \"table\"}`.\n\n#### SQLite backend\n\n- Creates `vectors.db` under `storage_path`.\n- Table schema: `id`, `content`, `embedding` (JSON text), `metadata_json` (JSON text).\n- Retrieval loads rows and computes cosine distance in Python (NumPy if available, pure Python fallback if not).\n- Also filters by `kind` from `metadata_json`.\n\n### 6) Retrieval flow for RAG\n\nWhen you call `search(\"...\", limit=N)`, the flow is:\n\n1. Embed query text.\n2. Query `kind=table` and `kind=relationship` separately.\n3. Merge both result sets.\n4. Sort by distance (smaller is better).\n5. Apply `vector_distance_threshold`.\n6. Return top `N` schema contexts for your prompt.\n\nSo the LLM receives real schema context instead of guessing table/column names.\n\n### 7) Minimal end-to-end internal view\n\n```python\nfrom naturalsql.sql.sqlschema import SQLSchemaExtractor\nfrom naturalsql.controller.controllervector import VectorManager\n\n# 1) extract\nextractor = SQLSchemaExtractor(connection, db_type=\"postgresql\")\nschema_bundle = extractor.extract_schema()\n\n# 2) normalize -\u003e semantic docs\ndocuments_payload = extractor.formated_for_ia(schema_bundle)\n\n# 3) embed + persist (chroma or sqlite depending on config)\nvm = VectorManager(storage_path=\"./metadata_vdb\", force_reset=False, config=config)\nvm.index_documents(documents_payload)\n\n# 4) retrieve for RAG\ncontext_docs = vm.search_relevant_tables(\"sales from last month\", limit=3)\n```\n\n### Backend comparison\n\n| Aspect | Chroma | SQLite |\n|---|---|---|\n| Storage files | Chroma persistent directory | `vectors.db` file |\n| Vector query | Native Chroma ANN query | Python cosine distance over stored vectors |\n| Metadata filtering | Native `where` filter | JSON metadata filter in Python |\n| Dependency profile | Requires `chromadb` | Uses stdlib `sqlite3` + optional NumPy |\n| Operational style | Dedicated vector DB behavior | Lightweight embedded local store |\n\n### Current limitations\n\nThese points are based on current implementation behavior:\n\n- No hybrid ranking strategy yet: retrieval is distance-based over embeddings only (no lexical/BM25 rerank).\n- Relationship text is concise (`A.col -\u003e B.col`), so very complex semantics are not explicitly encoded.\n- SQLite backend computes similarity in Python, so very large corpora may be slower than native vector engines.\n- `build_vector_db()` returns `indexed_documents` as total indexed documents (tables + relationships), while wording may be interpreted as only tables.\n- Schema extraction is structural (tables/columns/FKs). It does not ingest business definitions, comments, or curated ontology unless you add them as extra documents.\n\nIn practice, this still provides strong RAG context for SQL generation because table fields and foreign-key topology are the highest-value grounding signals.\n\n## Tests\n\nSee [test/README.md](test/README.md) for testing instructions.\n\n## License\n\n[Apache License 2.0](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzay-m3%2Fnaturalsql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzay-m3%2Fnaturalsql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzay-m3%2Fnaturalsql/lists"}