{"id":51311437,"url":"https://github.com/james-tn/cosmosdb-search-poc","last_synced_at":"2026-07-01T04:03:46.477Z","repository":{"id":367399448,"uuid":"1280648631","full_name":"james-tn/cosmosdb-search-poc","owner":"james-tn","description":"Advanced search on Azure Cosmos DB for NoSQL (full-text, vector, hybrid RRF) + RAG agent — agreements-domain POC","archived":false,"fork":false,"pushed_at":"2026-06-25T21:03:28.000Z","size":1898,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-25T21:20:21.667Z","etag":null,"topics":["azure-cosmos-db","azure-openai","full-text-search","hybrid-search","rag","semantic-search","streamlit","vector-search"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/james-tn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-25T19:53:25.000Z","updated_at":"2026-06-25T21:03:33.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/james-tn/cosmosdb-search-poc","commit_stats":null,"previous_names":["james-tn/cosmosdb-search-poc"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/james-tn/cosmosdb-search-poc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/james-tn%2Fcosmosdb-search-poc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/james-tn%2Fcosmosdb-search-poc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/james-tn%2Fcosmosdb-search-poc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/james-tn%2Fcosmosdb-search-poc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/james-tn","download_url":"https://codeload.github.com/james-tn/cosmosdb-search-poc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/james-tn%2Fcosmosdb-search-poc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34992075,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-01T02:00:05.325Z","response_time":130,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure-cosmos-db","azure-openai","full-text-search","hybrid-search","rag","semantic-search","streamlit","vector-search"],"created_at":"2026-07-01T04:03:44.528Z","updated_at":"2026-07-01T04:03:46.470Z","avatar_url":"https://github.com/james-tn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Advanced Search on Azure Cosmos DB for NoSQL — Agreements Search POC\n\nAn interactive proof-of-concept that runs **native search in Azure Cosmos DB for NoSQL**\nover a contract-management corpus of **agreements / envelopes**. It demonstrates the search\ncapabilities a search platform needs *today* **and** the AI-native differentiators\n(vector + hybrid search, RAG) that go beyond a legacy Elasticsearch deployment — all on a\nsingle operational database, with **no separate search service**.\n\n\u003e Built to evaluate functional parity (and upside) of Cosmos DB search vs. a legacy\n\u003e Elasticsearch platform. Everything below runs against a **real Cosmos DB account** and a\n\u003e real **Azure OpenAI `text-embedding-3-small`** deployment.\n\n![Search Explorer](docs/img/01-search-explorer.png)\n\n---\n\n## Capabilities demonstrated\n\n| Capability | What it shows | Cosmos DB feature used |\n|---|---|---|\n| 🔤 **Full-text search (BM25)** | Keyword/phrase relevance ranking | `ORDER BY RANK FullTextScore(c.content, …)` |\n| 🧠 **Vector / semantic search** | Find by *meaning* (NDAs without the word \"NDA\") | `VectorDistance(c.contentVector, @q)` over a **DiskANN** index |\n| 🔀 **Hybrid search** | Best of lexical + semantic, fused | `ORDER BY RANK RRF(VectorDistance(…), FullTextScore(…))` |\n| 🧩 **Boolean + range filters** | \"text + range in one query\" (the headline ES feature) | `WHERE c.status = … AND c.expirationDate \u003e= …` |\n| 🪆 **Nested clause / attribute queries** | Match inside `clauses[]` / `customAttributes[]` | `EXISTS (SELECT VALUE x FROM x IN c.clauses WHERE …)` |\n| ⌨️ **Typeahead / prefix** | Search-as-you-type on titles | `STARTSWITH(c.title, @prefix, true)` |\n| 📊 **Faceted aggregations** | Counts by type/status/account + date histogram + cardinality | `GROUP BY`, `COUNT(1)`, `LEFT(date,7)`, `COUNT(DISTINCT)` pattern |\n| ✨ **Highlighting** | Snippets with matched terms emphasised | application-layer (recommended pattern) |\n| 🤖 **RAG agent** | NL question → grounded, cited answer | hybrid retrieval → **Azure OpenAI Responses API** |\n\nEvery search in the UI also prints the **equivalent Cosmos DB SQL** it executed.\n\n---\n\n## Screens\n\n| Search Explorer (with equivalent SQL) | Faceted Analytics (GROUP BY) |\n|---|---|\n| ![](docs/img/01-search-explorer.png) | ![](docs/img/03-analytics.png) |\n\n| Compare all three capabilities | Contract Copilot (RAG) |\n|---|---|\n| ![](docs/img/02-compare.png) | ![](docs/img/04-agent.png) |\n\n---\n\n## Architecture\n\n```mermaid\nflowchart LR\n    U[User / Streamlit UI] --\u003e|query text| EMB[Azure OpenAI\u003cbr/\u003etext-embedding-3-small]\n    EMB --\u003e|1536-d vector| Q\n    U --\u003e|capability + filters| Q[cosmos_store.py\u003cbr/\u003equery builder]\n    Q --\u003e|\"FullTextScore / VectorDistance / RRF\u003cbr/\u003e+ filters + GROUP BY\"| COS[(Azure Cosmos DB\u003cbr/\u003efor NoSQL\u003cbr/\u003eDiskANN + Full-Text index)]\n    COS --\u003e|ranked agreements| U\n    U --\u003e|RAG: NL question| AG[agent.py]\n    AG --\u003e|hybrid retrieve| COS\n    AG --\u003e|context + question| RESP[Azure OpenAI\u003cbr/\u003eResponses API gpt-4.1-mini]\n    RESP --\u003e|grounded, cited answer| U\n```\n\n* **System of record *and* search** live in the same Cosmos DB container — no sync pipeline,\n  no separate search cluster to operate.\n* **Auth is Azure AD** end-to-end (`DefaultAzureCredential`); no keys in code.\n\nSee [`ARCHITECTURE.md`](ARCHITECTURE.md) for the data model, indexing policy, and the exact\nquery patterns. See [`WALKTHROUGH.md`](WALKTHROUGH.md) for a guided demo script.\n\n---\n\n## Prerequisites\n\n* **Python 3.9+**\n* **Azure CLI logged in**: `az login`. The app authenticates to **both** Cosmos DB and Azure\n  OpenAI with your identity. You need:\n  * **Cosmos DB Built-in Data Contributor** on the target account (data plane), and\n  * access to the Azure OpenAI endpoint (Cognitive Services user).\n* An **Azure Cosmos DB for NoSQL** account with the capabilities\n  `EnableNoSQLVectorSearch` and `EnableNoSQLFullTextSearch`.\n* An **Azure OpenAI** resource with `text-embedding-3-small` and a chat deployment\n  (e.g. `gpt-4.1-mini`).\n\n## Setup \u0026 run\n\n```bash\ncd cosmos-search-poc\npip install -r requirements.txt\n\ncp .env.example .env          # then edit .env with your resource names\naz login                      # authenticate (AAD)\n\n# 1) (first time only) provision the database + container with the search indexes\nCOSMOS_ACCOUNT=\u003caccount\u003e COSMOS_RG=\u003cresource-group\u003e ./provision/create_container.sh\n\n# 2) load sample agreements + embeddings into Cosmos DB (idempotent)\npython build_index.py\n\n# 3) launch the UI\nstreamlit run app.py          # or ./run.sh  → http://localhost:8501\n```\n\n### Optional: generate more sample data with an LLM\n```bash\npython generate_data.py --count 25 --seed 7   # writes data/generated_agreements.json\npython build_index.py                          # merges curated + generated, re-embeds\n```\n\n---\n\n## Project layout\n\n| Path | Purpose |\n|---|---|\n| `config.py` | Env-driven settings, AAD auth, Cosmos/OpenAI client factories |\n| `data_gen.py` | Curated agreement-domain sample data + predefined demo queries |\n| `generate_data.py` | LLM sample-data generator (Azure OpenAI → validated JSON) |\n| `embeddings.py` | Azure OpenAI `text-embedding-3-small` client |\n| `cosmos_store.py` | Query builders: FTS / vector / hybrid / filters / nested + GROUP BY + SQL emitter |\n| `agent.py` | RAG agent (hybrid retrieval + Responses API) |\n| `build_index.py` | Embed + upsert the corpus into Cosmos DB |\n| `app.py` | Streamlit UI (Search Explorer · Analytics · Copilot) |\n| `provision/` | `az` script + the exact vector/full-text/index policy JSON |\n| `tools/capture_screenshots.py` | Regenerate the docs screenshots (Playwright) |\n| `ARCHITECTURE.md`, `WALKTHROUGH.md` | Deep dive + guided demo |\n\n---\n\n## How this maps to the Elasticsearch feature inventory\n\n| Legacy Elasticsearch need | Demonstrated here |\n|---|---|\n| Text + range filters in one query | Scenario 1 (AWS renewal, last 6 months) |\n| Complex boolean filters | Scenario 4 (completed MSAs, account, expiry range) |\n| Nested document queries | Scenario 5 (auto-renewal clause, 60-day notice) |\n| Typeahead / `search_as_you_type` | Scenario 7 (title prefix) |\n| Facets / aggregations | Analytics panel (GROUP BY, histogram, cardinality) |\n| Highlights | App-layer snippet highlighting on every result |\n| **Semantic search (beyond ES keyword)** | Scenarios 2 \u0026 6 (vector) |\n| **Hybrid lexical + semantic (RRF)** | Scenarios 3 \u0026 8 (hybrid) |\n| **RAG / GenAI on operational data** | Contract Copilot tab |\n\n---\n\n## Notes \u0026 limitations\n\n* Cosmos DB full-text search is **GA**; **multi-language** and **fuzzy** matching are in\n  **preview**. `FullTextScore` is usable only in `ORDER BY RANK` and cannot be projected, so\n  result cards rank by position and **highlighting is done in the app layer** (the\n  recommended pattern).\n* `GROUP BY` with non-`VALUE` aggregates is executed by Cosmos DB per physical partition;\n  the Python SDK supports this for single-partition queries, so the Analytics panel runs the\n  real `GROUP BY` once per partition-key value and merges — exactly how the engine fans out.\n* `COUNT(DISTINCT)` is not a native single-query function; the cardinality metrics use the\n  documented `SELECT VALUE COUNT(1) FROM (SELECT DISTINCT VALUE c.field FROM c)` pattern.\n* The sample data is **synthetic and fictional**. Swap in your own by editing `data_gen.py`\n  (or `generate_data.py`) and re-running `build_index.py`.\n\n## Security\n\n* No credentials are committed. `.env` is gitignored; auth uses Azure AD by default.\n* The repo contains no Azure account names or endpoints — set them in your own `.env`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjames-tn%2Fcosmosdb-search-poc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjames-tn%2Fcosmosdb-search-poc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjames-tn%2Fcosmosdb-search-poc/lists"}