{"id":50278109,"url":"https://github.com/shivansh2904/vector-vault","last_synced_at":"2026-05-27T22:04:17.444Z","repository":{"id":360698077,"uuid":"1241244878","full_name":"Shivansh2904/vector-vault","owner":"Shivansh2904","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-27T13:26:08.000Z","size":302,"stargazers_count":0,"open_issues_count":14,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-27T15:05:43.057Z","etag":null,"topics":["docker","embeddings","faiss","fastapi","python","rag","react","semantic-search","sentence-transformers","typescript","vector-search"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Shivansh2904.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-17T06:17:03.000Z","updated_at":"2026-05-27T13:26:16.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Shivansh2904/vector-vault","commit_stats":null,"previous_names":["shivansh2904/vector-vault"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/Shivansh2904/vector-vault","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shivansh2904%2Fvector-vault","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shivansh2904%2Fvector-vault/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shivansh2904%2Fvector-vault/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shivansh2904%2Fvector-vault/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Shivansh2904","download_url":"https://codeload.github.com/Shivansh2904/vector-vault/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shivansh2904%2Fvector-vault/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33585230,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-27T02:00:06.184Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","embeddings","faiss","fastapi","python","rag","react","semantic-search","sentence-transformers","typescript","vector-search"],"created_at":"2026-05-27T22:03:57.792Z","updated_at":"2026-05-27T22:04:17.435Z","avatar_url":"https://github.com/Shivansh2904.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# VectorVault\n\n**Semantic search over your documents — runs entirely locally, no API keys, no data leaves your machine.**\n\n[![Python](https://img.shields.io/badge/Python-3.11-3776AB?style=flat-square\u0026logo=python\u0026logoColor=white)](https://python.org)\n[![FastAPI](https://img.shields.io/badge/FastAPI-0.109+-009688?style=flat-square\u0026logo=fastapi\u0026logoColor=white)](https://fastapi.tiangolo.com)\n[![FAISS](https://img.shields.io/badge/FAISS-1.7+-blue?style=flat-square)](https://github.com/facebookresearch/faiss)\n[![React](https://img.shields.io/badge/React-18-61DAFB?style=flat-square\u0026logo=react\u0026logoColor=black)](https://react.dev)\n[![TypeScript](https://img.shields.io/badge/TypeScript-5-3178C6?style=flat-square\u0026logo=typescript\u0026logoColor=white)](https://typescriptlang.org)\n[![Docker](https://img.shields.io/badge/Docker-ready-2496ED?style=flat-square\u0026logo=docker\u0026logoColor=white)](https://docker.com)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow?style=flat-square)](LICENSE)\n[![CI](https://img.shields.io/github/actions/workflow/status/Shivansh2904/vector-vault/ci.yml?style=flat-square\u0026label=CI)](https://github.com/Shivansh2904/vector-vault/actions)\n\n\u003cbr/\u003e\n\n*Upload PDFs, text files, Markdown, and Word documents. Ask questions in plain English. Get semantically ranked results — all without sending a single byte to any external service.*\n\n[Quick Start](#-quick-start) · [Architecture](#-architecture) · [API Reference](#-api-reference) · [Performance](#-performance)\n\n\u003c/div\u003e\n\n---\n\n## Why VectorVault?\n\nMost document search tools either require expensive API calls (OpenAI embeddings, Pinecone, etc.) or are just keyword search. VectorVault is different:\n\n| Feature | VectorVault | Keyword search | Cloud RAG |\n|---|---|---|---|\n| Semantic understanding | ✅ | ❌ | ✅ |\n| Runs offline | ✅ | ✅ | ❌ |\n| Zero cost | ✅ | ✅ | ❌ |\n| Privacy (data stays local) | ✅ | ✅ | ❌ |\n| Sub-50 ms search | ✅ | ✅ | ❌ |\n\n---\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│                         INDEXING PIPELINE                        │\n│                                                                   │\n│  PDF/TXT/MD/DOCX  ──▶  Chunker  ──▶  sentence-transformers      │\n│                        (512 words,     (all-MiniLM-L6-v2)        │\n│                         50 overlap)    384-dim embeddings         │\n│                                              │                    │\n│                                              ▼                    │\n│                                       FAISS FlatL2 Index          │\n│                                      (persisted to disk)          │\n└─────────────────────────────────────────────────────────────────┘\n\n┌─────────────────────────────────────────────────────────────────┐\n│                          QUERY PIPELINE                          │\n│                                                                   │\n│   Query string  ──▶  sentence-transformers  ──▶  FAISS search    │\n│                       (same model, L2-norm)      (top-K ANN)     │\n│                                                       │           │\n│                                                       ▼           │\n│                                              Ranked chunks        │\n│                                          (score, text, source)   │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n**Key design decisions:**\n- `FlatL2` index: exact nearest-neighbour search — no approximation errors, great for corpora up to ~1M vectors on modern hardware.\n- L2-normalised embeddings: makes L2 distance equivalent to cosine similarity.\n- Word-boundary chunking with 50-word overlap: prevents semantic context from being cut off at chunk boundaries.\n- Soft delete: FAISS doesn't support in-place row removal, so deleted chunks are tracked in a filter set.\n\n---\n\n## Features\n\n- **Multi-format ingestion** — PDF (via pypdf), plain text (.txt), Markdown (.md), and Word documents (.docx via python-docx)\n- **Smart chunking** — word-boundary aware, configurable chunk size and overlap\n- **Local embeddings** — `all-MiniLM-L6-v2` (22 M params, 384-dim, runs on CPU in milliseconds)\n- **FAISS vector index** — exact flat L2 search, persisted to disk across restarts\n- **Relevance scoring** — L2 distance converted to a [0, 1] similarity score\n- **REST API** — clean FastAPI backend with Pydantic-validated schemas and OpenAPI docs\n- **React UI** — dark-themed, responsive split-panel interface with upload, search, and document management\n- **Docker Compose** — one command to start the full stack\n- **CI/CD** — GitHub Actions pipeline covering both Python and TypeScript\n\n---\n\n## Tech Stack\n\n| Layer | Technology |\n|---|---|\n| Embeddings | `sentence-transformers` — all-MiniLM-L6-v2 |\n| Vector index | `faiss-cpu` — FlatL2 |\n| Document parsing | `pypdf` (PDF), `python-docx` (DOCX) |\n| Backend | `FastAPI` + `uvicorn` |\n| Data validation | `Pydantic v2` |\n| Frontend | `React 18` + `TypeScript 5` + `Vite` |\n| Styling | `Tailwind CSS v3` |\n| Containerisation | `Docker` + `Docker Compose` |\n| Web server | `nginx` (production frontend) |\n| CI | `GitHub Actions` |\n\n---\n\n## Deploy\n\n[![Deploy to Render](https://render.com/images/deploy-to-render-button.svg)](https://render.com/deploy?repo=https://github.com/Shivansh2904/vector-vault)\n\nA `render.yaml` blueprint provisions the backend on a Standard plan with a 5 GB persistent disk for the sentence-transformers model cache.\n\n---\n\n## Quick Start\n\n### Option A — Docker Compose (recommended)\n\n```bash\ngit clone https://github.com/Shivansh2904/vector-vault.git\ncd vector-vault\ndocker compose up --build\n```\n\n- Frontend: http://localhost:5173\n- Backend API: http://localhost:8000\n- API docs: http://localhost:8000/docs\n\n\u003e **Note:** The first build downloads the `all-MiniLM-L6-v2` model (~90 MB). Subsequent starts use the cached layer.\n\n### Option B — Manual setup\n\n**Backend:**\n```bash\ncd backend\npython -m venv .venv\nsource .venv/bin/activate  # Windows: .venv\\Scripts\\activate\npip install -r requirements.txt\nuvicorn main:app --reload --port 8000\n```\n\n**Frontend** (in a separate terminal):\n```bash\ncd frontend\nnpm install\nnpm run dev\n```\n\nVisit http://localhost:5173 — the dev server proxies API calls to port 8000.\n\n---\n\n## API Reference\n\n| Method | Endpoint | Description |\n|---|---|---|\n| `POST` | `/documents/upload` | Upload a document (multipart/form-data, field: `file`). Returns `doc_id`, `filename`, `chunk_count`. |\n| `GET` | `/documents` | List all indexed documents with metadata. |\n| `GET` | `/documents/{doc_id}` | Get metadata for a single document (`doc_id`, `filename`, `chunk_count`, `created_at`). Returns 404 if not found. |\n| `GET` | `/documents/{doc_id}/chunks` | Inspect the text chunks a document was split into. |\n| `DELETE` | `/documents/{doc_id}` | Remove a document and all its vectors from the index. |\n| `POST` | `/search` | Search the corpus. Body: `{\"query\": \"...\", \"top_k\": 5, \"rerank\": false, \"candidates_multiplier\": 3}`. When `rerank` is true, fetches `top_k * candidates_multiplier` candidates via the bi-encoder and re-scores them with a cross-encoder for higher accuracy. Returns ranked chunks with scores (and `rerank_score` when reranking). |\n| `GET` | `/health` | API status, total documents, total chunks, index vector count. |\n\nFull interactive docs available at `http://localhost:8000/docs` (Swagger UI) and `http://localhost:8000/redoc` (ReDoc).\n\n**Example — upload:**\n```bash\ncurl -X POST http://localhost:8000/documents/upload \\\n  -F \"file=@report.pdf\"\n# {\"doc_id\":\"3fa85f64-...\",\"filename\":\"report.pdf\",\"chunk_count\":42}\n```\n\n**Example — search:**\n```bash\ncurl -X POST http://localhost:8000/search \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"query\": \"what are the key risks?\", \"top_k\": 3}'\n```\n\n**Example — search with cross-encoder reranking:**\n```bash\ncurl -X POST http://localhost:8000/search \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"query\": \"what are the key risks?\", \"top_k\": 3, \"rerank\": true, \"candidates_multiplier\": 4}'\n```\n\n---\n\n## Two-Stage Retrieval (Bi-encoder + Cross-encoder)\n\nVectorVault supports the standard production search pattern of **two-stage retrieval**:\n\n1. **Stage 1 — Recall (bi-encoder):** the FAISS index returns a large candidate pool (`top_k * candidates_multiplier`) using cosine similarity on independently-embedded query and chunk vectors. This is fast — sub-millisecond per query against tens of thousands of vectors — but the bi-encoder never sees the query and chunk together, so its ranking is coarse.\n2. **Stage 2 — Precision (cross-encoder):** the candidate pool is re-scored by a cross-encoder (`cross-encoder/ms-marco-MiniLM-L-6-v2`) which takes `(query, chunk)` as a single input and outputs a relevance score. Cross-encoders are slower (no pre-computed vectors — every query-doc pair runs through the model) but materially more accurate.\n\nEnable it per-request by sending `\"rerank\": true`. The cross-encoder model is downloaded lazily on first use (~90 MB). When reranking is on, each result also includes a `rerank_score` alongside the original bi-encoder `score`.\n\n```\nQuery  ─▶  Bi-encoder  ─▶  FAISS  ─▶  N candidates  ─▶  Cross-encoder  ─▶  Top-K\n            (fast)                     (e.g. 15)         (accurate)         (e.g. 5)\n```\n\n**When to use it:** higher recall queries where the top result really matters (chatbot grounding, RAG, \"find the one paragraph that answers this\"). The latency cost is roughly 50–200 ms for 15–30 candidates on CPU.\n\n---\n\n## Project Structure\n\n```\nvector-vault/\n├── backend/\n│   ├── main.py           # FastAPI app, routes, request/response schemas\n│   ├── chunker.py        # Word-boundary text chunking utility\n│   ├── embedder.py       # sentence-transformers wrapper (L2-normalised)\n│   ├── reranker.py       # Cross-encoder reranker for two-stage retrieval\n│   ├── store.py          # FAISS index + metadata store with persistence\n│   ├── requirements.txt\n│   └── Dockerfile\n│\n├── frontend/\n│   ├── src/\n│   │   ├── App.tsx       # Main React application (split-panel UI)\n│   │   ├── api.ts        # Typed API client\n│   │   ├── main.tsx      # React entry point\n│   │   └── index.css     # Tailwind base + custom utilities\n│   ├── index.html\n│   ├── nginx.conf        # SPA-routing nginx config\n│   ├── package.json\n│   ├── vite.config.ts\n│   ├── tailwind.config.js\n│   ├── tsconfig.json\n│   └── Dockerfile        # Multi-stage Node build → nginx serve\n│\n├── data/                 # FAISS index + metadata (git-ignored, Docker volume)\n│   ├── index.faiss\n│   └── metadata.json\n│\n├── .github/\n│   └── workflows/\n│       └── ci.yml        # Parallel Python + TypeScript CI jobs\n│\n├── docker-compose.yml\n├── .gitignore\n├── LICENSE\n└── README.md\n```\n\n---\n\n## Performance\n\n| Operation | Typical latency | Notes |\n|---|---|---|\n| Embedding (single chunk) | ~5–15 ms | CPU, all-MiniLM-L6-v2 |\n| Embedding (batch, 32 chunks) | ~80–150 ms | CPU |\n| FAISS search (10K vectors) | \u003c 1 ms | FlatL2, exact |\n| FAISS search (1M vectors) | ~20–50 ms | FlatL2, exact |\n| Document upload (1-page PDF) | ~200–400 ms | Including embedding |\n| Document upload (100-page PDF) | ~3–8 s | Including embedding |\n\n**Scaling notes:**\n- For corpora over ~500K vectors, consider switching to `IndexIVFFlat` (approximate) or `IndexHNSW` for sub-millisecond search.\n- Embedding throughput scales linearly with CPU cores; a GPU would give ~10–50× speedup via `faiss-gpu`.\n- The `FlatL2` index memory footprint is `n_vectors × 384 × 4 bytes` — 1M vectors ≈ 1.5 GB RAM.\n\n---\n\n## Configuration\n\n| Environment variable | Default | Description |\n|---|---|---|\n| `DATA_DIR` | `data/` | Directory for FAISS index and metadata JSON |\n| `VITE_API_URL` | `http://localhost:8000` | Backend URL used by the frontend |\n\n---\n\n## Development\n\n**Run tests / linting:**\n```bash\n# Backend — syntax check all files\ncd backend \u0026\u0026 python -m py_compile main.py chunker.py embedder.py store.py\n\n# Frontend — type check + build\ncd frontend \u0026\u0026 npm run build\n```\n\n**API docs** are auto-generated at `/docs` and `/redoc` by FastAPI.\n\n---\n\n## License\n\nMIT © 2024 [Shivansh Mishra](https://github.com/Shivansh2904)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshivansh2904%2Fvector-vault","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshivansh2904%2Fvector-vault","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshivansh2904%2Fvector-vault/lists"}