{"id":49782401,"url":"https://github.com/laxmimerit/ragwire","last_synced_at":"2026-05-11T22:05:23.637Z","repository":{"id":346166877,"uuid":"1188526114","full_name":"laxmimerit/RAGWire","owner":"laxmimerit","description":"Production-grade RAG toolkit — ingest PDFs, DOCX, XLSX into Qdrant with LLM metadata extraction, hybrid search, and SHA256 deduplication.","archived":false,"fork":false,"pushed_at":"2026-04-19T19:57:16.000Z","size":5394,"stargazers_count":12,"open_issues_count":0,"forks_count":3,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-19T21:34:58.522Z","etag":null,"topics":["crewai","kgptalkie","langchain","qdrant","rag","rag-pipeline"],"latest_commit_sha":null,"homepage":"https://laxmimerit.github.io/RAGWire/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/laxmimerit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-22T07:45:58.000Z","updated_at":"2026-04-19T19:57:19.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/laxmimerit/RAGWire","commit_stats":null,"previous_names":["laxmimerit/ragwire"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/laxmimerit/RAGWire","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laxmimerit%2FRAGWire","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laxmimerit%2FRAGWire/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laxmimerit%2FRAGWire/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laxmimerit%2FRAGWire/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/laxmimerit","download_url":"https://codeload.github.com/laxmimerit/RAGWire/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laxmimerit%2FRAGWire/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32914570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-11T17:09:15.040Z","status":"ssl_error","status_checked_at":"2026-05-11T17:08:45.420Z","response_time":120,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crewai","kgptalkie","langchain","qdrant","rag","rag-pipeline"],"created_at":"2026-05-11T22:05:17.600Z","updated_at":"2026-05-11T22:05:23.632Z","avatar_url":"https://github.com/laxmimerit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/laxmimerit/RAGWire/main/assets/ragwire.png\" alt=\"RAGWire logo\" width=\"120\"/\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003eRAGWire\u003c/h1\u003e\n\u003cp align=\"center\"\u003eProduction-grade RAG toolkit for document ingestion and retrieval\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pypi.org/project/ragwire\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/ragwire\" alt=\"PyPI\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/laxmimerit/ragwire/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-MIT-green\" alt=\"License\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://youtube.com/kgptalkie\"\u003e\u003cimg src=\"https://img.shields.io/badge/YouTube-KGP%20Talkie-red\" alt=\"YouTube\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://laxmimerit.github.io/RAGWire/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/📖%20Full%20Documentation-laxmimerit.github.io%2FRAGWire-blue?style=for-the-badge\u0026logo=readthedocs\u0026logoColor=white\" alt=\"Documentation\"/\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## Features\n\n- **Document Loading** — PDF, DOCX, XLSX, PPTX and more via MarkItDown\n- **LLM Metadata Extraction** — extracts company, doc type, fiscal period using your LLM; fully customisable via YAML\n- **Smart Text Splitting** — markdown-aware and recursive chunking strategies\n- **Multiple Embedding Providers** — Ollama, OpenAI, HuggingFace, Google, FastEmbed\n- **Qdrant Vector Store** — dense, sparse, and hybrid search\n- **Advanced Retrieval** — similarity, MMR, and hybrid search with metadata filtering\n- **SHA256 Deduplication** — at both file and chunk level\n- **Directory Ingestion** — ingest an entire folder with one call, with optional recursive scan\n- **Env Var Substitution** — use `${VAR}` in `config.yaml` for secrets\n\n## Architecture\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/laxmimerit/RAGWire/main/assets/RAGWire-block-diagram.png\" alt=\"RAGWire Architecture\" width=\"100%\"/\u003e\n\u003c/p\u003e\n\n## Installation\n\n```bash\npip install ragwire\n\n# With Ollama support (local, no API key)\npip install \"ragwire[ollama]\"\n\n# With all providers\npip install \"ragwire[all]\"\n```\n\n## Quick Start\n\n```python\nfrom ragwire import RAGWire\n\nrag = RAGWire(\"config.yaml\")\n\n# Ingest files — SHA256 deduplication, safe to re-run\nstats = rag.ingest_documents([\"data/Apple_10k_2025.pdf\", \"data/Microsoft_10k_2025.pdf\"])\nprint(f\"Processed: {stats['processed']}, Skipped: {stats['skipped']}, Chunks: {stats['chunks_created']}\")\n\n# Or ingest an entire directory\nstats = rag.ingest_directory(\"data/\", recursive=True)\n\n# Basic retrieval — returns list of LangChain Document objects\nresults = rag.retrieve(\"What is the total revenue?\", top_k=5)\nfor doc in results:\n    print(doc.page_content[:300])\n    print(doc.metadata[\"company_name\"])   # str, lowercased — e.g. \"apple\"\n    print(doc.metadata[\"fiscal_year\"])    # list[int] — e.g. [2025]  ← NOT a plain int\n    print(doc.metadata[\"file_name\"])      # str — e.g. \"Apple_10k_2025.pdf\"\n\n# Retrieval with explicit metadata filters\nresults = rag.retrieve(\n    \"What is the net income?\",\n    filters={\"company_name\": \"apple\", \"fiscal_year\": 2025}  # pass year as int\n)\n\n# OR logic within a field — matches any of the listed values\nresults = rag.retrieve(\"Compare revenue trends\", filters={\"fiscal_year\": [2023, 2024, 2025]})\n\n# Agent-controlled filtering (recommended for AI agents)\nfilters = rag.extract_filters(\"Apple's revenue in 2025\")\n# → {\"company_name\": \"apple\", \"fiscal_year\": 2025} or None\nresults = rag.retrieve(\"Apple's revenue in 2025\", filters=filters)\n```\n\n## Configuration\n\nCopy `config.example.yaml` to `config.yaml` and edit. Secrets can be injected via environment variables:\n\n```yaml\nvectorstore:\n  url: \"https://your-cluster.qdrant.io\"\n  api_key: \"${QDRANT_API_KEY}\"\n\nllm:\n  provider: \"openai\"\n  model: \"gpt-5.4-nano\"\n  api_key: \"${OPENAI_API_KEY}\"\n```\n\nFull example:\n\n```yaml\nembeddings:\n  provider: \"ollama\"\n  model: \"qwen3-embedding:0.6b\"\n  base_url: \"http://localhost:11434\"\n\nllm:\n  provider: \"ollama\"\n  model: \"qwen3.5:9b\"\n  num_ctx: 16384\n\nvectorstore:\n  url: \"http://localhost:6333\"\n  collection_name: \"my_docs\"\n  use_sparse: true\n\nretriever:\n  search_type: \"hybrid\"\n  top_k: 5\n  auto_filter: false   # set true to enable LLM-based filter extraction from every query\n```\n\n## Embedding Providers\n\n```yaml\n# Ollama (local)\nembeddings:\n  provider: \"ollama\"\n  model: \"qwen3-embedding:0.6b\"\n\n# OpenAI\nembeddings:\n  provider: \"openai\"\n  model: \"text-embedding-3-small\"\n\n# HuggingFace (local)\nembeddings:\n  provider: \"huggingface\"\n  model_name: \"sentence-transformers/all-MiniLM-L6-v2\"\n\n# Google\nembeddings:\n  provider: \"google\"\n  model: \"models/embedding-001\"\n```\n\n## Component Usage\n\n```python\nfrom ragwire import (\n    MarkItDownLoader,\n    get_splitter,\n    get_markdown_splitter,\n    get_embedding,\n    QdrantStore,\n    MetadataExtractor,\n    hybrid_search,\n    mmr_search,\n)\n\n# Load a document\nloader = MarkItDownLoader()\nresult = loader.load(\"document.pdf\")\n\n# Split text\nsplitter = get_markdown_splitter(chunk_size=10000, chunk_overlap=2000)\nchunks = splitter.split_text(result[\"text_content\"])\n\n# Embeddings\nembedding = get_embedding({\"provider\": \"ollama\", \"model\": \"qwen3-embedding:0.6b\"})\n\n# Vector store\nstore = QdrantStore(config={\"url\": \"http://localhost:6333\"}, embedding=embedding)\nstore.set_collection(\"my_collection\")\nvectorstore = store.get_store()\n```\n\n## Architecture\n\n```\nragwire/\n├── core/          # Config loader + RAGWire orchestrator\n├── loaders/       # MarkItDown document converter\n├── processing/    # Text splitters + SHA256 hashing\n├── metadata/      # Pydantic schema + LLM extractor\n├── embeddings/    # Multi-provider embedding factory\n├── vectorstores/  # Qdrant wrapper with hybrid search\n├── retriever/     # Similarity, MMR, hybrid retrieval\n└── utils/         # Logging\n```\n\n## Troubleshooting\n\n| Error | Fix |\n|-------|-----|\n| Qdrant connection refused | `docker run -p 6333:6333 qdrant/qdrant` |\n| `markitdown[pdf]` missing | `pip install \"markitdown[pdf]\"` |\n| Ollama model not found | `ollama pull \u003cmodel-name\u003e` |\n| `fastembed` missing | `pip install fastembed` (needed for hybrid search) |\n| Embedding dimension mismatch | Set `force_recreate: true` in config once, then back to `false` |\n\n## License\n\nMIT © 2026 [KGP Talkie Private Limited](https://kgptalkie.com)\n\n## Links\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://laxmimerit.github.io/RAGWire/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/📖%20Documentation-Visit%20Docs-2ea44f?style=for-the-badge\u0026logo=gitbook\u0026logoColor=white\" alt=\"Documentation\"/\u003e\n  \u003c/a\u003e\n  \u0026nbsp;\n  \u003ca href=\"https://github.com/laxmimerit/ragwire\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/⭐%20GitHub-Star%20the%20Repo-181717?style=for-the-badge\u0026logo=github\u0026logoColor=white\" alt=\"GitHub\"/\u003e\n  \u003c/a\u003e\n  \u0026nbsp;\n  \u003ca href=\"https://youtube.com/kgptalkie\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/▶%20YouTube-KGP%20Talkie-FF0000?style=for-the-badge\u0026logo=youtube\u0026logoColor=white\" alt=\"YouTube\"/\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n- 🌐 Website: [kgptalkie.com](https://kgptalkie.com)\n- 📖 Docs: [laxmimerit.github.io/RAGWire](https://laxmimerit.github.io/RAGWire/)\n- 💻 GitHub: [github.com/laxmimerit/ragwire](https://github.com/laxmimerit/ragwire)\n- 📧 Email: udemy@kgptalkie.com\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaxmimerit%2Fragwire","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flaxmimerit%2Fragwire","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaxmimerit%2Fragwire/lists"}