https://github.com/laxmimerit/ragwire
Production-grade RAG toolkit — ingest PDFs, DOCX, XLSX into Qdrant with LLM metadata extraction, hybrid search, and SHA256 deduplication.
https://github.com/laxmimerit/ragwire
crewai kgptalkie langchain qdrant rag rag-pipeline
Last synced: about 2 months ago
JSON representation
Production-grade RAG toolkit — ingest PDFs, DOCX, XLSX into Qdrant with LLM metadata extraction, hybrid search, and SHA256 deduplication.
- Host: GitHub
- URL: https://github.com/laxmimerit/ragwire
- Owner: laxmimerit
- License: mit
- Created: 2026-03-22T07:45:58.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2026-04-19T19:57:16.000Z (3 months ago)
- Last Synced: 2026-04-19T21:34:58.522Z (3 months ago)
- Topics: crewai, kgptalkie, langchain, qdrant, rag, rag-pipeline
- Language: Python
- Homepage: https://laxmimerit.github.io/RAGWire/
- Size: 5.14 MB
- Stars: 12
- Watchers: 0
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
RAGWire
Production-grade RAG toolkit for document ingestion and retrieval
---
## Features
- **Document Loading** — PDF, DOCX, XLSX, PPTX and more via MarkItDown
- **LLM Metadata Extraction** — extracts company, doc type, fiscal period using your LLM; fully customisable via YAML
- **Smart Text Splitting** — markdown-aware and recursive chunking strategies
- **Multiple Embedding Providers** — Ollama, OpenAI, HuggingFace, Google, FastEmbed
- **Qdrant Vector Store** — dense, sparse, and hybrid search
- **Advanced Retrieval** — similarity, MMR, and hybrid search with metadata filtering
- **SHA256 Deduplication** — at both file and chunk level
- **Directory Ingestion** — ingest an entire folder with one call, with optional recursive scan
- **Env Var Substitution** — use `${VAR}` in `config.yaml` for secrets
## Architecture
## Installation
```bash
pip install ragwire
# With Ollama support (local, no API key)
pip install "ragwire[ollama]"
# With all providers
pip install "ragwire[all]"
```
## Quick Start
```python
from ragwire import RAGWire
rag = RAGWire("config.yaml")
# Ingest files — SHA256 deduplication, safe to re-run
stats = rag.ingest_documents(["data/Apple_10k_2025.pdf", "data/Microsoft_10k_2025.pdf"])
print(f"Processed: {stats['processed']}, Skipped: {stats['skipped']}, Chunks: {stats['chunks_created']}")
# Or ingest an entire directory
stats = rag.ingest_directory("data/", recursive=True)
# Basic retrieval — returns list of LangChain Document objects
results = rag.retrieve("What is the total revenue?", top_k=5)
for doc in results:
print(doc.page_content[:300])
print(doc.metadata["company_name"]) # str, lowercased — e.g. "apple"
print(doc.metadata["fiscal_year"]) # list[int] — e.g. [2025] ← NOT a plain int
print(doc.metadata["file_name"]) # str — e.g. "Apple_10k_2025.pdf"
# Retrieval with explicit metadata filters
results = rag.retrieve(
"What is the net income?",
filters={"company_name": "apple", "fiscal_year": 2025} # pass year as int
)
# OR logic within a field — matches any of the listed values
results = rag.retrieve("Compare revenue trends", filters={"fiscal_year": [2023, 2024, 2025]})
# Agent-controlled filtering (recommended for AI agents)
filters = rag.extract_filters("Apple's revenue in 2025")
# → {"company_name": "apple", "fiscal_year": 2025} or None
results = rag.retrieve("Apple's revenue in 2025", filters=filters)
```
## Configuration
Copy `config.example.yaml` to `config.yaml` and edit. Secrets can be injected via environment variables:
```yaml
vectorstore:
url: "https://your-cluster.qdrant.io"
api_key: "${QDRANT_API_KEY}"
llm:
provider: "openai"
model: "gpt-5.4-nano"
api_key: "${OPENAI_API_KEY}"
```
Full example:
```yaml
embeddings:
provider: "ollama"
model: "qwen3-embedding:0.6b"
base_url: "http://localhost:11434"
llm:
provider: "ollama"
model: "qwen3.5:9b"
num_ctx: 16384
vectorstore:
url: "http://localhost:6333"
collection_name: "my_docs"
use_sparse: true
retriever:
search_type: "hybrid"
top_k: 5
auto_filter: false # set true to enable LLM-based filter extraction from every query
```
## Embedding Providers
```yaml
# Ollama (local)
embeddings:
provider: "ollama"
model: "qwen3-embedding:0.6b"
# OpenAI
embeddings:
provider: "openai"
model: "text-embedding-3-small"
# HuggingFace (local)
embeddings:
provider: "huggingface"
model_name: "sentence-transformers/all-MiniLM-L6-v2"
# Google
embeddings:
provider: "google"
model: "models/embedding-001"
```
## Component Usage
```python
from ragwire import (
MarkItDownLoader,
get_splitter,
get_markdown_splitter,
get_embedding,
QdrantStore,
MetadataExtractor,
hybrid_search,
mmr_search,
)
# Load a document
loader = MarkItDownLoader()
result = loader.load("document.pdf")
# Split text
splitter = get_markdown_splitter(chunk_size=10000, chunk_overlap=2000)
chunks = splitter.split_text(result["text_content"])
# Embeddings
embedding = get_embedding({"provider": "ollama", "model": "qwen3-embedding:0.6b"})
# Vector store
store = QdrantStore(config={"url": "http://localhost:6333"}, embedding=embedding)
store.set_collection("my_collection")
vectorstore = store.get_store()
```
## Architecture
```
ragwire/
├── core/ # Config loader + RAGWire orchestrator
├── loaders/ # MarkItDown document converter
├── processing/ # Text splitters + SHA256 hashing
├── metadata/ # Pydantic schema + LLM extractor
├── embeddings/ # Multi-provider embedding factory
├── vectorstores/ # Qdrant wrapper with hybrid search
├── retriever/ # Similarity, MMR, hybrid retrieval
└── utils/ # Logging
```
## Troubleshooting
| Error | Fix |
|-------|-----|
| Qdrant connection refused | `docker run -p 6333:6333 qdrant/qdrant` |
| `markitdown[pdf]` missing | `pip install "markitdown[pdf]"` |
| Ollama model not found | `ollama pull ` |
| `fastembed` missing | `pip install fastembed` (needed for hybrid search) |
| Embedding dimension mismatch | Set `force_recreate: true` in config once, then back to `false` |
## License
MIT © 2026 [KGP Talkie Private Limited](https://kgptalkie.com)
## Links
- 🌐 Website: [kgptalkie.com](https://kgptalkie.com)
- 📖 Docs: [laxmimerit.github.io/RAGWire](https://laxmimerit.github.io/RAGWire/)
- 💻 GitHub: [github.com/laxmimerit/ragwire](https://github.com/laxmimerit/ragwire)
- 📧 Email: udemy@kgptalkie.com