{"id":30290780,"url":"https://github.com/patrickjaja/workoflow-rag","last_synced_at":"2025-10-29T20:06:09.917Z","repository":{"id":301320118,"uuid":"1007293479","full_name":"patrickjaja/workoflow-rag","owner":"patrickjaja","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-26T08:24:27.000Z","size":78,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-26T09:27:51.031Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/patrickjaja.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-23T19:08:25.000Z","updated_at":"2025-06-26T08:24:31.000Z","dependencies_parsed_at":"2025-06-26T09:38:33.240Z","dependency_job_id":null,"html_url":"https://github.com/patrickjaja/workoflow-rag","commit_stats":null,"previous_names":["patrickjaja/workoflow-rag"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/patrickjaja/workoflow-rag","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrickjaja%2Fworkoflow-rag","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrickjaja%2Fworkoflow-rag/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrickjaja%2Fworkoflow-rag/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrickjaja%2Fworkoflow-rag/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/patrickjaja","download_url":"https://codeload.github.com/patrickjaja/workoflow-rag/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrickjaja%2Fworkoflow-rag/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270786441,"owners_count":24644563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-16T23:55:24.955Z","updated_at":"2025-10-29T20:06:09.911Z","avatar_url":"https://github.com/patrickjaja.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hybrid Search RAG API\n\nA production-ready Retrieval-Augmented Generation (RAG) system with advanced hybrid search capabilities, combining semantic understanding with keyword precision for superior document retrieval. Designed for on-premise deployment and seamless n8n integration.\n\n## Features\n\n- **Hybrid Search**: Combines semantic (dense) and keyword (sparse) search for optimal results\n- **Advanced Ranking**: Multi-stage ranking with Reciprocal Rank Fusion and LLM-based reranking\n- **Multi-Format Support**: Processes PDF, TXT, CSV, and JSON files with format-specific strategies\n- **Intelligent Chunking**: Document-type specific chunking with configurable overlap\n- **Azure OpenAI Integration**: Uses text-embedding-3-large (3072 dimensions) and gpt-4o-mini\n- **MinIO Storage**: S3-compatible object storage with automatic deduplication\n- **Qdrant Vector Database**: High-performance vector search with named vectors\n- **RESTful API**: FastAPI-based endpoints with comprehensive error handling\n- **Docker Deployment**: Fully containerized with health checks and monitoring\n- **Enhanced Name Search**: Special handling for person/entity name queries\n\n\n## Architecture\n\n```\n┌─────────────┐     ┌─────────────┐     ┌─────────────┐\n│   FastAPI   │────▶│   MinIO     │     │   Qdrant    │\n│     API     │     │  Storage    │     │   Vector    │\n└─────────────┘     └─────────────┘     │  Database   │\n       │                                 └─────────────┘\n       │                                        ▲\n       ▼                                        │\n┌─────────────┐     ┌─────────────┐            │\n│ Unstructured│────▶│    Azure    │────────────┘\n│  Processor  │     │   OpenAI    │\n└─────────────┘     └─────────────┘\n```\n\n## Core Concepts\n\n### Hybrid Search Architecture\n\nThis system implements a sophisticated hybrid search approach that combines the strengths of both semantic and keyword-based search:\n\n#### 1. **Dense Vectors (Semantic Search)**\n- Uses Azure OpenAI's `text-embedding-3-large` model (3072 dimensions)\n- Captures semantic meaning and context\n- Excellent for conceptual queries and paraphrasing\n- Handles synonyms and related concepts naturally\n\n#### 2. **Sparse Vectors (Keyword Search)**\n- Uses Qdrant's built-in sparse vector implementation\n- Preserves exact keyword matching capabilities\n- Critical for technical terms, names, and specific identifiers\n- Ensures important keywords aren't lost in semantic abstraction\n\n### Document Ranking \u0026 Reranking Process\n\n```\n┌─────────────────┐\n│   User Query    │\n└────────┬────────┘\n         │\n    ┌────┴────┐\n    │ Process │\n    └────┬────┘\n         │\n    ┌────┴────┐              ┌────┴────┐\n    │  Dense  │              │ Sparse  │\n    │ Search  │              │ Search  │\n    │(Semantic)│             │(Keyword)│\n    └────┬────┘              └────┬────┘\n         │                        │\n         └──────────┬─────────────┘\n                     │\n              ┌──────┴──────┐\n              │     RRF     │\n              │   Fusion    │\n              └──────┬──────┘\n                     │\n              ┌──────┴──────┐\n              │  Optional   │\n              │ LLM Rerank  │\n              └──────┬──────┘\n                     │\n              ┌──────┴──────┐\n              │   Results   │\n              └─────────────┘\n```\n\n#### Stage 1: Initial Retrieval\n1. **Query Processing**: User query is simultaneously:\n   - Embedded into a dense vector for semantic search\n   - Tokenized into sparse vectors for keyword search\n\n2. **Parallel Search**: Both search methods run concurrently in Qdrant:\n   - Dense search finds semantically similar documents\n   - Sparse search finds keyword matches\n\n3. **Reciprocal Rank Fusion (RRF)**:\n   ```\n   RRF_score = Σ(1 / (k + rank_i))\n   ```\n   - Combines results from both searches\n   - `k=60` (constant) prevents bias toward top results\n   - Creates unified ranking preserving both semantic and keyword relevance\n\n#### Stage 2: LLM-Based Reranking (Optional)\n1. **Context Enrichment**: Top-K results are sent to GPT-4o-mini\n2. **Relevance Assessment**: LLM evaluates each result against the query\n3. **Smart Reordering**: Results are reranked based on:\n   - Contextual understanding\n   - Query intent matching\n   - Information completeness\n\n### Key Advantages\n\n#### 1. **Superior Retrieval Quality**\n- **Best of Both Worlds**: Captures both meaning and precision\n- **Robust to Query Variations**: Works with natural language and specific terms\n- **Context-Aware**: Understands document relationships and intent\n\n#### 2. **Enhanced Name Search**\n- Special handling for person/entity queries\n- Exact match prioritization for names\n- Prevents semantic drift for proper nouns\n\n#### 3. **Flexibility**\n- **Configurable Alpha Weight** (`HYBRID_ALPHA=0.7`): Tune semantic vs keyword importance\n- **Search Type Selection**: Choose hybrid, dense-only, or sparse-only per query\n- **Optional Reranking**: Balance speed vs accuracy based on use case\n\n#### 4. **Performance Optimization**\n- **Parallel Processing**: Dense and sparse searches run concurrently\n- **Batch Embeddings**: Efficient processing of multiple documents\n- **Fallback Strategies**: Graceful degradation if one method fails\n\n#### 5. **Document Intelligence**\n- **Format-Specific Processing**: Optimal handling for PDFs, CSVs, JSON, TXT\n- **Smart Chunking**: Preserves context with configurable overlap\n- **Metadata Preservation**: Maintains source, position, and type information\n\n## Quick Start\n\n### 1. Prerequisites\n\n- Docker and Docker Compose\n- Python 3.11+ (for local testing)\n- 8GB+ RAM recommended\n\n### 2. Clone and Setup\n\n```bash\ngit clone \u003crepository\u003e\ncd python-rag\n\n# Ensure .env file has your Azure credentials\n# (Already configured in the provided .env)\n```\n\n### 3. Start Services\n\n```bash\n# Start all services\n#docker-compose up -d\n\n# Check service health\ndocker-compose ps\n\n# View logs\ndocker-compose logs -f\n```\n\n### 4. Test the API\n\n```bash\n# Install test dependencies\npip install aiohttp\n\n# Run test suite\npython test_api.py\n```\n\n## API Endpoints\n\n### Health Check\n```bash\nGET /health\n```\n\n### Upload File\n```bash\nPOST /upload\nContent-Type: multipart/form-data\n\n# Example with curl:\ncurl -X POST -F \"file=@document.pdf\" http://localhost:8000/upload\n```\n\n### Search\n```bash\nPOST /search\nContent-Type: application/json\n\n{\n  \"query\": \"your search query\",\n  \"top_k\": 10,\n  \"search_type\": \"hybrid\"  # Options: \"hybrid\", \"dense\", \"sparse\"\n}\n```\n\n\n### Ask\n```bash\nPOST /ask\nContent-Type: application/json\n\n{\n  \"query\": \"your search query\"\n}\n```\n\n### Refresh Index\n```bash\nPOST /index/refresh\n```\n\n### Get Statistics\n```bash\nGET /stats\n```\n\n## n8n Integration\n\nThis API is designed to work as a tool in n8n workflows for RAG patterns.\n\n### n8n HTTP Request Node Configuration\n\n1. **Search Endpoint**:\n   - Method: POST\n   - URL: `http://your-host:8000/search`\n   - Body Type: JSON\n   - Body:\n     ```json\n     {\n       \"query\": \"{{ $json.query }}\",\n       \"top_k\": 10,\n       \"search_type\": \"hybrid\"\n     }\n     ```\n\n2. **Upload Endpoint**:\n   - Method: POST\n   - URL: `http://your-host:8000/upload`\n   - Body Type: Form-Data\n   - Send Binary Data: Yes\n\n### Example n8n Workflow\n\n```json\n{\n  \"nodes\": [\n    {\n      \"name\": \"RAG Search\",\n      \"type\": \"n8n-nodes-base.httpRequest\",\n      \"parameters\": {\n        \"method\": \"POST\",\n        \"url\": \"http://localhost:8000/search\",\n        \"jsonParameters\": true,\n        \"options\": {},\n        \"bodyParametersJson\": {\n          \"query\": \"{{ $json.userQuery }}\",\n          \"top_k\": 5\n        }\n      }\n    }\n  ]\n}\n```\n\n## Configuration\n\nKey settings in `.env` that control ranking behavior:\n\n```bash\n# Chunking\nCHUNK_SIZE=512          # Characters per chunk\nCHUNK_OVERLAP=50        # Overlap between chunks\n\n# Search \u0026 Ranking\nHYBRID_ALPHA=0.7        # Dense vs sparse weight (0.7 = 70% semantic, 30% keyword)\nTOP_K_RESULTS=10        # Final results to return\nRERANK_TOP_K=20         # Candidates for LLM reranking\nENABLE_RERANKING=true   # Toggle LLM-based reranking\nRRF_K=60               # Reciprocal Rank Fusion constant\n\n# Azure OpenAI\nAZURE_EMBEDDING_DEPLOYMENT=text-embedding-3-large\nAZURE_LLM_DEPLOYMENT=gpt-4o-mini\n```\n\n## Advanced Usage\n\n### Custom Filters\n\nSearch with metadata filters:\n\n```json\n{\n  \"query\": \"workshop\",\n  \"filters\": {\n    \"file_type\": \"pdf\",\n    \"filename\": \"Workshop_Manual.pdf\"\n  }\n}\n```\n\n### Batch Processing\n\nProcess multiple files from MinIO:\n\n```bash\n# Upload files to MinIO bucket\n# Then refresh index\ncurl -X POST http://localhost:8000/index/refresh\n```\n\n## Performance Optimization\n\n### Search Quality Tuning\n\n1. **Hybrid Alpha (`HYBRID_ALPHA`)**:\n   - `0.0`: Pure keyword search (best for exact matches)\n   - `0.5`: Balanced semantic and keyword\n   - `0.7`: Default - emphasizes semantic understanding\n   - `1.0`: Pure semantic search (best for concepts)\n\n2. **Chunk Configuration**:\n   - **Size**: Larger chunks (1024) preserve context, smaller (256) increase precision\n   - **Overlap**: Higher overlap (100) prevents boundary loss, lower (0) maximizes coverage\n\n3. **Reranking Strategy**:\n   - Enable for critical queries requiring highest accuracy\n   - Disable for real-time applications needing sub-second response\n   - Adjust `RERANK_TOP_K` to balance quality vs API costs\n\n### Performance Tips\n\n1. **Embedding Batch Size**: Adjust `EMBEDDING_BATCH_SIZE` for API rate limits\n2. **Concurrent Searches**: Hybrid search runs dense and sparse in parallel\n3. **Caching**: Results are cached for repeated queries\n4. **Index Optimization**: Regular index refresh maintains search quality\n\n## Monitoring\n\n### Check Logs\n```bash\n# All services\ndocker-compose logs -f\n\n# Specific service\ndocker-compose logs -f app\n```\n\n### MinIO Console\nAccess at: http://localhost:9001\n- Username: minioadmin\n- Password: minioadmin\n\n### Qdrant Dashboard\nAccess at: http://localhost:6333/dashboard\n\n## Troubleshooting\n\n### Services Not Starting\n```bash\n# Check Docker resources\ndocker system df\n\n# Restart services\ndocker-compose down\ndocker-compose up -d\n```\n\n### Slow Embeddings\n- Check Azure OpenAI rate limits\n- Reduce `EMBEDDING_BATCH_SIZE`\n- Enable request caching\n\n### Search Quality Issues\n\n1. **Poor Semantic Results**:\n   - Increase `HYBRID_ALPHA` toward 1.0\n   - Check embedding model deployment\n   - Verify chunk size isn't too small\n\n2. **Missing Exact Matches**:\n   - Decrease `HYBRID_ALPHA` toward 0.0\n   - Ensure sparse vectors are being generated\n   - Check tokenization isn't removing important terms\n\n3. **Irrelevant Results**:\n   - Enable reranking with `ENABLE_RERANKING=true`\n   - Increase `RERANK_TOP_K` for more candidates\n   - Adjust chunk overlap for better context\n\n## n8n Integration\n\nThe API provides REST endpoints that can be easily integrated with n8n workflows:\n\n1. Use HTTP Request nodes to interact with the API\n2. Available endpoints:\n   - `/search`: Search your knowledge base\n   - `/ask`: Get AI-powered answers\n   - `/stats`: Monitor your RAG system\n\n## Use Cases\n\n### Ideal For\n- **Technical Documentation**: Balances technical terms with conceptual search\n- **Knowledge Management**: Handles diverse query types from different users\n- **Customer Support**: Finds answers using both keywords and intent\n- **Research Libraries**: Combines citation search with topic exploration\n- **Enterprise Search**: Handles acronyms, names, and concepts equally well\n\n### Example Scenarios\n\n1. **Technical Query**: \"SSL certificate error\"\n   - Sparse search ensures \"SSL\" and \"certificate\" are found\n   - Dense search includes related concepts like \"TLS\" or \"security\"\n\n2. **Conceptual Query**: \"How to improve team communication\"\n   - Dense search dominates, finding semantically related content\n   - Sparse search still catches exact phrase matches\n\n3. **Name Search**: \"John Smith project updates\"\n   - Enhanced name detection prioritizes exact \"John Smith\" matches\n   - Semantic search finds related project content\n\n## Development\n\n### Local Development\n```bash\n# Install dependencies\npip install -r requirements.txt\n\n# Run locally (requires services running)\ncd app\nuvicorn main:app --reload\n```\n\n\n### Adding New File Types\n1. Extend `DocumentProcessor` in `services/document_processor.py`\n2. Add parsing logic for the new type\n3. Update chunking strategy if needed\n\n## Enterprise Advantages\n\n### Why Hybrid Search RAG?\n\n1. **Accuracy**: Traditional semantic-only RAG systems can miss critical exact matches (product codes, names, technical terms). Our hybrid approach ensures nothing is lost.\n\n2. **Flexibility**: Single embedding models can't handle all query types equally well. By combining approaches, we excel at both natural language questions and specific keyword searches.\n\n3. **Performance**: Parallel processing and intelligent caching provide fast responses even with large document collections.\n\n4. **Control**: On-premise deployment with configurable ranking weights gives you full control over search behavior and data security.\n\n5. **Integration**: REST API design makes it easy to integrate with existing workflows, especially n8n automation.\n\n## License\n\nThis project is provided as-is for on-premise deployment.\n\n## Support\n\nFor issues or questions:\n1. Check the logs first\n2. Ensure all services are healthy\n3. Verify Azure credentials are correct\n4. Check example data format matches your use case\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatrickjaja%2Fworkoflow-rag","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpatrickjaja%2Fworkoflow-rag","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatrickjaja%2Fworkoflow-rag/lists"}