{"id":27854844,"url":"https://github.com/muskanpaliwal/rag-tool-zenml","last_synced_at":"2025-05-04T09:58:57.704Z","repository":{"id":288259563,"uuid":"967388400","full_name":"MuskanPaliwal/rag-tool-zenml","owner":"MuskanPaliwal","description":"A ZenML-based RAG system for document Q\u0026A with multi-format support. My exploration project to understand how RAG systems work under the hood.","archived":false,"fork":false,"pushed_at":"2025-04-28T15:38:01.000Z","size":44,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-04T09:58:53.087Z","etag":null,"topics":["document-processor","embeddings","faiss","langchain-python","nlp","pyhton3","ragsys","search","sentence-transformers","vector-database","zenml","zenml-pipelines"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MuskanPaliwal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-16T11:33:30.000Z","updated_at":"2025-04-28T15:48:21.000Z","dependencies_parsed_at":"2025-04-16T15:44:21.247Z","dependency_job_id":"e1d36cd0-fd88-4553-a470-dae0fe79a7d8","html_url":"https://github.com/MuskanPaliwal/rag-tool-zenml","commit_stats":null,"previous_names":["muskanpaliwal/rag-tool-zenml"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MuskanPaliwal%2Frag-tool-zenml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MuskanPaliwal%2Frag-tool-zenml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MuskanPaliwal%2Frag-tool-zenml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MuskanPaliwal%2Frag-tool-zenml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MuskanPaliwal","download_url":"https://codeload.github.com/MuskanPaliwal/rag-tool-zenml/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252317081,"owners_count":21728526,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["document-processor","embeddings","faiss","langchain-python","nlp","pyhton3","ragsys","search","sentence-transformers","vector-database","zenml","zenml-pipelines"],"created_at":"2025-05-04T09:58:57.014Z","updated_at":"2025-05-04T09:58:57.688Z","avatar_url":"https://github.com/MuskanPaliwal.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ZenML RAG System\n\nA Retrieval-Augmented Generation (RAG) system built with ZenML pipelines for document question-answering.\n\n## Overview\n\nThis RAG system allows you to:\n\n1. Process documents (PDFs, DOCx, TXT, HTML, etc.) into a vector database\n2. Query the processed documents using natural language\n3. Retrieve the most relevant document chunks for your queries\n\n## Features\n\n- **Multi-format Document Support**: Process PDFs, Word documents, text files, HTML, and more\n- **Smart Text Chunking**: Split documents intelligently with customizable chunk sizes\n- **Efficient Embedding**: Generate embeddings using SentenceTransformers models\n- **Fast Vector Search**: Use FAISS for efficient similarity search\n- **Hybrid Search**: Combine semantic search with keyword matching for better results\n- **ZenML Integration**: Leverage ZenML for pipeline orchestration and reproducibility\n- **CLI Interface**: Simple command-line interface for document processing and querying\n\n## Project Structure\n\n```\nrag_system/\n├── src/\n│   ├── utils/\n│   │   ├── documents_processor.py   # Document loading and processing\n│   │   ├── text_splitter.py         # Text chunking\n│   │   └── vector_utils.py          # Vector operations utilities\n│   ├── models/\n│   │   └── embeddings.py            # Embedding models\n│   ├── data/\n│   │   └── vector_store.py          # Vector storage and retrieval\n│   └── pipelines/\n│       ├── document_pipeline.py     # Document processing pipeline\n│       └── query_pipeline.py        # Query pipeline\n├── rag_system.py                    # Main RAG system interface\n├── main.py                          # Command-line entry point\n└── README.md                        # Documentation\n```\n\n## Installation\n\n1. Clone the repository:\n\n```bash\ngit clone https://github.com/yourusername/rag-system.git\ncd rag-system\n```\n\n2. Install the required dependencies:\n\n```bash\npip install zenml langchain sentence-transformers faiss-cpu pypdf\n```\n\n3. For additional document format support:\n\n```bash\npip install unstructured\n```\n\n## Usage\n\n### Command Line Interface\n\nThe `main.py` script provides a simple command-line interface with three modes of operation:\n\n```bash\n# Process documents\npython main.py process --document-path path/to/documents/ --storage-path ./vector_db\n\n# Query documents\npython main.py query --storage-path ./vector_db --query \"What is the main topic of these documents?\"\n\n# Interactive mode (ask multiple questions)\npython main.py interactive --storage-path ./vector_db\n```\n\n### Options\n\n- `--document-path`, `-d`: Path to document or directory to process\n- `--storage-path`, `-s`: Path to store the vector database (default: temporary directory)\n- `--query`, `-q`: Query string for searching documents\n- `--chunk-size`: Size of document chunks (default: 1000)\n- `--chunk-overlap`: Overlap between chunks (default: 200)\n- `--top-k`, `-k`: Number of results to return for queries (default: 3)\n- `--embedding-model`, `-m`: Name of embedding model to use (default: \"all-MiniLM-L6-v2\")\n- `--hybrid-search`: Use hybrid search combining semantic and keyword matching\n\n### Programmatic Usage\n\nYou can also use the RAG system programmatically in your Python code:\n\n```python\nfrom rag_system import RAGSystem\n\n# Initialize RAG system\nrag = RAGSystem(storage_path=\"./vector_db\")\n\n# Process a document or directory\nresult = rag.process_documents(\n    document_path=\"path/to/documents/\",\n    chunk_size=1000,\n    chunk_overlap=200\n)\nprint(f\"Processed {result['num_chunks']} document chunks\")\n\n# Query the processed documents\nanswer = rag.query(\n    query=\"What is the main topic discussed in these documents?\",\n    top_k=3,\n    hybrid_search=True\n)\n\n# Print results\nfor result in answer['results']:\n    print(f\"Rank {result['rank']} (Score: {result['score']:.4f})\")\n    print(f\"Content: {result['content']}\")\n    print(f\"Source: {result['source']}\")\n```\n\n## Customization\n\n### Embedding Models\n\nYou can use different SentenceTransformers models by changing the `embedding_model` parameter:\n\n- `all-MiniLM-L6-v2` (default): Fast and balanced\n- `all-mpnet-base-v2`: Higher quality but slower\n- `paraphrase-multilingual-MiniLM-L12-v2`: For multilingual support\n\n### Vector Search\n\n- Change `index_type` to \"IP\" (Inner Product) for cosine similarity instead of L2 distance\n- Use `hybrid_search=True` to combine semantic search with keyword matching\n\n### Document Chunking\n\n- Modify `chunk_size` and `chunk_overlap` to optimize for your specific documents\n- For longer documents, increase chunk size\n- For technical documents, decrease chunk size and increase overlap\n\n## Integration with LLMs\n\nTo create a complete RAG system, integrate with an LLM:\n\n```python\nfrom rag_system import RAGSystem\nimport openai  # or any other LLM API\n\n# Initialize RAG system and process documents\nrag = RAGSystem(storage_path=\"./vector_db\")\n\n# Process documents if needed\nif not os.path.exists(\"./vector_db\"):\n    rag.process_documents(\"path/to/documents/\")\n\n# Query function with LLM integration\ndef answer_question(query, top_k=3):\n    # Get relevant context from RAG system\n    results = rag.query(query, top_k=top_k)\n    \n    # Prepare context for the LLM\n    context = \"\\n\\n\".join([r[\"content\"] for r in results[\"results\"]])\n    \n    # Create prompt with context\n    prompt = f\"Answer the question based on the following context:\\n\\nContext:\\n{context}\\n\\nQuestion: {query}\\n\\nAnswer:\"\n    \n    # Call LLM API\n    response = openai.ChatCompletion.create(\n        model=\"gpt-3.5-turbo\",\n        messages=[\n            {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n            {\"role\": \"user\", \"content\": prompt}\n        ]\n    )\n    \n    return {\n        \"answer\": response.choices[0].message[\"content\"],\n        \"sources\": [r[\"source\"] for r in results[\"results\"]]\n    }\n\n# Example usage\nresult = answer_question(\"What are the key benefits described in the document?\")\nprint(result[\"answer\"])\nprint(f\"Sources: {result['sources']}\")\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **FileNotFoundError**: Ensure the document path is correct and accessible.\n2. **Memory Issues**: For large documents, reduce batch size or chunk size.\n3. **CUDA Errors**: Set device to 'cpu' in the embeddings module if you encounter GPU-related errors.\n4. **Unsupported File Types**: Ensure you have the necessary dependencies for all file types (e.g., `unstructured` for Word documents).\n\n## License\n\nThis project is licensed under the MIT License.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuskanpaliwal%2Frag-tool-zenml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmuskanpaliwal%2Frag-tool-zenml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuskanpaliwal%2Frag-tool-zenml/lists"}