{"id":50396231,"url":"https://github.com/FareedKhan-dev/scalable-rag-pipeline","last_synced_at":"2026-06-16T13:00:38.588Z","repository":{"id":345904482,"uuid":"1123662413","full_name":"FareedKhan-dev/scalable-rag-pipeline","owner":"FareedKhan-dev","description":"A scalable RAG platform combining LangGraph agents, hybrid retrieval (Vector+Graph), and Ray orchestration on Kubernetes.","archived":false,"fork":false,"pushed_at":"2025-12-27T18:14:56.000Z","size":2297,"stargazers_count":117,"open_issues_count":0,"forks_count":50,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-03-21T23:32:10.433Z","etag":null,"topics":["ai-agents","aws","langchain","multi-agent-systems","python","rag-pipeline"],"latest_commit_sha":null,"homepage":"https://medium.com/@fareedkhandev/1168dcd36260","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FareedKhan-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"docs/security.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-27T10:54:35.000Z","updated_at":"2026-03-21T07:15:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/FareedKhan-dev/scalable-rag-pipeline","commit_stats":null,"previous_names":["fareedkhan-dev/scalable-rag-pipeline"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/FareedKhan-dev/scalable-rag-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FareedKhan-dev%2Fscalable-rag-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FareedKhan-dev%2Fscalable-rag-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FareedKhan-dev%2Fscalable-rag-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FareedKhan-dev%2Fscalable-rag-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FareedKhan-dev","download_url":"https://codeload.github.com/FareedKhan-dev/scalable-rag-pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FareedKhan-dev%2Fscalable-rag-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34406824,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","aws","langchain","multi-agent-systems","python","rag-pipeline"],"created_at":"2026-05-30T21:01:21.423Z","updated_at":"2026-06-16T13:00:38.574Z","avatar_url":"https://github.com/FareedKhan-dev.png","language":"Python","funding_links":[],"categories":["Browse The Shelves"],"sub_categories":["Self-hosted AI dev stacks"],"readme":"# Enterprise Agentic RAG Platform\n\n![Architecture](https://img.shields.io/badge/Architecture-Event--Driven-blueviolet)\n![Orchestration](https://img.shields.io/badge/Orchestration-LangGraph%20%2B%20Ray-orange)\n![Infrastructure](https://img.shields.io/badge/Infrastructure-AWS%20EKS%20%2B%20Terraform-blue)\n![Compute](https://img.shields.io/badge/AI%20Compute-Nvidia%20A10G%20%2F%20vLLM-green)\n\n## Table of Contents\n1. [System Overview](#1-system-overview)\n2. [RAG Methodologies \u0026 Agentic Logic](#2-rag-methodologies--agentic-logic)\n3. [Prerequisites \u0026 Tooling](#3-prerequisites--tooling)\n4. [Phase 1: Infrastructure Initialization (Terraform)](#4-phase-1-infrastructure-initialization-terraform)\n5. [Phase 2: Cluster Bootstrapping (Kubernetes)](#5-phase-2-cluster-bootstrapping-kubernetes)\n6. [Phase 3: The Data Plane (Ray \u0026 Databases)](#6-phase-3-the-data-plane-ray--databases)\n7. [Phase 4: The Control Plane (API Deployment)](#7-phase-4-the-control-plane-api-deployment)\n8. [Phase 5: Data Ingestion Pipeline](#8-phase-5-data-ingestion-pipeline)\n9. [Validation \u0026 Testing](#9-validation--testing)\n10. [Cost Optimization \u0026 Scaling](#10-cost-optimization--scaling)\n11. [Troubleshooting](#11-troubleshooting)\n\n## 1. System Overview\n\nThis repository contains the source code and Infrastructure-as-Code (IaC) definitions for a production-grade **Retrieval-Augmented Generation (RAG)** system. Unlike standard RAG implementations, this platform utilizes an **Agentic Architecture** (via LangGraph) to perform multi-step reasoning, query expansion, and hybrid retrieval (Vector + Knowledge Graph).\n\n### High-Level Architecture\n\nThe system is decoupled into two primary processing planes:\n1.  **Control Plane (The Brain):** Handles HTTP requests, state management, agent orchestration, and business logic. Runs on low-cost CPU nodes.\n2.  **Data Plane (The Muscle):** Handles heavy compute tasks including LLM Inference, Embedding generation, and Graph Extraction. Runs on autoscaling GPU nodes via **Ray**.\n\n\u003cimg src=\"https://miro.medium.com/v2/resize:fit:4800/format:webp/1*UjVAQrrQcw5lKN4NuuhB1g.jpeg\" alt=\"RAG Platform Architecture\" /\u003e\n\n---\n\n## 2. RAG Methodologies \u0026 Agentic Logic\n\nThis platform implements advanced RAG techniques to solve common failure modes (hallucination, retrieval misses).\n\n### 2.1. The Planning Agent (`services/api/app/agents/`)\nInstead of a linear chain, we use **LangGraph** to model the RAG process as a state machine.\n*   **Planner Node:** Analyzes user intent. Decides whether to perform a direct answer, a retrieval, or use a tool (Code Interpreter).\n*   **Query Rewriter:** Uses an LLM to rewrite the user's query, resolving coreferences (e.g., changing \"How much does *it* cost?\" to \"How much does *Kubernetes* cost?\").\n*   **HyDE (Hypothetical Document Embeddings):** Generates a fake \"ideal\" answer, embeds it, and uses that vector to find real documents. This bridges the semantic gap between a question and a declarative statement.\n\n### 2.2. Hybrid Retrieval\n*   **Vector Search (Qdrant):** Uses **BGE-M3** embeddings (dense retrieval) to find semantically similar text chunks.\n*   **Graph Search (Neo4j):** Executes Cypher queries to find entities and their relationships (e.g., `(Entity A)-[RELATED_TO]-\u003e(Entity B)`). This captures structural knowledge that vector search misses.\n\n---\n\n## 3. Prerequisites \u0026 Tooling\n\nEnsure the following tools are installed on your workstation (Bastion Host):\n\n*   **AWS CLI (`v2.x`):** Configured with AdministratorAccess.\n*   **Terraform (`v1.5+`):** For infrastructure provisioning.\n*   **Kubectl (`v1.29+`):** For Kubernetes interaction.\n*   **Helm (`v3.x`):** For chart management.\n*   **Python (`3.10+`)**: For local scripting.\n*   **Docker:** For building container images.\n\n---\n\n## 4. Phase 1: Infrastructure Initialization (Terraform)\n\nWe use Terraform to provision the \"Hardware\" layer: VPC, EKS Control Plane, S3, and RDS.\n\n### 4.1. Remote State Setup (Manual Step)\nTerraform requires a backend to store the state file safely.\n1.  Log in to the **AWS Console**.\n2.  Navigate to **S3** and create a bucket named: `rag-platform-terraform-state-prod-001` (Must be unique globally).\n3.  Navigate to **DynamoDB** and create a table named `terraform-state-lock`.\n    *   **Partition Key:** `LockID` (String).\n\n### 4.2. Provisioning Resources\nNavigate to the infrastructure directory:\n```bash\ncd infra/terraform\n```\n\nInitialize the backend and providers:\n```bash\nterraform init\n```\n\nReview the execution plan. This will show creation of:\n*   **VPC:** 10.0.0.0/16 with 3 Public, 3 Private, and 3 Database subnets.\n*   **EKS:** Cluster named `rag-platform-cluster` (Version 1.29).\n*   **RDS:** Aurora Postgres Serverless v2.\n*   **IAM:** OIDC Providers and IRSA roles.\n\n```bash\nterraform plan -var=\"db_password=YourStrongPassword#123\" -out=tfplan\n```\n\nApply the infrastructure (Estimated time: 20 minutes):\n```bash\nterraform apply tfplan\n```\n\n### 4.3. Connection Configuration\nOnce Terraform completes, configure `kubectl` to communicate with the new cluster:\n```bash\naws eks update-kubeconfig --region us-east-1 --name rag-platform-cluster\n```\n\nVerify connectivity:\n```bash\nkubectl get nodes\n# Expected Output: ip-10-0-x-x.ec2.internal   Ready   \u003cnone\u003e   m6i.large\n```\n\n---\n\n## 5. Phase 2: Cluster Bootstrapping (Kubernetes)\n\nThe EKS cluster is currently empty. We need to install the core system controllers.\n\n### 5.1. Run Bootstrap Script\nExecute the helper script to install **Karpenter** (Autoscaler), **KubeRay Operator**, **External Secrets**, and **Ingress Controller**.\n\n```bash\ncd scripts\nchmod +x bootstrap_cluster.sh\n./bootstrap_cluster.sh\n```\n\n### 5.2. Configure Karpenter Provisioners\nKarpenter is responsible for analyzing unschedulable pods and spinning up EC2 instances dynamically.\n\n**Apply the CPU Provisioner (For API \u0026 System pods):**\n```bash\nkubectl apply -f infra/karpenter/provisioner-cpu.yaml\n```\n*   *Technical Detail:* This targets `m6i`, `c6i` instances and uses Spot pricing where available.\n\n**Apply the GPU Provisioner (For AI Inference):**\n```bash\nkubectl apply -f infra/karpenter/provisioner-gpu.yaml\n```\n*   *Technical Detail:* This targets `g5` (Nvidia A10G) instances. It creates a taint `nvidia.com/gpu=true:NoSchedule` to prevent non-AI pods from accidentally using expensive nodes.\n\n---\n\n## 6. Phase 3: The Data Plane (Ray \u0026 Databases)\n\nWe now deploy the \"Muscle\" of the system.\n\n### 6.1. Deploy Vector \u0026 Graph Databases\nIn a full production environment, you might use Terraform managed services (AWS Neptune / Qdrant Cloud), but for this setup, we deploy HA clusters inside K8s.\n\n```bash\n# Deploy Qdrant\nhelm upgrade --install qdrant deploy/helm/qdrant --namespace default\n\n# Deploy Neo4j\nhelm upgrade --install neo4j deploy/helm/neo4j --namespace default\n```\n\n### 6.2. Deploy Ray Cluster\nThe Ray Cluster consists of a Head Node (orchestrator) and Worker Groups.\n```bash\nkubectl apply -f deploy/ray/ray-cluster.yaml\n```\n*Verification:*\n```bash\nkubectl get pods -l ray.io/cluster=rag-ray-cluster\n# Wait until the 'ray-head' pod is Running.\n```\n\n### 6.3. Deploy Model Services (Ray Serve)\nWe deploy two separate Ray Services. These utilize the `ServeConfigV2` specification.\n\n**A. Embedding Service (BGE-M3):**\n```bash\nkubectl apply -f deploy/ray/ray-serve-embed.yaml\n```\n\n**B. LLM Service (vLLM / Llama-3-70B):**\nThis is the most resource-intensive step.\n```bash\nkubectl apply -f deploy/ray/ray-serve-llm.yaml\n```\n\n**What happens technically:**\n1.  The `RayService` CRD submits a request to the Ray Head.\n2.  Ray realizes it needs 1 GPU (`nvidia.com/gpu: 1` resource request).\n3.  The Ray Worker pod goes into `Pending` state.\n4.  **Karpenter** detects the pending pod, calls AWS Fleet API, and provisions a `g5.xlarge` instance.\n5.  Once the node joins (approx. 90s), the pod starts, downloads the weights (approx. 40GB) from HuggingFace, and initializes the vLLM engine (PagedAttention).\n\n---\n\n## 7. Phase 4: The Control Plane (API Deployment)\n\n### 7.1. Secret Management\nCreate the Kubernetes Secret containing database credentials and keys.\n```bash\nkubectl create secret generic app-env-secret \\\n  --from-literal=DATABASE_URL=\"postgresql+asyncpg://ragadmin:YourStrongPassword#123@rag-platform-cluster-postgres.cluster-xxxx.us-east-1.rds.amazonaws.com:5432/rag_db\" \\\n  --from-literal=REDIS_URL=\"redis://rag-redis-prod.xxxx.ng.0001.use1.cache.amazonaws.com:6379/0\" \\\n  --from-literal=NEO4J_PASSWORD=\"password\" \\\n  --from-literal=JWT_SECRET_KEY=\"$(openssl rand -hex 32)\" \\\n  --from-literal=QDRANT_HOST=\"qdrant\" \\\n  --from-literal=RAY_LLM_ENDPOINT=\"http://llm-service:8000/llm\" \\\n  --from-literal=RAY_EMBED_ENDPOINT=\"http://embed-service:8000/embed\"\n```\n\n### 7.2. Deploy the API\nDeploy the FastAPI application using Helm.\n```bash\nhelm upgrade --install api deploy/helm/api\n```\n\n### 7.3. Apply Ingress\nConfigure the Load Balancer to route traffic to the API.\n```bash\nkubectl apply -f deploy/ingress/nginx.yaml\n```\n*   *Note:* Get your Load Balancer DNS name via `kubectl get ingress`. Map your domain (CNAME) to this DNS.\n\n---\n\n## 8. Phase 5: Data Ingestion Pipeline\n\nThe system requires data to function. The ingestion pipeline is an asynchronous, distributed Ray Job.\n\n### 8.1. Bulk Upload to S3\nUpload your dataset (PDF, DOCX, HTML) to the S3 bucket created by Terraform.\n```bash\n# Retrieve bucket name\nBUCKET_NAME=$(cd infra/terraform \u0026\u0026 terraform output -raw s3_documents_bucket_name)\n\n# Run bulk uploader script\npython scripts/bulk_upload_s3.py ./data/finance_reports $BUCKET_NAME\n```\n\n### 8.2. Trigger Ingestion Job\nNormally triggered by S3 Events, we can manually submit the job to the Ray Cluster.\n\n1.  **Port Forward Ray Dashboard:**\n    ```bash\n    kubectl port-forward service/rag-ray-cluster-head-svc 8265:8265\n    ```\n\n2.  **Submit Job via Python SDK:**\n    ```bash\n    python -m pipelines.jobs.s3_event_handler\n    ```\n\n**Technical Workflow:**\n1.  **Ray Data** reads binaries from S3 lazily.\n2.  **MapBatches (CPU):** `unstructured` library parses PDFs (OCR via Tesseract if needed) and chunks text (512 tokens).\n3.  **MapBatches (GPU - Embed):** Chunks are sent to the `embed-service` Actor.\n4.  **MapBatches (GPU - Graph):** Chunks are sent to the `llm-service` to extract `(Subject, Predicate, Object)` tuples.\n5.  **Write:**\n    *   Vectors -\u003e Qdrant (Upsert).\n    *   Nodes/Edges -\u003e Neo4j (MERGE queries).\n\n---\n\n## 9. Validation \u0026 Testing\n\n### 9.1. Health Checks\nVerify the API connects to all subsystems.\n```bash\ncurl https://\u003cYOUR_ALB_DNS\u003e/health/readiness\n# Expected: {\"redis\": \"up\", \"neo4j\": \"up\"}\n```\n\n### 9.2. End-to-End Chat Test\nPerform a request to verify the Agentic flow (Authentication required).\n\n1.  **Obtain Token (Dev Mode):** Use the `jwt.py` utility or disable auth in `config.py` temporarily for testing.\n2.  **Curl Request:**\n    ```bash\n    curl -X POST https://\u003cYOUR_ALB_DNS\u003e/api/v1/chat/stream \\\n      -H \"Content-Type: application/json\" \\\n      -d '{\n        \"message\": \"Analyze the financial risks mentioned in the Q3 report.\",\n        \"session_id\": \"test-session-1\"\n      }'\n    ```\n\n---\n\n## 10. Cost Optimization \u0026 Scaling\n\nThe system uses aggressive scaling policies to minimize costs:\n\n1.  **Spot Instances:** `provisioner-gpu.yaml` is configured to request Spot instances (`karpenter.sh/capacity-type: spot`). This reduces GPU costs by ~70%.\n2.  **Scale-to-Zero:**\n    *   The **Ray Autoscaler** is configured in `ray-serve-llm.yaml` with `min_replicas: 1` (can be 0 for dev).\n    *   If `min_replicas` is 0 and no requests arrive, Ray kills the Pod.\n    *   **Karpenter** sees the node is empty (TTL 30s) and terminates the EC2 instance.\n\n---\n\n## 11. Troubleshooting\n\n*   **Pod Pending (Insufficient CPU/Mem):** Check `kubectl describe pod \u003cpod_name\u003e`. If it says `FailedScheduling`, check if Karpenter logs show `launching node`.\n*   **Ray Actor Death:** Check Ray Dashboard `http://localhost:8265`. Common issue is OOM (Out Of Memory) on the GPU. Decrease `max_num_seqs` in `llama-70b.yaml`.\n*   **Database Connection Refused:** Ensure Security Groups in `infra/terraform/vpc.tf` allow traffic on ports 5432 (Postgres), 6333 (Qdrant), and 7687 (Neo4j) from the EKS Subnet CIDR.\n\n---\n\n## 12. Contributing\n\n1.  Create a feature branch (`git checkout -b feature/amazing-feature`).\n2.  Commit your changes.\n3.  Run tests (`make test`).\n4.  Push to the branch.\n5.  Open a Pull Request.\n\n---\n\n## License\n\nDistributed under the MIT License. See `LICENSE` for more information.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFareedKhan-dev%2Fscalable-rag-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFareedKhan-dev%2Fscalable-rag-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFareedKhan-dev%2Fscalable-rag-pipeline/lists"}