{"id":51595366,"url":"https://github.com/monkey-mode/rag-data-pipeline","last_synced_at":"2026-07-11T18:30:27.987Z","repository":{"id":364612101,"uuid":"1266113572","full_name":"monkey-mode/rag-data-pipeline","owner":"monkey-mode","description":"RAG data pipeline: FastAPI microservices, MinIO presigned uploads, Redis queue, LangChain + ChromaDB, Claude chat","archived":false,"fork":false,"pushed_at":"2026-06-13T17:46:39.000Z","size":68,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-13T19:22:40.541Z","etag":null,"topics":["chromadb","fastapi","langchain","minio","rag"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/monkey-mode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-11T10:30:54.000Z","updated_at":"2026-06-13T17:32:12.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/monkey-mode/rag-data-pipeline","commit_stats":null,"previous_names":["monkey-mode/rag-data-pipeline"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/monkey-mode/rag-data-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monkey-mode%2Frag-data-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monkey-mode%2Frag-data-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monkey-mode%2Frag-data-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monkey-mode%2Frag-data-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/monkey-mode","download_url":"https://codeload.github.com/monkey-mode/rag-data-pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monkey-mode%2Frag-data-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35372639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-11T02:00:05.354Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chromadb","fastapi","langchain","minio","rag"],"created_at":"2026-07-11T18:30:26.072Z","updated_at":"2026-07-11T18:30:27.980Z","avatar_url":"https://github.com/monkey-mode.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RAG Data Pipeline\n\nA fullstack hands-on RAG (Retrieval-Augmented Generation) data pipeline built as Python microservices. The core design principle is **separation of responsibilities**: the upload path never blocks on processing.\n\n## Architecture\n\nHigh-level sketch: [docs/architecture.excalidraw](docs/architecture.excalidraw) — open at [excalidraw.com](https://excalidraw.com) or with the VS Code Excalidraw extension.\n\n```mermaid\nsequenceDiagram\n    actor User\n    participant DocsAPI as documents-service\n    participant ChatsAPI as chats-service\n    participant RagWorker as rag-worker\n    participant MinIO\n    participant Redis\n    participant Postgres\n    participant ChromaDB\n    participant Claude as Claude API\n\n    Note over User,MinIO: STEP 1 — Presign \u0026 Direct Upload\n    User-\u003e\u003e+DocsAPI: POST /documents { filename }\n    DocsAPI-\u003e\u003ePostgres: INSERT documents (status=uploading)\n    DocsAPI-\u003e\u003eMinIO: Generate presigned PUT URL\n    DocsAPI--\u003e\u003e-User: { id, upload_url, status: \"uploading\" }\n    User-\u003e\u003eMinIO: PUT file bytes (never through the API)\n    MinIO--\u003e\u003eUser: 200 OK\n\n    Note over MinIO,Redis: STEP 2 — Bucket Notification → Queue\n    MinIO-\u003e\u003eRedis: RPUSH minio:events (s3:ObjectCreated)\n\n    Note over Redis,ChromaDB: STEP 3 — RAG Worker Processes\n    Redis--\u003e\u003eRagWorker: BLPOP minio:events\n    RagWorker-\u003e\u003ePostgres: UPDATE documents SET status=processing\n    RagWorker-\u003e\u003eMinIO: Download object {document_id}/{filename}\n    RagWorker-\u003e\u003eRagWorker: LangChain loader → chunk (500 chars, 50 overlap)\n    RagWorker-\u003e\u003eChromaDB: Delete old chunks, upsert new (embed locally)\n    RagWorker-\u003e\u003ePostgres: UPDATE documents SET status=ready, chunk_count=N\n\n    Note over User,DocsAPI: STEP 4 — Poll for Status\n    User-\u003e\u003e+DocsAPI: GET /documents/{id}\n    DocsAPI-\u003e\u003ePostgres: SELECT * FROM documents WHERE id=...\n    DocsAPI--\u003e\u003e-User: { status: \"ready\", chunk_count: N }\n\n    Note over User,Claude: STEP 5 — Chat (LCEL RAG chain)\n    User-\u003e\u003e+ChatsAPI: POST /query { question }\n    ChatsAPI-\u003e\u003eChromaDB: Embed question, similarity search\n    ChatsAPI-\u003e\u003e+Claude: Context chunks + question (prompt | llm | parser)\n    Claude--\u003e\u003e-ChatsAPI: Generated answer\n    ChatsAPI-\u003e\u003ePostgres: INSERT chats (question, answer, sources)\n    ChatsAPI--\u003e\u003e-User: { answer, sources: [chunks + distances] }\n```\n\nKey design points:\n\n- File bytes never pass through the API services — clients upload straight to MinIO via presigned URLs.\n- MinIO is configured (docker-compose.yml) to RPUSH put-events onto the Redis list `minio:events`; the worker BLPOPs it as a queue.\n- Object keys are `\u003cdocument_id\u003e/\u003cfilename\u003e`; the worker derives the document ID from the key to stamp status (`uploading → processing → ready|failed`) in Postgres.\n- Embeddings use Chroma's default local embedding function (all-MiniLM-L6-v2) — no API key needed for ingestion. The worker writes and the chats service queries the same `documents` collection, so both must use the same embedding function (the chats service wraps it in a LangChain `Embeddings` adapter).\n- The chats service composes retrieval + generation as one LCEL graph: `RunnableParallel(docs, question).assign(answer = context | prompt | ChatAnthropic | StrOutputParser)`.\n\n## Layout\n\n| Path | What it is |\n|---|---|\n| `services/documents/` | Async FastAPI (port 8001). Presigned upload slots, status tracking, and document management: download / reprocess / delete. Owns the `documents` table (creates it on startup). |\n| `services/rag_worker/` | Queue consumer: Redis event → fetch from MinIO → LangChain loaders (txt/md/pdf/html) → chunk → ChromaDB upsert → stamp status. |\n| `services/chats/` | FastAPI (port 8002). `POST /query` runs the LCEL RAG chain (retrieval + Claude answer); logs exchanges in its own `chats` table. Requires `ANTHROPIC_API_KEY`. |\n| `ui/` | Vanilla-JS UI (port 3000). `index.html`: drag-and-drop upload with live status badges, chat with answers + source chunks, per-document actions. `backoffice.html`: document admin + chunk inspector (per-document chunk text and metadata, download/reprocess/delete). |\n| `simple-rag/` | Earlier single-script version of the same pipeline (ingest + notebook for exploring ChromaDB). Kept for reference/learning. |\n| `docker-compose.yml` | Infrastructure only: MinIO (9000/9001), Redis (6379), Postgres (5432), ChromaDB (8000). Services run locally in the venv. |\n\n## API\n\n### documents-service (:8001)\n\n| Method | Path | Purpose |\n|---|---|---|\n| POST | `/documents` | Create record (`status=uploading`), return presigned PUT URL |\n| GET | `/documents` | List all documents with status |\n| GET | `/documents/{id}` | Single document status |\n| GET | `/documents/{id}/chunks` | Inspect derived chunks (text + metadata) — powers the backoffice |\n| GET | `/documents/{id}/download` | Presigned GET URL for the original file |\n| POST | `/documents/{id}/reprocess` | Re-enqueue chunking/embedding via the Redis queue |\n| DELETE | `/documents/{id}` | Delete everywhere: chunks, MinIO object, DB row |\n\n### chats-service (:8002)\n\n| Method | Path | Purpose |\n|---|---|---|\n| POST | `/query` | RAG query: retrieve chunks, generate answer with Claude, log exchange |\n| GET | `/chats` | Chat history (question, answer, sources) |\n\n## Quickstart\n\n```bash\nmake setup        # create venv + install all service dependencies\necho 'ANTHROPIC_API_KEY=sk-ant-...' \u003e .env   # needed by the chats service\n```\n\nEverything at once:\n\n```bash\nmake run          # infra + all three services + UI in one terminal (Ctrl-C stops all)\n```\n\nOr individually, for isolated logs and per-service restarts:\n\n```bash\nmake infra        # start MinIO, Redis, Postgres, ChromaDB (+ bucket/event init)\n\n# four terminals:\nmake documents    # :8001\nmake chats        # :8002 (reads .env automatically)\nmake worker       # queue consumer\nmake ui           # http://localhost:3000\n```\n\n`make help` lists all targets. MinIO console: http://localhost:9001 (minioadmin/minioadmin).\n\n### Smoke test (no UI)\n\n```bash\ncurl -X POST localhost:8001/documents -H 'Content-Type: application/json' -d '{\"filename\": \"doc.pdf\"}'\ncurl -X PUT --upload-file doc.pdf \"\u003cupload_url from previous response\u003e\"\ncurl localhost:8001/documents/\u003cid\u003e          # poll until status=ready\ncurl -X POST localhost:8002/query -H 'Content-Type: application/json' -d '{\"question\": \"...\"}'\n```\n\n### simple-rag (standalone learning script)\n\n```bash\nmake ingest                  # ingest simple-rag/data/* into its local chroma_db\nmake query Q=\"your question\" # similarity search against it\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonkey-mode%2Frag-data-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonkey-mode%2Frag-data-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonkey-mode%2Frag-data-pipeline/lists"}