{"id":51455662,"url":"https://github.com/mrshibly/rag-assistant-fastapi","last_synced_at":"2026-07-06T00:34:16.739Z","repository":{"id":368828112,"uuid":"1287036802","full_name":"mrshibly/rag-assistant-fastapi","owner":"mrshibly","description":"A  FastAPI AI Assistant supporting document ingestion, RAG chat, context memory, and tool calling. Powered by LLM and local embeddings.","archived":false,"fork":false,"pushed_at":"2026-07-02T12:39:47.000Z","size":348,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-06T00:34:16.178Z","etag":null,"topics":["context-memory","fastapi","llm","rag","tool-calling"],"latest_commit_sha":null,"homepage":"https://rag-assistant-fastapi.onrender.com/docs","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mrshibly.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-02T10:16:06.000Z","updated_at":"2026-07-02T12:39:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mrshibly/rag-assistant-fastapi","commit_stats":null,"previous_names":["mrshibly/rag-assistant-fastapi"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mrshibly/rag-assistant-fastapi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrshibly%2Frag-assistant-fastapi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrshibly%2Frag-assistant-fastapi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrshibly%2Frag-assistant-fastapi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrshibly%2Frag-assistant-fastapi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mrshibly","download_url":"https://codeload.github.com/mrshibly/rag-assistant-fastapi/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrshibly%2Frag-assistant-fastapi/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35174071,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-05T02:00:06.290Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["context-memory","fastapi","llm","rag","tool-calling"],"created_at":"2026-07-06T00:34:16.288Z","updated_at":"2026-07-06T00:34:16.726Z","avatar_url":"https://github.com/mrshibly.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mini AI Assistant\n\n[![Live API Docs](https://img.shields.io/badge/API-Live_Docs-blue?style=for-the-badge\u0026logo=fastapi)](https://rag-assistant-fastapi.onrender.com/docs)\n[![Live Health Check](https://img.shields.io/badge/API-Health_Check-green?style=for-the-badge\u0026logo=statuspage)](https://rag-assistant-fastapi.onrender.com/health)\n\nA mini AI assistant built with FastAPI that supports knowledge ingestion, RAG-based chat, session context memory, and tool calling.\n\n\n## Features\n\n- **Knowledge Ingestion** — Upload PDF, TXT, or Markdown files. Documents are chunked, embedded, and stored in ChromaDB for retrieval.\n- **RAG Chat** — Ask questions about uploaded documents. Uses semantic search to find relevant context and generates answers via LLM.\n- **Context Memory** — Maintains conversation history within a session. The assistant remembers previous messages and resolves references.\n- **Tool Calling** — Automatically calls tools (order status lookup, product search) when the user's query matches a tool's purpose.\n\n## Architecture\n\n```mermaid\nflowchart TD\n    %% Styling Definitions\n    classDef ingest fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af;\n    classDef chat fill:#d1fae5,stroke:#059669,stroke-width:2px,color:#065f46;\n    classDef llm fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e;\n\n    %% 1. Document Ingestion Pipeline\n    subgraph Ingestion [\"1. DOCUMENT INGESTION PIPELINE\"]\n        direction LR\n        I1[\"Upload Document\u003cbr/\u003e(PDF / TXT / MD)\"]:::ingest --\u003e I2[\"Load Document\u003cbr/\u003e(PyPDF2 / text)\"]:::ingest\n        I2 --\u003e I3[\"Split into Chunks\u003cbr/\u003e(RecursiveCharacter)\"]:::ingest\n        I3 --\u003e I4[\"Generate Embeddings\u003cbr/\u003e(fastembed BGE-small)\"]:::ingest\n        I4 --\u003e I5[\"Store in Database\u003cbr/\u003e(ChromaDB Collection)\"]:::ingest\n    end\n\n    %% 2. Chat \u0026 Retrieval Pipeline\n    subgraph Chat [\"2. CHAT \u0026 RETRIEVAL PIPELINE\"]\n        direction TB\n        C1[\"User Message\u003cbr/\u003e(ChatRequest)\"]:::chat --\u003e C2[\"Load Session Memory\u003cbr/\u003e(MemoryService)\"]:::chat\n        C2 --\u003e C3[\"Retrieve Context\u003cbr/\u003e(ChromaDB Query)\"]:::chat\n        C3 --\u003e C4[\"Orchestrate LLM\u003cbr/\u003e(Groq Llama-3.3)\"]:::llm\n        \n        %% Decision Branches\n        C4 --\u003e B1[\"Execute Tool\u003cbr/\u003e(orders.json / products.json)\"]:::chat\n        C4 --\u003e B2[\"RAG Context Answer\u003cbr/\u003e(From retrieved text)\"]:::chat\n        C4 --\u003e B3[\"Direct Response\u003cbr/\u003e(General / memory)\"]:::chat\n        \n        %% Convergence\n        B1 --\u003e M1[\"Update Conversation Memory\u003cbr/\u003e(add user/assistant messages)\"]:::chat\n        B2 --\u003e M1\n        B3 --\u003e M1\n        \n        M1 --\u003e R1[\"Return Response\u003cbr/\u003e(ChatResponse)\"]:::chat\n    end\n```\n\n\u003e See [docs/architecture.md](docs/architecture.md) for the highly detailed step-by-step pipeline flowchart (including subgraphs, validation steps, and decision routing details).\n\n## Tech Stack\n\n| Component | Technology | Rationale |\n|-----------|-----------|-----------|\n| Framework | FastAPI | Required by task; async-first, auto-generated docs |\n| LLM | Groq (llama-3.3-70b-versatile) | Fast inference, free tier, strong tool-calling ability |\n| Embeddings | fastembed (BAAI/bge-small-en-v1.5) | Local execution — ONNX quantized model, extremely fast, no PyTorch dependency |\n| Vector DB | ChromaDB | Zero-config, persistent, pip-installable; simpler than FAISS for this scope |\n| Chunking | langchain-text-splitters | Battle-tested `RecursiveCharacterTextSplitter` |\n\n## Setup\n\n### Option 1: Local\n\n```bash\ngit clone \u003crepo-url\u003e\ncd StudioButterfly-Task\npython -m venv venv\nsource venv/bin/activate  # Windows: venv\\Scripts\\activate\npip install -r requirements.txt\ncp .env.example .env      # Edit .env and set GROQ_API_KEY\nuvicorn app.main:app --reload\n```\n\n### Option 2: Docker\n\n```bash\ncp .env.example .env      # Edit .env and set GROQ_API_KEY\ndocker compose up --build\n```\n\nGet a free Groq API key at [console.groq.com](https://console.groq.com).\n\nThe API is available at `http://localhost:8000`. Interactive docs at `http://localhost:8000/docs`.\n\n## API Endpoints\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `POST` | `/api/v1/ingest` | Upload a document to the knowledge base |\n| `POST` | `/api/v1/chat` | Send a message to the assistant |\n| `DELETE` | `/api/v1/sessions/{session_id}` | Clear session memory |\n| `GET` | `/health` | Health check |\n\n### `POST /api/v1/ingest`\n\n```bash\ncurl -X POST http://localhost:8000/api/v1/ingest -F \"file=@document.pdf\"\n```\n\n```json\n{ \"filename\": \"document.pdf\", \"chunks\": 12 }\n```\n\n### `POST /api/v1/chat`\n\n```bash\ncurl -X POST http://localhost:8000/api/v1/chat \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"session_id\": \"user1\", \"message\": \"What is in the document?\"}'\n```\n\n```json\n{\n  \"response\": \"The document covers...\",\n  \"sources\": [\"document.pdf\"],\n  \"tool_used\": null\n}\n```\n\n## Usage Examples\n\n**Knowledge Q\u0026A:**\n```\nUser: \"What are the main topics in the uploaded file?\"\n→ Retrieves relevant chunks, answers from document content.\n```\n\n**Context Memory:**\n```\nUser: \"My name is John.\"\nUser: \"What's my name?\"\nAssistant: \"Your name is John.\"\n```\n\n**Tool Calling:**\n```\nUser: \"Where is my order ORD001?\"\nAssistant: \"Your order ORD001 has been shipped, estimated delivery on 2026-07-02.\"\n\nUser: \"Do you have a wireless mouse?\"\nAssistant: \"Yes! Wireless Mouse — $25, 12 units in stock.\"\n```\n\n## Running Tests\n\n```bash\npytest tests/ -v\n```\n\n## Design Decisions\n\n### Why Groq over OpenAI/Ollama?\nGroq provides extremely fast inference (sub-second responses) with a generous free tier. Unlike Ollama, it requires no local GPU or model downloads, making the project easy to set up for reviewers.\n\n### Why ChromaDB over FAISS?\nChromaDB is a higher-level abstraction that handles persistence, metadata filtering, and collection management out of the box. FAISS requires manual serialization, ID management, and metadata tracking. For this scope, ChromaDB reduces boilerplate while providing the same cosine similarity search.\n\n### Why local embeddings (fastembed) instead of API-based or PyTorch-based?\nUsing `fastembed` with `BAAI/bge-small-en-v1.5` locally means only ONE API key is needed (Groq). Unlike `sentence-transformers`, `fastembed` runs the model using ONNX Runtime. This completely avoids downloading the heavy PyTorch library (~2.6GB), reducing the Docker image footprint from 3.5GB+ to just ~350MB, while running inference in milliseconds. BGE-small is also highly accurate, ranking high on the MTEB leaderboard.\n\n### Why in-memory session storage?\nFor a take-home assignment, in-memory `dict[session_id → messages]` is the right trade-off between simplicity and functionality. A production system would use Redis or a database, but that adds infrastructure complexity without demonstrating additional AI pipeline knowledge.\n\n### Tool-Calling Strategy\nRather than using a framework-level tool-calling API (which varies by LLM provider), I use a **prompt-driven approach**: the system prompt defines tool schemas as JSON, and the LLM outputs a structured `{\"tool\": \"name\", \"args\": {...}}` object when appropriate. The application parses this, executes the tool, and sends the result back to the LLM for natural-language formatting. This approach is:\n- **Provider-agnostic** — works with any LLM that follows instructions\n- **Transparent** — tool call decisions are visible in the response\n- **Extensible** — adding a new tool requires only a function + a JSON definition\n\n### Prompt Design\nA single system prompt handles all routing with priority rules:\n1. **Tools first** — product/order queries always trigger tools, even if document context is available\n2. **Knowledge retrieval** — non-tool queries use RAG context\n3. **Direct response** — general conversation (greetings, memory recall)\n4. **Fallback** — explicit \"I couldn't find that information\" message\n\nThis priority ordering prevents the LLM from \"hallucinating\" answers from retrieved context when a precise tool lookup is available.\n\n## Pipeline Explanation\n\n### Ingestion Pipeline\n1. User uploads a PDF, TXT, or Markdown file via `POST /api/v1/ingest`.\n2. The document is loaded (PyPDF2 for PDF, plain read for TXT/MD).\n3. Text is split into chunks using `RecursiveCharacterTextSplitter` (500 chars, 50 overlap).\n4. Each chunk is embedded using `BAAI/bge-small-en-v1.5` via `fastembed`.\n5. Embeddings and chunks are stored in ChromaDB with source metadata.\n\n### Retrieval Approach\n- The user's query is embedded using the same model.\n- ChromaDB performs cosine similarity search, returning the top-3 most relevant chunks.\n- Retrieved context is prepended to the user's message before sending to the LLM.\n\n### Memory Implementation\n- In-memory dictionary keyed by `session_id`, storing `[{role, content}]` messages.\n- Auto-trimmed to the last 20 messages to prevent context overflow.\n- Full history is passed to the LLM on each request for continuity.\n- Sessions can be explicitly cleared via `DELETE /api/v1/sessions/{id}`.\n\n## Project Structure\n\n```\napp/\n├── main.py                    # FastAPI app, lifespan, global error handler\n├── config.py                  # Settings (pydantic-settings)\n├── api/v1/\n│   ├── router.py              # V1 router aggregator\n│   └── endpoints/\n│       ├── ingest.py          # Document upload endpoint\n│       └── chat.py            # Chat + session management endpoints\n├── schemas/\n│   └── chat.py                # Pydantic request/response models\n├── services/\n│   ├── ingestion.py           # Load → chunk → embed → store\n│   ├── retrieval.py           # Vector similarity search\n│   ├── llm.py                 # Groq LLM client \u0026 orchestration\n│   ├── memory.py              # Session conversation memory\n│   └── tools.py               # Tool definitions \u0026 execution\n└── data/\n    ├── orders.json            # Sample order data\n    └── products.json          # Sample product data\ntests/\n└── test_services.py           # Unit tests for tools \u0026 memory\n```\n\n## Error Handling\n\n- **Unsupported file types** → 400 with descriptive message\n- **Empty file uploads** → 400 with descriptive message\n- **LLM API failures** → Graceful fallback message instead of 500\n- **Tool formatting failures** → Returns raw tool result as fallback\n- **Order/product not found** → Descriptive \"not found\" messages\n- **Unhandled exceptions** → Global handler returns clean JSON error with logging\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrshibly%2Frag-assistant-fastapi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrshibly%2Frag-assistant-fastapi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrshibly%2Frag-assistant-fastapi/lists"}