{"id":28324176,"url":"https://github.com/md-emon-hasan/autodocthinker","last_synced_at":"2026-06-29T05:31:43.469Z","repository":{"id":316403577,"uuid":"977570619","full_name":"Md-Emon-Hasan/AutoDocThinker","owner":"Md-Emon-Hasan","description":"Agentic AI system that allows users to upload documents (PDFs, DOCX, etc.) and natural language questions. It uses LLM-based RAG to extract relevant information. The architecture includes multi-agent components such as document retrievers, summarizers, web searchers, and tool routers — enabling dynamic reasoning and accurate responses.","archived":false,"fork":false,"pushed_at":"2026-05-07T12:38:57.000Z","size":34485,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-05-07T14:40:00.567Z","etag":null,"topics":["agentic-ai","ai-agents","ai-assistant","ai-document-search","auto-document-analysis","conversation-memory","document-intelligence","document-qa","document-retrieval","duckduckgo-tool","langgraph","llm-apps","llm-reasoning","planner-executor-agent","qna-system","rag","semantic-search","smart-document-search","tool-usage-llm","vector-search"],"latest_commit_sha":null,"homepage":"https://autodocthinker.onrender.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Md-Emon-Hasan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-04T14:09:01.000Z","updated_at":"2026-05-07T12:39:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"f657ce99-e5fd-4ae2-a8aa-ab5fd13e3c53","html_url":"https://github.com/Md-Emon-Hasan/AutoDocThinker","commit_stats":null,"previous_names":["md-emon-hasan/autodocthinker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Md-Emon-Hasan/AutoDocThinker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Md-Emon-Hasan%2FAutoDocThinker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Md-Emon-Hasan%2FAutoDocThinker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Md-Emon-Hasan%2FAutoDocThinker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Md-Emon-Hasan%2FAutoDocThinker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Md-Emon-Hasan","download_url":"https://codeload.github.com/Md-Emon-Hasan/AutoDocThinker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Md-Emon-Hasan%2FAutoDocThinker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34915001,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-29T02:00:05.398Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic-ai","ai-agents","ai-assistant","ai-document-search","auto-document-analysis","conversation-memory","document-intelligence","document-qa","document-retrieval","duckduckgo-tool","langgraph","llm-apps","llm-reasoning","planner-executor-agent","qna-system","rag","semantic-search","smart-document-search","tool-usage-llm","vector-search"],"created_at":"2025-05-25T17:10:32.330Z","updated_at":"2026-06-29T05:31:43.459Z","avatar_url":"https://github.com/Md-Emon-Hasan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AutoDocThinker: Agentic RAG System with Intelligent Search Engine\n\n[![Python](https://img.shields.io/badge/python-3.11-blue?style=for-the-badge\u0026logo=python\u0026logoColor=white)](https://python.org) [![FastAPI](https://img.shields.io/badge/FastAPI-009688?style=for-the-badge\u0026logo=fastapi\u0026logoColor=white)](https://fastapi.tiangolo.com/) [![LangChain](https://img.shields.io/badge/🦜_LangChain-000000?style=for-the-badge)](https://python.langchain.com/) [![LangGraph](https://img.shields.io/badge/LangGraph-1C1C1C?style=for-the-badge\u0026logo=langchain\u0026logoColor=white)](https://langchain-ai.github.io/langgraph/) [![PyTorch](https://img.shields.io/badge/PyTorch-EE4C2C?style=for-the-badge\u0026logo=pytorch\u0026logoColor=white)](https://pytorch.org/) [![Hugging Face](https://img.shields.io/badge/Hugging%20Face-FFD21E?style=for-the-badge\u0026logo=huggingface\u0026logoColor=black)](https://huggingface.co/) [![ChromaDB](https://img.shields.io/badge/ChromaDB-000000?style=for-the-badge\u0026logo=chromadb\u0026logoColor=white)](https://www.trychroma.com/) [![Docker](https://img.shields.io/badge/Docker-2496ED?style=for-the-badge\u0026logo=docker\u0026logoColor=white)](https://www.docker.com/) [![React](https://img.shields.io/badge/React-20232A?style=for-the-badge\u0026logo=react\u0026logoColor=61DAFB)](https://reactjs.org/) [![Tailwind CSS](https://img.shields.io/badge/Tailwind_CSS-38B2AC?style=for-the-badge\u0026logo=tailwind-css\u0026logoColor=white)](https://tailwindcss.com/) [![Groq](https://img.shields.io/badge/Groq-000000?style=for-the-badge\u0026logo=groq\u0026logoColor=white)](https://groq.com/) [![Version](https://img.shields.io/badge/version-3.0.0-brightgreen?style=for-the-badge)](https://github.com/Md-Emon-Hasan/AutoDocThinker)\n\n**AutoDocThinker** (v3.0) is an advanced **Agentic RAG (Retrieval-Augmented Generation)** system designed to bridge the gap between static documents and dynamic intelligence, solving the critical problem of information overload in data-rich environments. Built on a **Modular Monolithic Architecture** with **FastAPI, LangGraph, and ChromaDB**, the system transforms unstructured data (PDFs, Word docs, Web URLs, plain text) into an interactive knowledge base, enabling users to query complex information using natural language. Unlike traditional keyword search that fails to understand context, AutoDocThinker employs a **four-mode RAG workflow engine** — **Naive, Advanced, CRAG (Corrective RAG), and Self-RAG** — to adaptively route, retrieve, evaluate, and regenerate answers. The **Hybrid Search engine** fuses **dense vector retrieval (ChromaDB)** with **sparse BM25 indexing** via **Reciprocal Rank Fusion (RRF)**, followed by **CrossEncoder reranking**, to deliver precision-first results. Seven **domain-specific presets** (Medical, Legal, Finance, Technical, Education, Customer Support, General) tune prompts and retrieval behavior per use case, while a full **chat session system** maintains multi-turn conversation history. This end-to-end solution not only automates research and Level-1 support tasks but also delivers **10x productivity gains** by synthesizing accurate, citation-backed answers in seconds — effectively turning a repository of \"dead\" files into an active, decision-driving organizational brain.\n\n\u003c!-- 🎥 Project Demo Video --\u003e\n[![Project demo video](https://github.com/user-attachments/assets/a18dc570-35fc-4c42-8bad-fd6be37b6c0a)](https://github.com/user-attachments/assets/a18dc570-35fc-4c42-8bad-fd6be37b6c0a)\n\n\u003c!-- 📸 Project Screenshots --\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/user-attachments/assets/b806fce0-58ed-4d95-a737-9b65c79006ff\" \n       alt=\"Project Screenshot 1\" \n       width=\"100%\" /\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/user-attachments/assets/2c051f0a-3b23-41be-9dab-e4082467a349\" \n       alt=\"Project Screenshot 2\" \n       width=\"100%\" /\u003e\n\u003c/p\u003e\n\n---\n\n## **Live Demo**\n\n**Try it now**: [AutoDocThinker: Agentic RAG System with Intelligent Search Engine](https://autodocthinker.onrender.com/)\n\n---\n\n## **Features \u0026 Functionalities**\n\n| #  | Module                   | Technology Stack                        | Implementation Details                                       |\n|----|--------------------------|-----------------------------------------|--------------------------------------------------------------|\n| 1  | **Backend Framework**    | FastAPI + Uvicorn                       | Async support, auto OpenAPI docs, lifecycle hooks            |\n| 2  | **LLM Processing**       | Groq + LLaMA-3-70B                      | Configurable temperature, output parsing, retry logic        |\n| 3  | **Document Parsing**     | PyMuPDF + python-docx + BeautifulSoup   | PDF, DOCX, TXT, URL, raw text with metadata preservation     |\n| 4  | **Text Chunking**        | RecursiveCharacterTextSplitter          | Adaptive chunk optimizer with configurable size and overlap  |\n| 5  | **Vector Embeddings**    | all-MiniLM-L6-v2 (HuggingFace)         | Efficient 384-dimensional dense embeddings                   |\n| 6  | **Vector Database**      | ChromaDB                                | Persistent storage, similarity search, source-level deletion |\n| 7  | **Sparse Index**         | BM25 (rank-bm25)                        | Keyword-based sparse retrieval with custom tokenizer         |\n| 8  | **Hybrid Search**        | Dense + Sparse fusion via RRF           | Reciprocal Rank Fusion merges both retrieval signals         |\n| 9  | **Reranking**            | CrossEncoder (sentence-transformers)    | Re-scores top-K candidates for precision-first results       |\n| 10 | **Compression**          | LLM-based context compression           | Reduces retrieved chunks to only query-relevant sentences    |\n| 11 | **RAG Workflows**        | LangGraph (4 modes)                     | Naive, Advanced, CRAG, Self-RAG with conditional edges       |\n| 12 | **Domain Presets**       | 7 domain profiles                       | General, Medical, Legal, Finance, Education, Technical, CS   |\n| 13 | **Prompt Engineering**   | Domain-aware prompt templates           | Separate system prompts per domain and per RAG workflow      |\n| 14 | **Chat System**          | Session-based multi-turn chat           | Session management, history store, auto title generation     |\n| 15 | **Web Fallback**         | Wikipedia API + LangChain               | Auto-triggered on low-confidence or empty index              |\n| 16 | **CLI Interface**        | Interactive terminal CLI                | Commands for ingestion, querying, and session management     |\n| 17 | **Source Management**    | Per-source ingestion tracking           | Deduplication, source registry, per-source deletion          |\n| 18 | **Index Management**     | Full index lifecycle control            | Status, per-source removal, full clear                       |\n| 19 | **User Interface**       | React 18 + Vite + Tailwind CSS          | SPA with chat, ingestion, domains, index, and admin pages    |\n| 20 | **Containerization**     | Docker + Docker Compose                 | Production-ready multi-service deployment                    |\n\n---\n\n## **Project Structure**\n\n```\nAutoDocThinker/\n│\n├── .github/\n│   └── workflows/\n│       ├── ci-cd.yml                         # Full CI/CD pipeline (lint → test → build → deploy)\n│       └── docker.yml                        # Docker build \u0026 push to GHCR on release\n│\n├── backend/                                  # FastAPI backend application\n│   ├── .dockerignore\n│   ├── .env.example                          # Environment variables template\n│   ├── .flake8                               # Flake8 linting configuration\n│   ├── Dockerfile                            # Backend Docker image\n│   ├── pyproject.toml                        # Project metadata and tool config\n│   ├── requirements.txt                      # Python dependencies\n│   ├── run.py                                # Backend entry point (Uvicorn launcher)\n│   ├── split.py                              # Dev utility for splitting test output\n│   │\n│   ├── app/                                  # Main application package\n│   │   ├── __init__.py\n│   │   ├── application.py                    # FastAPI app factory\n│   │   ├── dependencies.py                   # DI container (IoC box)\n│   │   ├── exceptions.py                     # Global exception handlers\n│   │   ├── lifecycle.py                      # Startup / shutdown hooks\n│   │   ├── logging_config.py                 # Structured logging setup\n│   │   ├── main.py                           # ASGI entry point\n│   │   │\n│   │   ├── api/                              # HTTP route handlers\n│   │   │   ├── __init__.py\n│   │   │   ├── admin_routes.py               # GET /admin/summary\n│   │   │   ├── chat_routes.py                # Chat session CRUD \u0026 query\n│   │   │   ├── domain_routes.py              # Domain preset listing\n│   │   │   ├── health_routes.py              # GET /health\n│   │   │   ├── index_routes.py               # Index status, clear, per-source delete\n│   │   │   ├── ingestion_routes.py           # File upload, URL, raw text ingestion\n│   │   │   ├── rag_routes.py                 # RAG query, mode listing, profiles\n│   │   │   └── router.py                     # Central router aggregator\n│   │   │\n│   │   ├── chat/                             # Chat session management\n│   │   │   ├── __init__.py\n│   │   │   ├── history_store.py              # In-memory chat history store\n│   │   │   ├── memory.py                     # LangChain memory adapter\n│   │   │   ├── message.py                    # Message dataclass\n│   │   │   ├── service.py                    # Chat service (create/get/query session)\n│   │   │   ├── session.py                    # Session model\n│   │   │   └── title_generator.py            # Auto-generate session titles via LLM\n│   │   │\n│   │   ├── cli/                              # Interactive command-line interface\n│   │   │   ├── __init__.py\n│   │   │   ├── commands.py                   # CLI command definitions\n│   │   │   ├── interactive.py                # REPL loop\n│   │   │   └── printing.py                   # Rich terminal output helpers\n│   │   │\n│   │   ├── core/                             # Core config \u0026 constants\n│   │   │   ├── __init__.py\n│   │   │   ├── config.py                     # RAGConfig frozen dataclass (v3.0.0)\n│   │   │   ├── constants.py                  # App-wide constant values\n│   │   │   ├── environment.py                # Env var loader\n│   │   │   ├── errors.py                     # Base custom exception classes\n│   │   │   └── paths.py                      # Path resolution helpers\n│   │   │\n│   │   ├── domain/                           # Domain preset system\n│   │   │   ├── __init__.py\n│   │   │   ├── defaults.py                   # Default domain selection logic\n│   │   │   ├── models.py                     # Domain Pydantic models\n│   │   │   ├── registry.py                   # Domain registry (name → preset)\n│   │   │   ├── selector.py                   # Domain auto-selector\n│   │   │   ├── validator.py                  # Domain input validator\n│   │   │   └── presets/                      # Per-domain configuration\n│   │   │       ├── __init__.py\n│   │   │       ├── customer_support.py\n│   │   │       ├── education.py\n│   │   │       ├── finance.py\n│   │   │       ├── general.py\n│   │   │       ├── legal.py\n│   │   │       ├── medical.py\n│   │   │       └── technical.py\n│   │   │\n│   │   ├── indexing/                         # Hybrid index (vector + BM25)\n│   │   │   ├── __init__.py\n│   │   │   ├── bm25_index.py                 # BM25 sparse index implementation\n│   │   │   ├── chroma_store.py               # ChromaDB collection wrapper\n│   │   │   ├── deduplication.py              # Chunk deduplication logic\n│   │   │   ├── hybrid_index.py               # Unified hybrid index interface\n│   │   │   ├── locking.py                    # Thread-safe write locking\n│   │   │   ├── persistence.py                # Index persistence helpers\n│   │   │   ├── source_registry.py            # Per-source tracking registry\n│   │   │   ├── stats.py                      # Index statistics\n│   │   │   ├── tokenizer.py                  # Custom BM25 tokenizer\n│   │   │   └── vector_index.py               # Vector index operations\n│   │   │\n│   │   ├── ingestion/                        # Document ingestion pipeline\n│   │   │   ├── __init__.py\n│   │   │   ├── chunk_optimizer.py            # Adaptive chunking strategy\n│   │   │   ├── document.py                   # Document dataclass\n│   │   │   ├── document_processor.py         # Load → clean → metadata injection\n│   │   │   ├── file_validation.py            # File type and size validation\n│   │   │   ├── metadata.py                   # Metadata extraction helpers\n│   │   │   ├── service.py                    # Ingestion orchestrator\n│   │   │   ├── source_id.py                  # Deterministic source ID generation\n│   │   │   ├── supported_types.py            # Allowed file type registry\n│   │   │   └── loaders/                      # Format-specific document loaders\n│   │   │       ├── __init__.py\n│   │   │       ├── base.py                   # Abstract loader interface\n│   │   │       ├── docx_loader.py            # DOCX (python-docx) loader\n│   │   │       ├── factory.py                # Routes file_type → loader instance\n│   │   │       ├── pdf_loader.py             # PDF (PyMuPDF) loader\n│   │   │       ├── text_loader.py            # Raw pasted-text loader\n│   │   │       ├── txt_loader.py             # Plain .txt file loader\n│   │   │       └── url_loader.py             # Web URL scraper (BeautifulSoup)\n│   │   │\n│   │   ├── llm/                              # LLM \u0026 embedding clients\n│   │   │   ├── __init__.py\n│   │   │   ├── chain_factory.py              # LangChain chain builder\n│   │   │   ├── embedding_client.py           # HuggingFace embedding wrapper\n│   │   │   ├── fallback.py                   # LLM fallback / error recovery\n│   │   │   ├── groq_client.py                # Groq API client (LLaMA-3)\n│   │   │   ├── output_parser.py              # Structured LLM output parser\n│   │   │   └── wikipedia_client.py           # Wikipedia API client\n│   │   │\n│   │   ├── prompts/                          # Prompt templates\n│   │   │   ├── __init__.py\n│   │   │   ├── answer.py                     # Final answer generation prompt\n│   │   │   ├── base.py                       # Base prompt template\n│   │   │   ├── compression.py                # Context compression prompt\n│   │   │   ├── crag.py                       # CRAG-specific prompts\n│   │   │   ├── evaluation.py                 # Relevance evaluation prompt\n│   │   │   ├── query_rewrite.py              # Query rewriting prompt\n│   │   │   ├── self_rag.py                   # Self-RAG reflection prompts\n│   │   │   └── domain/                       # Domain-specific system prompts\n│   │   │       ├── __init__.py\n│   │   │       ├── customer_support.py\n│   │   │       ├── education.py\n│   │   │       ├── finance.py\n│   │   │       ├── general.py\n│   │   │       ├── legal.py\n│   │   │       ├── medical.py\n│   │   │       └── technical.py\n│   │   │\n│   │   ├── rag/                              # RAG orchestration layer\n│   │   │   ├── __init__.py\n│   │   │   ├── citations.py                  # Citation extraction and formatting\n│   │   │   ├── formatting.py                 # Response formatter\n│   │   │   ├── history.py                    # Conversation history helpers\n│   │   │   ├── modes.py                      # RAG mode enum (naive/advanced/crag/self_rag)\n│   │   │   ├── service.py                    # RAG service (query entry point)\n│   │   │   └── state.py                      # LangGraph shared state schema\n│   │   │\n│   │   ├── retrieval/                        # Retrieval \u0026 ranking pipeline\n│   │   │   ├── __init__.py\n│   │   │   ├── bm25_search.py                # BM25 sparse search\n│   │   │   ├── compressor.py                 # LLM-based chunk compressor\n│   │   │   ├── filters.py                    # Metadata pre-filters\n│   │   │   ├── fusion.py                     # Reciprocal Rank Fusion (RRF)\n│   │   │   ├── hybrid_search.py              # Combined dense + sparse search\n│   │   │   ├── ranking.py                    # Score normalization \u0026 ranking\n│   │   │   ├── reranker.py                   # CrossEncoder reranker\n│   │   │   ├── scoring.py                    # Relevance scoring utilities\n│   │   │   ├── service.py                    # Retrieval service (main interface)\n│   │   │   └── vector_search.py              # ChromaDB vector search\n│   │   │\n│   │   ├── schemas/                          # Pydantic request/response schemas\n│   │   │   ├── __init__.py\n│   │   │   ├── chat.py                       # Chat session schemas\n│   │   │   ├── common.py                     # Shared base schemas\n│   │   │   ├── domain.py                     # Domain schemas\n│   │   │   ├── error.py                      # Error response schema\n│   │   │   ├── health.py                     # Health check schema\n│   │   │   ├── history.py                    # History schemas\n│   │   │   ├── index.py                      # Index schemas\n│   │   │   ├── ingestion.py                  # Ingestion request/response schemas\n│   │   │   ├── rag.py                        # RAG query request/response schemas\n│   │   │   ├── rag_profile.py                # RAG profile schema\n│   │   │   └── source.py                     # Source metadata schema\n│   │   │\n│   │   ├── storage/                          # File and vector storage management\n│   │   │   ├── __init__.py\n│   │   │   ├── cleanup.py                    # Storage cleanup utilities\n│   │   │   ├── file_storage.py               # File system operations\n│   │   │   ├── paths.py                      # Storage path resolution\n│   │   │   ├── upload_storage.py             # Upload directory management\n│   │   │   └── vector_storage.py             # Vector store path management\n│   │   │\n│   │   ├── utils/                            # Shared utility modules\n│   │   │   ├── __init__.py\n│   │   │   ├── hashing.py                    # Content hashing (SHA-256)\n│   │   │   ├── retry.py                      # Exponential backoff retry decorator\n│   │   │   ├── serialization.py              # JSON serialization helpers\n│   │   │   ├── testing.py                    # Test utility helpers\n│   │   │   ├── text.py                       # Text normalization utilities\n│   │   │   ├── time.py                       # Timestamp helpers\n│   │   │   └── validation.py                 # Input validation utilities\n│   │   │\n│   │   └── workflows/                        # LangGraph workflow definitions\n│   │       ├── __init__.py\n│   │       ├── finalize.py                   # Shared finalization node\n│   │       ├── advanced/                     # Advanced RAG workflow\n│   │       │   ├── __init__.py\n│   │       │   ├── compat.py                 # Backward-compat adapter\n│   │       │   ├── edges.py                  # Conditional edge logic\n│   │       │   ├── graph.py                  # LangGraph graph definition\n│   │       │   └── nodes.py                  # Workflow node functions\n│   │       ├── crag/                         # Corrective RAG workflow\n│   │       │   ├── __init__.py\n│   │       │   ├── compat.py\n│   │       │   ├── edges.py\n│   │       │   ├── graph.py\n│   │       │   └── nodes.py\n│   │       ├── naive/                        # Naive RAG workflow\n│   │       │   ├── __init__.py\n│   │       │   ├── compat.py\n│   │       │   ├── edges.py\n│   │       │   ├── graph.py\n│   │       │   └── nodes.py\n│   │       └── self_rag/                     # Self-RAG workflow\n│   │           ├── __init__.py\n│   │           ├── compat.py\n│   │           ├── edges.py\n│   │           ├── graph.py\n│   │           └── nodes.py\n│   │\n│   ├── data/\n│   │   └── vector_store/                     # ChromaDB persistent storage\n│   │\n│   ├── notebooks/\n│   │   ├── experiment.ipynb                  # Exploratory experiments\n│   │   └── fix-final.ipynb                   # Debug notebook\n│   │\n│   ├── uploads/                              # User-uploaded documents (runtime)\n│   │\n│   └── tests/                               # Full test suite\n│       ├── conftest.py                       # Shared fixtures and DI overrides\n│       ├── api/\n│       │   ├── test_admin_routes.py\n│       │   ├── test_chat_routes.py\n│       │   ├── test_domain_routes.py\n│       │   ├── test_health_route.py\n│       │   ├── test_index_routes.py\n│       │   ├── test_ingest_text_routes.py\n│       │   ├── test_ingestion_routes.py\n│       │   ├── test_rag_routes.py\n│       │   └── test_upload_routes.py\n│       ├── chat/\n│       │   ├── test_chat_service.py\n│       │   ├── test_chat_session.py\n│       │   ├── test_history_store.py\n│       │   ├── test_make_message.py\n│       │   ├── test_memory.py\n│       │   └── test_title_generator.py\n│       ├── core/\n│       │   ├── test_application.py\n│       │   ├── test_c_l_i.py\n│       │   ├── test_config.py\n│       │   ├── test_constants.py\n│       │   ├── test_environment.py\n│       │   ├── test_errors.py\n│       │   ├── test_lifecycle_and_exceptions.py\n│       │   ├── test_logging.py\n│       │   └── test_paths.py\n│       ├── domain/\n│       │   ├── test_defaults.py\n│       │   ├── test_domain_profile.py\n│       │   ├── test_domain_prompt_constants.py\n│       │   ├── test_registry.py\n│       │   ├── test_selector.py\n│       │   └── test_validator.py\n│       ├── indexing/\n│       │   ├── test_b_m25_index.py\n│       │   ├── test_b_m25_search.py\n│       │   ├── test_chroma_store.py\n│       │   ├── test_compressor.py\n│       │   ├── test_deduplication.py\n│       │   ├── test_filters.py\n│       │   ├── test_fusion.py\n│       │   ├── test_hybrid_index.py\n│       │   ├── test_hybrid_search.py\n│       │   ├── test_locking.py\n│       │   ├── test_persistence.py\n│       │   ├── test_ranking.py\n│       │   ├── test_reranker.py\n│       │   ├── test_retrieval_service.py\n│       │   ├── test_scoring.py\n│       │   ├── test_source_registry.py\n│       │   ├── test_stats.py\n│       │   ├── test_tokenizer.py\n│       │   ├── test_vector_index.py\n│       │   └── test_vector_search.py\n│       ├── ingestion/\n│       │   ├── test_base_loader.py\n│       │   ├── test_chunk_optimizer.py\n│       │   ├── test_document.py\n│       │   ├── test_document_processor.py\n│       │   ├── test_docx_loader.py\n│       │   ├── test_file_validation.py\n│       │   ├── test_ingestion_service.py\n│       │   ├── test_loader_factory.py\n│       │   ├── test_metadata.py\n│       │   ├── test_pdf_loader.py\n│       │   ├── test_source_id.py\n│       │   ├── test_standalone_functions.py\n│       │   ├── test_supported_types.py\n│       │   ├── test_text_loader.py\n│       │   ├── test_txt_loader.py\n│       │   └── test_url_loader.py\n│       ├── llm/\n│       │   ├── test_chain_factory.py\n│       │   ├── test_embedding_client.py\n│       │   ├── test_fallback.py\n│       │   ├── test_groq_client.py\n│       │   ├── test_output_parser.py\n│       │   ├── test_prompts.py\n│       │   └── test_wikipedia.py\n│       ├── rag/\n│       │   ├── test_advanced_workflow.py\n│       │   ├── test_c_r_a_g_workflow.py\n│       │   ├── test_citations.py\n│       │   ├── test_finalize.py\n│       │   ├── test_history.py\n│       │   ├── test_modes.py\n│       │   ├── test_naive_workflow.py\n│       │   ├── test_process_query.py\n│       │   ├── test_r_a_g_service.py\n│       │   ├── test_self_r_a_g_workflow.py\n│       │   └── test_state.py\n│       ├── schemas/\n│       │   └── test_schemas.py\n│       ├── storage/\n│       │   └── test_storage.py\n│       └── utils/\n│           └── test_utils.py\n│\n├── frontend/                                 # React frontend application\n│   ├── .dockerignore\n│   ├── .gitignore\n│   ├── Dockerfile                            # Frontend Docker image (Nginx)\n│   ├── index.html                            # HTML entry point\n│   ├── package.json                          # Node.js dependencies\n│   ├── package-lock.json\n│   ├── README.md\n│   ├── vite.config.js                        # Vite bundler config\n│   ├── public/\n│   │   └── favicon.svg\n│   └── src/\n│       ├── api.js                            # Centralized API client (fetch wrappers)\n│       ├── App.jsx                           # Root component with React Router\n│       ├── index.css                         # Global Tailwind CSS styles\n│       ├── main.jsx                          # React entry point\n│       └── components/\n│           ├── AdminPage.jsx                 # System summary dashboard\n│           ├── ChatPage.jsx                  # Multi-turn AI chat interface\n│           ├── DomainsPage.jsx               # Domain preset browser\n│           ├── IndexPage.jsx                 # Index status and management\n│           ├── IngestPage.jsx                # Document upload / URL / text ingestion\n│           └── Sidebar.jsx                   # Navigation sidebar\n│\n├── .gitignore\n├── demo.mp4                                  # Project demo video\n├── demo.png                                  # Project screenshot\n├── docker-compose.yml                        # Multi-service orchestration\n├── Dockerfile                                # Root multi-stage Docker image\n├── LICENSE\n├── README.md\n├── render.yml                                # Render.com deployment config\n└── run.py                                    # Root entry point (starts backend)\n```\n\n---\n\n## **Architecture Pattern: Modular Monolithic Architecture**\n\nThis project follows a **Modular Monolithic Architecture** with the following design patterns:\n\n| Pattern | Where Used | Purpose |\n|---------|------------|---------|\n| **App Factory** | `app/application.py` | Configurable FastAPI app creation |\n| **IoC Container** | `app/dependencies.py` | Dependency injection box wires all services |\n| **Frozen Config** | `app/core/config.py` | Immutable `RAGConfig` dataclass for all settings |\n| **Strategy** | `app/workflows/*/` | Four interchangeable RAG workflow strategies |\n| **State Machine** | `app/workflows/*/graph.py` | LangGraph conditional state transitions |\n| **Template Method** | `app/ingestion/loaders/base.py` | Common loader interface per file type |\n| **Repository** | `app/indexing/hybrid_index.py` | Unified data access over vector + BM25 stores |\n| **Registry** | `app/domain/registry.py` | Name-keyed domain preset lookup |\n| **Singleton** | `app/dependencies.py` | Single shared instances of index, LLM, embedder |\n\n---\n\n## **System Architecture**\n\n```mermaid\ngraph TD\n    UI[React Frontend]:::ui --\u003e|HTTP REST| API[FastAPI Server]:::server\n\n    API --\u003e IGR[Ingestion Routes]:::route\n    API --\u003e RAGR[RAG Routes]:::route\n    API --\u003e CHR[Chat Routes]:::route\n    API --\u003e IDR[Index Routes]:::route\n    API --\u003e DMR[Domain \u0026 Admin Routes]:::route\n\n    IGR --\u003e IS[Ingestion Service]:::processor\n    IS --\u003e DP[Document Processor + Chunk Optimizer]:::splitter\n    DP --\u003e HI[Hybrid Index]:::database\n    HI --\u003e VI[Vector Index / ChromaDB]:::database\n    HI --\u003e BI[BM25 Sparse Index]:::database\n\n    RAGR --\u003e RS[RAG Service]:::rag\n    RS --\u003e WS{Workflow Selector}:::router\n    WS --\u003e|naive| NW[Naive RAG]:::workflow\n    WS --\u003e|advanced| AW[Advanced RAG]:::workflow\n    WS --\u003e|crag| CW[CRAG Workflow]:::workflow\n    WS --\u003e|self_rag| SW[Self-RAG Workflow]:::workflow\n\n    NW \u0026 AW \u0026 CW \u0026 SW --\u003e RET[Retrieval Service]:::retriever\n    RET --\u003e HS[Hybrid Search Dense + Sparse]:::retriever\n    HS --\u003e VI\n    HS --\u003e BI\n    HS --\u003e RRF[RRF Fusion + CrossEncoder Reranker]:::retriever\n    RRF --\u003e LLM[Groq LLM / LLaMA-3-70B]:::llm\n\n    CW \u0026 SW --\u003e|low confidence| WK[Wikipedia Fallback]:::fallback\n    WK --\u003e LLM\n\n    LLM --\u003e FR[Formatted Response + Citations]:::executor\n    FR --\u003e API\n\n    CHR --\u003e CS[Chat Session Service]:::chat\n    CS --\u003e HS2[History Store + Title Generator]:::chat\n    DMR --\u003e DR[Domain Registry → 7 Presets]:::domain\n\n    classDef ui fill:#4e79a7,color:white;\n    classDef server fill:#f28e2b,color:white;\n    classDef route fill:#e15759,color:white;\n    classDef processor fill:#76b7b2,color:white;\n    classDef splitter fill:#edc948,color:#333;\n    classDef database fill:#8cd17d,color:#333;\n    classDef rag fill:#499894,color:white;\n    classDef router fill:#b07aa1,color:white;\n    classDef workflow fill:#86bcb6,color:#333;\n    classDef retriever fill:#59a14f,color:white;\n    classDef fallback fill:#f1ce63,color:#333;\n    classDef llm fill:#d37295,color:white;\n    classDef executor fill:#b3b3b3,color:#333;\n    classDef chat fill:#a0d6e5,color:#333;\n    classDef domain fill:#ff9da7,color:#333;\n```\n\n---\n\n## **Installation**\n\n### Prerequisites\n\n- Python 3.11+\n- Node.js 18+ (for frontend)\n- Groq API Key\n\n### Using pip\n\n```bash\n# Clone the repository\ngit clone https://github.com/Md-Emon-Hasan/AutoDocThinker.git\ncd AutoDocThinker\n\n# Create virtual environment\npython -m venv venv\nsource venv/bin/activate  # Windows: venv\\Scripts\\activate\n\n# Install backend dependencies\ncd backend\npip install -r requirements.txt\n\n# Copy and configure environment\ncp .env.example .env\n# Edit .env with your API keys\n\n# Run the backend\npython run.py\n```\n\n### Using Docker\n\nThe project includes a **Root Multi-stage Dockerfile** that builds both the React frontend and the FastAPI backend into a single deployable container.\n\n```bash\n# RECOMMENDED: Build and run with Docker Compose\ndocker-compose up -d --build\n\n# OR: Build the root Docker image manually\ndocker build -t auto-doc-thinker .\n\n# Run the container\ndocker run -p 5000:5000 --env-file backend/.env auto-doc-thinker\n```\n\n\u003e [!NOTE]\n\u003e The container serves the **Frontend UI at the same port as the Backend (5000)** when built via the root Dockerfile.\n\n---\n\n## **Configuration**\n\nKey environment variables in `backend/.env`:\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `GROQ_API_KEY` | Groq API key for LLaMA-3 | required |\n| `HUGGINGFACEHUB_API_TOKEN` | HuggingFace token for embeddings | required |\n| `GOOGLE_API_KEY` | Google API key (optional integrations) | optional |\n| `TAVILY_API_KEY` | Tavily search API key | optional |\n| `SERPER_API_KEY` | Serper web search API key | optional |\n| `FLASK_ENV` | Environment mode | development |\n| `SECRET_KEY` | Application secret key | change in production |\n\nCore RAG parameters are set via the frozen `RAGConfig` dataclass in `backend/app/core/config.py`:\n\n| Parameter | Description | Default |\n|-----------|-------------|---------|\n| `default_domain` | Domain preset used when none specified | `general` |\n| `default_mode` | RAG workflow used when none specified | `advanced` |\n| `initial_k` | Candidates retrieved before reranking | `20` |\n| `rerank_top_k` | Final chunks passed to LLM after reranking | `5` |\n| `crag_high_confidence` | CRAG score threshold for direct answer | `0.6` |\n| `crag_low_confidence` | CRAG score threshold for Wikipedia fallback | `0.3` |\n| `supported_extensions` | Accepted file types | `.pdf`, `.docx`, `.txt` |\n\n---\n\n## **API Endpoints**\n\n| Endpoint | Method | Description |\n|----------|--------|-------------|\n| `/health` | GET | Health check |\n| `/docs` | GET | Swagger interactive API documentation |\n| `/redoc` | GET | ReDoc API documentation |\n| `/rag-modes` | GET | List available RAG modes |\n| `/rag-profiles` | GET | List RAG profiles per domain |\n| `/rag/query` | POST | Run a RAG query (domain + mode + history) |\n| `/ingest/source` | POST | Ingest document from file path or URL source |\n| `/ingest/upload` | POST | Upload a file (PDF / DOCX / TXT) |\n| `/ingest/text` | POST | Ingest raw pasted text |\n| `/index/status` | GET | Get index stats (chunk count, sources) |\n| `/index/source/{source_id}` | DELETE | Remove a specific ingested source |\n| `/index` | DELETE | Clear the entire index |\n| `/chat/sessions` | POST | Create a new chat session |\n| `/chat/sessions/{id}` | GET | Retrieve an existing session |\n| `/chat/sessions/{id}/select-profile` | POST | Set domain and RAG mode for a session |\n| `/chat/sessions/{id}/query` | POST | Send a message in a session |\n| `/domains` | GET | List all available domain presets |\n| `/admin/summary` | GET | System summary (domains, chunk count) |\n\n---\n\n## **Usage**\n\n1. **Select a Domain**: Choose the domain that best matches your documents (e.g., Medical, Legal, Finance)\n2. **Select a RAG Mode**: Pick `naive` for speed, `advanced` for quality, `crag` or `self_rag` for highest accuracy\n3. **Upload a Document**: Choose PDF, DOCX, TXT, paste a URL, or type raw text directly\n4. **Click \"Ingest\"**: System loads, chunks, embeds, and indexes into the Hybrid Index (Vector + BM25)\n5. **Ask Questions**: Chat with your documents using natural language in the Chat page\n6. **Get AI Answers**: Responses include source citations; if no relevant documents exist, Wikipedia fallback activates automatically\n7. **Manage Index**: Use the Index page to view ingested sources or remove specific documents\n\n---\n\n## **Running Tests**\n\n### Backend Tests\nNavigate to the `backend` directory first:\n\n```bash\ncd backend\n```\n\nThen run the tests:\n\n```bash\n# Run all tests\npytest tests/ -v\n\n# Run with coverage\npytest tests/ -v --cov=app --cov-report=html\n\n# Run async tests\npytest tests/ -v --asyncio-mode=auto\n\n# Run a specific test module\npytest tests/rag/ -v\npytest tests/indexing/ -v\n```\n\n---\n\n## **Testing Strategy \u0026 Quality Assurance**\n\nWe employ a comprehensive testing strategy using **Pytest** and **unittest.mock** to ensure reliability and maintainability across all modules.\n\n### **1. Unit Testing (White-Box Testing)**\n- **Isolation**: Each module (Ingestion, Indexing, Retrieval, RAG, Chat, LLM, Schemas, Storage, Utils) is tested in isolation.\n- **Mocking**: External dependencies (Groq API, ChromaDB, Wikipedia, HuggingFace) are mocked to ensure tests are deterministic and do not require network access.\n- **Technique**: `patch` and `MagicMock` simulate external behaviors, error conditions, and edge cases.\n\n### **2. Edge Case \u0026 Error Handling**\n- **Boundary Value Analysis**: Testing empty inputs, invalid file types, oversized payloads, missing sessions, and unknown domains.\n- **Exception Handling**: Verifying the system gracefully handles API rate limits (429), LLM downtime (500), and invalid ingestion requests (400) with correct HTTP status codes.\n\n### **3. Integration Testing (Simulated)**\n- **Workflow Graph**: Each of the four LangGraph workflows (Naive, Advanced, CRAG, Self-RAG) is tested by simulating state transitions through nodes and conditional edges.\n- **API Endpoints**: All FastAPI routes are tested with `TestClient` to verify HTTP status codes, response schemas, and error payloads.\n- **Hybrid Index**: End-to-end ingestion → hybrid search → RRF fusion → reranker pipeline tested with in-memory mocks.\n\n### **4. Module Coverage**\nTests span every backend module: `api`, `chat`, `core`, `domain`, `indexing`, `ingestion`, `llm`, `rag`, `schemas`, `storage`, `utils`, and all four workflow variants.\n\n### **5. Code Quality Metrics**\n- **100% Test Coverage Goal**: Every code path executed during test runs.\n- **Linting**: Strict adherence to **PEP 8** standards enforced via `flake8` (config in `.flake8`), `isort`, and `black`.\n- **Type Safety**: Pydantic v2 models enforce runtime data validation across all API boundaries.\n\n---\n\n## **Log Management**\n\nThe application uses a structured logging system for monitoring and debugging, configured in `app/logging_config.py`.\n\n- **Storage**: Logs are stored in `logs/app.log`.\n- **Rotation**: Automatic log rotation (10MB per file, keeping last 5 backups) prevents disk overflow.\n- **Format**: `YYYY-MM-DD HH:MM:SS - logger_name - LEVEL - [file:line] - message`\n- **Levels**:\n  - `INFO`: General operational events (requests, ingestion, state transitions).\n  - `DEBUG`: Detailed debugging information (only in development).\n  - `ERROR`: Exceptions and critical failures (stack traces included).\n\n---\n\n## **Tech Stack**\n\n| Category | Technologies |\n|----------|--------------|\n| **Backend** | FastAPI, Uvicorn, Python 3.11 |\n| **AI / LLM** | Groq API (LLaMA-3-70B), LangChain, LangGraph |\n| **Embeddings** | HuggingFace `all-MiniLM-L6-v2`, CrossEncoder reranker |\n| **Vector Database** | ChromaDB (persistent dense vector store) |\n| **Sparse Index** | BM25 via `rank-bm25` |\n| **Hybrid Search** | Dense + Sparse fusion with Reciprocal Rank Fusion (RRF) |\n| **Web Fallback** | Wikipedia API via LangChain |\n| **Frontend** | React 18, Vite, Tailwind CSS |\n| **DevOps** | Docker, Docker Compose, GitHub Actions, Render |\n\n---\n\n## **CI/CD Pipeline**\n\nThis project uses **GitHub Actions** for continuous integration and deployment.\n\n### Pipeline Stages\n\n```\n┌─────────┐    ┌─────────┐    ┌──────────┐    ┌─────────┐    ┌──────────┐\n│  Lint   │───▶│  Test   │───▶│ Security │───▶│  Build  │───▶│  Deploy  │\n│ (Black, │    │(pytest) │    │ (Safety, │    │(Docker) │    │ (Render) │\n│ Flake8) │    │         │    │  Bandit) │    │         │    │          │\n└─────────┘    └─────────┘    └──────────┘    └─────────┘    └──────────┘\n```\n\n### Workflow Files\n\n| File | Trigger | Purpose |\n|------|---------|---------|\n| `ci-cd.yml` | Push/PR to main | Full CI/CD pipeline |\n| `docker.yml` | Release published | Build \u0026 push to GHCR |\n\n### Required Secrets\n\n| Secret | Description |\n|--------|-------------|\n| `GROQ_API_KEY` | Groq API key for test runs |\n| `RENDER_DEPLOY_HOOK` | Render deploy webhook URL |\n\n---\n\n## **Author**\n\n**Md Emon Hasan**\n\n- Email: [emon.mlengineer@gmail.com](mailto:emon.mlengineer@gmail.com)\n- LinkedIn: [md-emon-hasan](https://www.linkedin.com/in/md-emon-hasan-695483237/)\n- GitHub: [Md-Emon-Hasan](https://github.com/Md-Emon-Hasan)\n- Facebook: [Md-Emon-Hasan](https://www.facebook.com/mdemon.hasan2001/)\n- WhatsApp: [+8801834363533](https://wa.me/8801834363533)\n\n---\n\n## **License**\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n---\n\n## **Contributing**\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmd-emon-hasan%2Fautodocthinker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmd-emon-hasan%2Fautodocthinker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmd-emon-hasan%2Fautodocthinker/lists"}