{"id":19324306,"url":"https://github.com/su-shubham/lio","last_synced_at":"2025-02-24T06:15:16.904Z","repository":{"id":261907607,"uuid":"885688835","full_name":"su-shubham/Lio","owner":"su-shubham","description":"Streamlined Document Processing \u0026 AI-Powered Customer Engagement with Multi-Format File Support and Retrieval-Augmented Chat","archived":false,"fork":false,"pushed_at":"2024-11-09T06:11:37.000Z","size":182,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-02-15T12:50:07.086Z","etag":null,"topics":["dramtiq","fastapi","qdrant","redis","web-queue-worker"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/su-shubham.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-09T05:57:32.000Z","updated_at":"2024-11-13T12:52:34.000Z","dependencies_parsed_at":"2024-11-09T07:19:37.053Z","dependency_job_id":"5504a02b-2a9b-4c16-9fde-ca55b5080f66","html_url":"https://github.com/su-shubham/Lio","commit_stats":null,"previous_names":["su-shubham/lio"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/su-shubham%2FLio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/su-shubham%2FLio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/su-shubham%2FLio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/su-shubham%2FLio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/su-shubham","download_url":"https://codeload.github.com/su-shubham/Lio/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240427266,"owners_count":19799470,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dramtiq","fastapi","qdrant","redis","web-queue-worker"],"created_at":"2024-11-10T02:04:34.654Z","updated_at":"2025-02-24T06:15:16.802Z","avatar_url":"https://github.com/su-shubham.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🚀 Lio\n\nLio is a comprehensive service designed for efficient document processing and interactive chat capabilities powered by retrieval-augmented generation. Leverage the power of advanced retrieval techniques to integrate document context seamlessly into chat interactions for more precise and context-aware responses.\n\n🔄 Architecture Overview: Built on a robust web-queue-worker pattern where **FastAPI** handles web requests, **Redis** manages message queues, and **Dramatiq** workers process background tasks asynchronously. Powered by **Qdrant** vector database for efficient document storage and retrieval, leveraging **Cohere/OpenAI** embeddings, and **LangChain** for seamless AI interactions and document processing pipeline.\n\n## 📑 Table of Contents\n- [🎯 Prerequisites](#prerequisites)\n- [⚙️ Setup and Installation](#setup-and-installation)\n- [▶️ Running the Application](#running-the-application)\n- [👷 Running Background Workers](#running-background-workers)\n- [🏗️ Project Structure](#project-structure)\n\n## 🎯 Prerequisites\nEnsure you have the following installed on your system:\n- 🐍 **Python 3.12.x**\n- 📦 **PDM** (Python Dependency Management)\n- 🐳 **Docker** for running Redis \n\n## ⚙️ Setup and Installation\n\n### 📥 Step 1: Install PDM \nPDM is used to manage Python dependencies. Run the following command to install PDM if it's not already installed:\n```bash\npip install pdm\n```\n\n### 📦 Step 2: Install Project Dependencies\nNavigate to your project directory and run:\n```bash\npdm install\n```\n\nThis command will create a virtual environment and install the dependencies listed in `pyproject.toml`.\n\n### 🐳 Step 3: Start Redis Using Docker\nRun the following commands to start Redis and Qdrant services:\n\n**Start Redis:**\n```bash\ndocker run --name redis_container -p 6379:6379 -d redis:latest\n```\n\nThese commands will download the Redis and Qdrant images (if not already available) and start the containers.\n\n### 🔑 Step 4: Create a `.env` File\nCreate a `.env` file in the root directory to set environment variables:\n```env\nQDRANT_HOST=localhost\nQDRANT_PORT=6333\nCOHERE_API_KEY=your_cohere_api_key\nOPENAI_API_KEY=your_openai_api_key\n```\n\n## ▶️ Running the Application\n\n### 🚀 Run the FastAPI Main Application\nUse PDM to start the FastAPI server:\n```bash\npdm run uvicorn app.main:app --host 0.0.0.0 --port 8000\n```\n\nThis command runs your FastAPI application, which can be accessed at [http://localhost:8000](http://localhost:8000).\n\n## 👷 Running Background Workers\n\n### ⚡ Run Dramatiq Workers\nDramatiq is used for background tasks. Start the workers using:\n```bash\npdm run dramatiq app.workers\n```\n\nThis command starts the Dramatiq worker processes for handling background tasks.\n\n---\n\n## 📝 End Notes\n\n### 🔍 Quick Troubleshooting Guide\n- ❌ If PDM install fails:\n  ```bash\n  pdm install --no-lock\n  ```\n- 🔄 If Redis container fails:\n  ```bash\n  docker restart redis_container\n  ```\n- 💡 Check logs with:\n  ```bash\n  docker logs redis_container\n  ```\n\n## 📷 Screenshots:\n![alt text](image.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsu-shubham%2Flio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsu-shubham%2Flio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsu-shubham%2Flio/lists"}