{"id":17487492,"url":"https://github.com/tech-c-p/conversai","last_synced_at":"2026-02-12T16:31:07.560Z","repository":{"id":258128573,"uuid":"873422325","full_name":"Tech-C-P/ConversAI","owner":"Tech-C-P","description":"ConversAI is an innovative conversational AI framework designed for intelligent text extraction and querying across various document formats and web content, leveraging advanced natural language processing techniques.","archived":false,"fork":false,"pushed_at":"2024-10-16T07:35:44.000Z","size":1070,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-04-10T16:52:30.718Z","etag":null,"topics":["beautifulsoup","chatbot","genai","gradio","groq","langchain","large-language-models","llama3","mlops","nlp","ocr","pymupdf","python"],"latest_commit_sha":null,"homepage":"https://huggingface.co/spaces/techconsptr/ConversAI, https://convers-ai-test.vercel.app/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Tech-C-P.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-16T06:27:10.000Z","updated_at":"2024-11-22T23:00:14.000Z","dependencies_parsed_at":"2024-10-17T19:40:07.057Z","dependency_job_id":null,"html_url":"https://github.com/Tech-C-P/ConversAI","commit_stats":null,"previous_names":["tech-c-p/conversai"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Tech-C-P/ConversAI","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tech-C-P%2FConversAI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tech-C-P%2FConversAI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tech-C-P%2FConversAI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tech-C-P%2FConversAI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Tech-C-P","download_url":"https://codeload.github.com/Tech-C-P/ConversAI/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tech-C-P%2FConversAI/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273124651,"owners_count":25050093,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautifulsoup","chatbot","genai","gradio","groq","langchain","large-language-models","llama3","mlops","nlp","ocr","pymupdf","python"],"created_at":"2024-10-19T03:04:46.067Z","updated_at":"2026-02-12T16:31:07.548Z","avatar_url":"https://github.com/Tech-C-P.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🤖📚 ConversAI \n\n**[ConversAI](https://convers-ai-test.vercel.app/)** is an innovative conversational AI framework designed to empower users with intelligent interactions\nacross various document formats and web content. Utilizing advanced natural language processing (NLP) techniques,\nConversAI enables seamless text extraction and querying capabilities, making it an invaluable tool for researchers,\nstudents, professionals, and anyone who regularly interacts with text-based information.\n\n## Explore Our Chatbot Solutions\n\nFor advanced, tailored chatbot solutions, please visit [ConversAI Website](https://convers-ai-test.vercel.app/).\nWe offer a unique and customizable chatbot solution tailored to meet your specific needs. If you’re interested in enhancing your customer engagement with our chatbot technology, please reach out to us via email or visit our website for more information.\n\n### Demo Video\nhttps://github-production-user-asset-6210df.s3.amazonaws.com/45705878/376930530-b0cd1b42-a823-4577-8394-dee7d00ac12c.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20241016%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20241016T070830Z\u0026X-Amz-Expires=300\u0026X-Amz-Signature=645688e7c16270ca684c8c31fc729322d9aa9e0264416402f0881d2bacdd9b78\u0026X-Amz-SignedHeaders=host\n\n## 📖 Introduction \n\nIn an era characterized by information overload, efficient data processing is crucial. ConversAI addresses this\nchallenge by leveraging state-of-the-art technologies to transform unstructured data into actionable insights. Whether\nextracting meaningful information from PDFs, fetching transcripts from YouTube videos, or gathering data from multiple\nweb pages, ConversAI provides a user-friendly interface that simplifies these complex tasks.\n\n## 🏗️ Architecture Overview\n\n![PDF Upload](demo/conversai-architecture-diagram.png)\n\n### Key Technologies Used:\n\n- **Gradio**: For building the interactive user interface.\n- **Langchain**: For chaining various data retrieval and processing tasks.\n- **EasyOCR**: For optical character recognition in scannable PDFs.\n- **Pymupdf**: For handling searchable PDF documents.\n- **BeautifulSoup**: For web scraping and HTML parsing.\n\nWith its modular design, ConversAI is not just a tool but a platform that can be extended and customized to fit diverse\nuser requirements.\n\n## ✨ Features\n\n| Feature                      | Description                                                                 |\n|------------------------------|-----------------------------------------------------------------------------|\n| **Text Extraction**          | Extracts text from both searchable and scannable PDFs.                      |\n| **Web Crawler**              | Gathers and processes text from websites efficiently.                       |\n| **YouTube Transcript Retrieval** | Fetches and processes video transcripts for easy querying.                |\n| **Conversational Interface** | Interactive Gradio UI for seamless user interaction.                       |\n| **Configurable**             | Easily customizable to fit specific use cases.                              |\n| **Advanced Document Retrieval** | Utilizes sophisticated algorithms for improved accuracy and relevance.     |\n## 🌟 Benefits\n\n- **Time-Saving**: Automates text extraction, allowing users to focus on analysis rather than manual data entry.\n- **Accuracy**: High-precision text extraction capabilities ensure that users receive reliable data.\n- **User-Friendly**: Intuitive interface makes it accessible for users with varying technical skills.\n- **Scalable**: Modular architecture allows for easy integration of additional features and functionalities.\n\n## 🚀 Getting Started\n\n### 🗂️ Prerequisites\n\nBefore running ConversAI, ensure you have the following dependencies installed:\n\n```bash\napt-get update \u0026\u0026 apt-get upgrade -y\napt-get install poppler-utils -y\n```\n\nAdditionally, you need to set up your environment variables for the GROQ API:\n\n1. Sign up on [Groq](https://www.groq.com) and obtain your API key.\n2. Set the `GROQ_API_KEY` in your environment variables.\n\n### Installation\n\n1. Clone the repository:\n\n   ```bash\n   git https://github.com/Tech-C-P/ConversAI.git\n   cd ConversAI\n   ```\n\n2. Install the required packages:\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n### Running the Application\n\nTo launch the application, run the following command:\n\n```bash\npython app.py\n```\n\nThe Gradio interface will open in your default web browser.\n\n### Important Configuration Note\n\nIn case a GPU is unavailable, please modify the `config.ini` file as follows:\n\n- Under the `[EMBEDDINGS]` section, change:\n  ```ini\n  device = cuda\n  ```\n  to:\n  ```ini\n  device = cpu\n  ```\n\n- Under the `[EASYOCR]` section, change:\n  ```ini\n  gpu = true\n  ```\n  to:\n  ```ini\n  gpu = false\n  ```\n\nThese adjustments will ensure that the application runs smoothly on CPU resources.\n\n###  📝 Important Note \n\nAfter using the interface, be sure to click the \"Clear\" button to reset the fields. This is crucial because session\nmanagement has not been implemented in this version, and failing to clear inputs may lead to unintended data persistence\nduring subsequent interactions.\n\n## 🗂️ Directory Structure\n\nHere's a comprehensive view of the project's directory tree:\n\n```\nConversAI/\n├── app.py                      # Main application file\n├── requirements.txt            # Required Python packages\n├── src/                        # Source code directory\n│   ├── components/             # Component modules\n│   │   ├── loaders/            # Data loaders\n│   │   │   ├── pdfLoader.py\n│   │   │   ├── websiteCrawler.py\n│   │   │   └── youtubeLoader.py\n│   │   ├── rag/                # Retrieval-Augmented Generation components\n│   │   │   └── RAG.py\n│   │   ├── utils/              # Utility functions and classes\n│   │   │   ├── functions.py\n│   │   │   ├── exceptions.py\n│   │   │   └── logging.py\n│   │   └── vectors/            # Vector storage and processing\n│   │       └── vectorstore.py\n│   ├── pipelines/              # Pipeline logic for data processing\n│   │   └── completePipeline.py\n│   └── settings/               # Configuration files\n│       ├── config.ini\n│       └── params.yaml\n└── README.md                   # Project documentation\n```\n\n## 🎥 Demo\n\n### Text Input Interface\n\n![Text Input](demo/TextInterface.png)\n\n### PDF Upload Interface\n\n![PDF Upload](demo/PDFInterface.png)\n\n### Web Crawler Interface\n\n![Web Crawler](demo/WebsiteInterface.png)\n\n## 📝 Conclusion\n\nConversAI is more than just a tool; it’s a comprehensive solution for managing and extracting insights from a multitude\nof document formats and web sources. With its powerful capabilities and user-friendly interface, ConversAI is poised to\nmake information retrieval and processing easier and more efficient than ever before.\n\nSure! Here’s an updated section to include your contributions and acknowledgments:\n\n---\n\n## 👤 Author Information\n\n| Author           | 📧 Email                            | 💻 GitHub                                           | 🔗 LinkedIn                                           | 🐦 Twitter                                        | 🌐 Portfolio                                         |\n|------------------|-------------------------------------|-----------------------------------------------------|-------------------------------------------------------|---------------------------------------------------|------------------------------------------------------|\n| **Rauhan Ahmed** | rauhaan.siddiqui@gmail.com          | [RauhanAhmed](https://github.com/RauhanAhmed)       | [Rauhan Ahmed](https://www.linkedin.com/in/rauhan-ahmed) | [@ahmed_rauh46040](https://x.com/ahmed_rauh46040) | [rauhanahmed.org](https://rauhanahmed.org)           |\n| **Ishwor Subedi**| ishwor.subedi@email.com             | [ishworrsubedii](https://github.com/ishworrsubedii) | [Ishwor Subedi](https://www.linkedin.com/in/ishworrsubedii) | [@ishworr__](https://x.com/ishworr_)              | [ishwor-subedi.com.np](https://ishwor-subedi.com.np) |\n| **Faham**        | faham@email.com                     | [Faham](https://github.com/iamfaham)                | [Faham](https://www.linkedin.com/in/iamfaham)             | [@faham_twitter](https://x.com/iamfaham)          | [iamfaham.com](https://iamfaham.netlify.app)            |\n\n## 🙏 Acknowledgments \n\nThis project was developed while working as an AI Engineer\nat [Tech Consulting Partners](https://www.techconsultingpartners.com). I built ConversAI from scratch, implementing\nadvanced document retrieval methods, reranking techniques, hybrid search methodologies, multiple integrations with large\nlanguage models (LLMs), and lots of other complex functionalities.\n\nThe backend includes user management features, sophisticated data storage solutions (including S3 storage management),\ndatabase management, and vector databases. The deployment strategy leverages robust APIs, Docker containers, CI/CD\npractices, model monitoring, and cloud platform deployment.\n\nThis open-source prototype serves as a stepping stone towards a more comprehensive project aimed at public good,\nshowcasing the immense potential of advanced AI technologies in everyday applications. I extend my heartfelt gratitude\nto Tech Consulting Partners for entrusting me with this initiative and for their invaluable support throughout the\ndevelopment process.\n\n## 📜 License \n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n---\n\nWe hope you enjoy using ConversAI! For any questions or feedback, please reach out via the project repository or email.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftech-c-p%2Fconversai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftech-c-p%2Fconversai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftech-c-p%2Fconversai/lists"}