{"id":27879454,"url":"https://github.com/jatin-mehra119/crawlgpt","last_synced_at":"2025-05-05T03:22:05.900Z","repository":{"id":286150081,"uuid":"921030316","full_name":"Jatin-Mehra119/CRAWLGPT","owner":"Jatin-Mehra119","description":"A powerful web content crawler with LLM-powered RAG (Retrieval Augmented Generation) capabilities. CrawlGPT extracts content from URLs, processes it through intelligent summarization, and enables natural language interactions using modern LLM technology.","archived":false,"fork":false,"pushed_at":"2025-04-04T15:53:01.000Z","size":112,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-04T16:42:16.300Z","etag":null,"topics":["ai-chatbot","async-programming","data-extraction-from-websites","database-management","faiss","llm","web-crawler"],"latest_commit_sha":null,"homepage":"https://huggingface.co/spaces/jatinmehra/CRAWL-GPT-CHAT","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jatin-Mehra119.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-23T07:53:37.000Z","updated_at":"2025-04-04T15:53:05.000Z","dependencies_parsed_at":"2025-04-04T16:42:17.811Z","dependency_job_id":"300804c7-28c3-4df8-9a24-423680a13025","html_url":"https://github.com/Jatin-Mehra119/CRAWLGPT","commit_stats":null,"previous_names":["jatin-mehra119/crawlgpt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jatin-Mehra119%2FCRAWLGPT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jatin-Mehra119%2FCRAWLGPT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jatin-Mehra119%2FCRAWLGPT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jatin-Mehra119%2FCRAWLGPT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jatin-Mehra119","download_url":"https://codeload.github.com/Jatin-Mehra119/CRAWLGPT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252430275,"owners_count":21746630,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-chatbot","async-programming","data-extraction-from-websites","database-management","faiss","llm","web-crawler"],"created_at":"2025-05-05T03:22:05.379Z","updated_at":"2025-05-05T03:22:05.887Z","avatar_url":"https://github.com/Jatin-Mehra119.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CrawlGPT 🤖\n\nA powerful web content crawler with LLM-powered RAG (Retrieval Augmented Generation) capabilities. CrawlGPT extracts content from URLs, processes it through intelligent summarization, and enables natural language interactions using modern LLM technology.\n\n## 🌟 Key Features\n\n### Core Features\n- **Intelligent Web Crawling**\n  - Async web content extraction using Playwright\n  - Smart rate limiting and validation\n  - Configurable crawling strategies\n  \n- **Advanced Content Processing** \n  - Automatic text chunking and summarization\n  - Vector embeddings via FAISS\n  - Context-aware response generation\n\n- **Streamlit Chat Interface**\n  - Clean, responsive UI\n  - Real-time content processing\n  - Conversation history\n  - User authentication\n\n### Technical Features\n- **Vector Database**\n  - FAISS-powered similarity search\n  - Efficient content retrieval\n  - Persistent storage\n\n- **User Management**\n  - SQLite database backend\n  - Secure password hashing\n  - Chat history tracking\n\n- **Monitoring \u0026 Utils**\n  - Request metrics collection\n  - Progress tracking\n  - Data import/export\n  - Content validation\n\n## 🎥 Demo\n### [Deployed APP 🚀🤖](https://huggingface.co/spaces/jatinmehra/CRAWL-GPT-CHAT)\n\n[streamlit-chat_app video.webm](https://github.com/user-attachments/assets/ae1ddca0-9e3e-4b00-bf21-e73bb8e6cfdf)\n  \n_Example of CRAWLGPT in action!_\n\n## 🔧 Requirements\n\n-   Python \u003e= 3.8\n-   Operating System: OS Independent\n-   Required packages are handled by the setup script.\n\n\n## 🚀 Quick Start\n\n1.  Clone the Repository:\n    \n    ```git clone https://github.com/Jatin-Mehra119/CRAWLGPT.git\n    cd CRAWLGPT\n    ```\n    \n2.  Run the Setup Script:\n\n    ```\n    python -m setup_env\n    ``` \n    \n    _This script installs dependencies, creates a virtual environment, and prepares the project._\n    \n3.  Update Your Environment Variables:\n    \n    -   Create or modify the `.env` file.\n    -   Add your Groq API key and Ollama API key. Learn how to get API keys.\n    \n    \n    ```\n    GROQ_API_KEY=your_groq_api_key_here\n    OLLAMA_API_TOKEN=your_ollama_api_key_here\n    ```\n    \n4.  Activate the Virtual Environment:\n    \n    ```\n    source .venv/bin/activate  # On Unix/macOS\n    .venv\\Scripts\\activate  # On Windows\n    ``` \n    \n5.  Run the Application:\n\t```\n\tpython -m streamlit run src/crawlgpt/ui/chat_app.py\n\t```\n\n## 📦 Dependencies\n\n### Core Dependencies\n\n-   `streamlit==1.41.1`\n-   `groq==0.15.0`\n-   `sentence-transformers==3.3.1`\n-   `faiss-cpu==1.9.0.post1`\n-   `crawl4ai==0.4.247`\n-   `python-dotenv==1.0.1`\n-   `pydantic==2.10.5`\n-   `aiohttp==3.11.11`\n-   `beautifulsoup4==4.12.3`\n-   `numpy==2.2.0`\n-   `tqdm==4.67.1`\n-   `playwright\u003e=1.41.0`\n-   `asyncio\u003e=3.4.3`\n\n### Development Dependencies\n\n-   `pytest==8.3.4`\n-   `pytest-mockito==0.0.4`\n-   `black==24.2.0`\n-   `isort==5.13.0`\n-   `flake8==7.0.0`\n\n## 🏗️ Project Structure\n\n\n```\ncrawlgpt/\n├── src/\n│   └── crawlgpt/\n│       ├── core/                         # Core functionality\n│       │   ├── database.py                 # SQL database handling\n│       │   ├── LLMBasedCrawler.py          # Main crawler implementation\n│       │   ├── DatabaseHandler.py          # Vector database (FAISS)\n│       │   └── SummaryGenerator.py         # Text summarization\n│       ├── ui/                           # User Interface\n│       │   ├── chat_app.py                 # Main Streamlit app\n│       │   ├── chat_ui.py                  # Development UI\n│       │   └── login.py                    # Authentication UI\n│       └── utils/                        # Utilities\n│           ├── content_validator.py        # URL/content validation\n│           ├── data_manager.py             # Import/export handling\n│           ├── helper_functions.py         # General helpers\n│           ├── monitoring.py               # Metrics collection\n│           └── progress.py                 # Progress tracking\n├── tests/                                # Test suite\n│   └── test_core/\n│       ├── test_database_handler.py       # Vector DB tests\n│       ├── test_integration.py            # Integration tests\n│       ├── test_llm_based_crawler.py      # Crawler tests\n│       └── test_summary_generator.py      # Summarizer tests\n├── .github/                             # CI/CD\n│   └── workflows/\n│       └── Push_to_hf.yaml              # HuggingFace sync\n├── Docs/\n│   └── MiniDoc.md                     # Documentation\n├── .dockerignore                      # Docker exclusions\n├── .gitignore                         # Git exclusions\n├── Dockerfile                         # Container config\n├── LICENSE                            # MIT License\n├── README.md                          # Project documentation\n├── README_hf.md                       # HuggingFace README\n├── pyproject.toml                     # Project metadata\n├── pytest.ini                         # Test configuration\n├── crawlgpt.db                        # Database \n└── setup_env.py                       # Environment setup\n``` \n\n## 🧪 Testing\n\nRun all tests\n```\npython -m pytest\n```\n_The tests include unit tests for core functionality and integration tests for end-to-end workflows._\n\n## 📝 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## 🔗 Links\n\n-   [Bug Tracker](https://github.com/Jatin-Mehra119/crawlgpt/issues)\n-   [Documentation](https://github.com/Jatin-Mehra119/crawlgpt/wiki)\n-   [Source Code](https://github.com/Jatin-Mehra119/crawlgpt)\n\n## 🧡 Acknowledgments\n\n-   Inspired by the potential of GPT models for intelligent content processing.\n-   Special thanks to the creators of Crawl4ai, Groq, FAISS, and Playwright for their powerful tools.\n\n## 👨‍💻 Author\n\n-   Jatin Mehra (jatinmehra@outlook.in)\n\n## 🤝 Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request. For major changes, open an issue first to discuss your proposal.\n\n1.  Fork the Project.\n2.  Create your Feature Branch:\n    ```\n    git checkout -b feature/AmazingFeature`\n    ```\n3.  Commit your Changes:\n    ```\n    git commit -m 'Add some AmazingFeature\n    ```\n4.  Push to the Branch:\n    ```\n    git push origin feature/AmazingFeature\n    ```\n5.  Open a Pull Request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjatin-mehra119%2Fcrawlgpt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjatin-mehra119%2Fcrawlgpt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjatin-mehra119%2Fcrawlgpt/lists"}