{"id":25381669,"url":"https://github.com/oussemabenhassena5/crawl4deepseek","last_synced_at":"2025-04-09T12:29:34.552Z","repository":{"id":275957889,"uuid":"927665876","full_name":"oussemabenhassena5/Crawl4DeepSeek","owner":"oussemabenhassena5","description":"Crawl4DeepSeek = Crawl4AI + DeepSeek 🚀                                  Smart, efficient, and built for deep web exploration! 🌐🤖","archived":false,"fork":false,"pushed_at":"2025-02-09T13:54:12.000Z","size":24,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-15T06:36:30.377Z","etag":null,"topics":["crawl4ai","deepseek","python","webcrawling","webscraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oussemabenhassena5.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-05T10:39:48.000Z","updated_at":"2025-02-09T13:54:15.000Z","dependencies_parsed_at":"2025-02-05T14:37:13.193Z","dependency_job_id":null,"html_url":"https://github.com/oussemabenhassena5/Crawl4DeepSeek","commit_stats":null,"previous_names":["oussemabenhassena5/crawl4deepseek"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oussemabenhassena5%2FCrawl4DeepSeek","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oussemabenhassena5%2FCrawl4DeepSeek/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oussemabenhassena5%2FCrawl4DeepSeek/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oussemabenhassena5%2FCrawl4DeepSeek/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oussemabenhassena5","download_url":"https://codeload.github.com/oussemabenhassena5/Crawl4DeepSeek/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248039525,"owners_count":21037725,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawl4ai","deepseek","python","webcrawling","webscraping"],"created_at":"2025-02-15T06:32:38.747Z","updated_at":"2025-04-09T12:29:34.528Z","avatar_url":"https://github.com/oussemabenhassena5.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🕸️ DeepSeek Crawler\n\n[![Python 3.12](https://img.shields.io/badge/Python-3.12-blue.svg)](https://www.python.org/downloads/)\n[![Crawl4AI](https://img.shields.io/badge/Framework-Crawl4AI-orange.svg)](https://docs.crawl4ai.com/)\n[![DeepSeek](https://img.shields.io/badge/AI-DeepSeek-green.svg)](https://github.com/deepseek-ai/DeepSeek-V3?tab=readme-ov-file)\n[![MIT License](https://img.shields.io/badge/License-MIT-purple.svg)](LICENSE)\n\n\u003e *Unleashing AI-Powered Web Scraping at Scale* 🚀\n\n## 🎯 What Makes This Special\n\nDeep Seek Crawler represents the next generation of web scraping, combining asyncio's power with DeepSeek's AI capabilities to transform chaotic web data into structured intelligence. Built for performance, scalability, and precision.\n\n## 🔥 Key Features\n\n### Intelligence Layer\n- **Smart Pagination**: Autonomous detection of result boundaries and page termination\n- **Duplicate Prevention**: Intelligent tracking of seen venues using efficient set operations\n- **Polite Crawling**: Built-in rate limiting with configurable sleep intervals\n- **Robust Error Handling**: Graceful handling of no-results scenarios\n\n### Engineering Excellence\n- **Asynchronous Architecture**: Built on Python's asyncio for maximum performance\n- **Modular Design**: Clean separation of concerns with utility modules\n- **Session Management**: Persistent crawling sessions with automatic cleanup\n- **CSV Export**: Structured data output with comprehensive venue information\n\n## 🏗️ Architecture\n\n```mermaid\ngraph TD\n    A[Main Crawler] --\u003e B[AsyncWebCrawler]\n    B --\u003e C[Page Processor]\n    C --\u003e D[LLM Strategy]\n    D --\u003e E[Data Exporter]\n    B --\u003e F[Browser Config]\n    C --\u003e G[Data Utils]\n    G --\u003e E\n```\n\n## 💻 Technical Implementation\n\n### Core Components\n- **AsyncWebCrawler**: High-performance asynchronous crawling engine\n- **LLM Strategy**: AI-powered content extraction and processing\n- **Browser Configuration**: Customizable crawler behavior settings\n- **Data Utilities**: Robust data processing and export functionality\n\n### Performance Features\n- **Efficient Memory Usage**: Set-based duplicate detection\n- **Controlled Crawling**: Configurable delay between requests\n- **Graceful Termination**: Smart detection of crawl completion\n- **Usage Statistics**: Built-in LLM strategy usage tracking\n\n## 🚀 Quick Start\n\n1. **Clone \u0026 Setup**:\n    ```bash\n    git clone https://github.com/oussemabenhassena5/Crawl4DeepSeek.git\n    cd Crawl4DeepSeek\n    python -m venv venv \u0026\u0026 source venv/bin/activate\n    pip install -r requirements.txt\n    ```\n\n2. **Configure Environment**:\n    ```bash\n    # .env file\n    GROQ_API_KEY=your_api_key\n    ```\n\n3. **Launch Crawler**:\n    ```bash\n    python crawler.py\n    ```\n\n## 📊 Project Structure\n\n```\ncrawl4deepseek/\n├── crawler.py            # Main crawling script\n├── config.py            # Configuration settings\n├── utils/\n│   ├── data_utils.py    # Data processing utilities\n│   └── scraper_utils.py # Crawling utility functions\n├── requirements.txt     # Project dependencies\n└── .env                # Environment configuration\n```\n\n## 🛠️ Engineering Highlights\n\n- **Async Processing**: Efficient handling of concurrent page fetches\n- **Smart State Management**: Tracking of seen venues and crawl progress\n- **Configurable Behavior**: Easy-to-modify crawler settings\n- **Comprehensive Logging**: Detailed crawl progress and statistics\n\n## 🔄 Development Workflow\n\nThe crawler follows a systematic approach:\n1. Initializes configurations and strategies\n2. Processes pages asynchronously\n3. Checks for duplicate venues\n4. Exports structured data\n5. Provides usage statistics\n\n## 🎯 Future Roadmap\n\n- [ ] Enhanced error recovery mechanisms\n- [ ] Multi-site crawling support\n- [ ] Advanced data validation\n- [ ] Performance optimization for large-scale crawls\n\n## 🤝 Contributing\n\nContributions are welcome! Feel free to submit issues and pull requests.\n\n## 📜 License\n\nDistributed under the MIT License. See `LICENSE` for more information.\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n**Built with 💻 by [Oussema Ben Hassena](https://github.com/oussemabenhassena5)**\n\n*Transforming Web Data into Intelligence*\n\n[LinkedIn](linkedin.com/in/oussema-ben-hassena-b445122a4) \n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foussemabenhassena5%2Fcrawl4deepseek","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foussemabenhassena5%2Fcrawl4deepseek","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foussemabenhassena5%2Fcrawl4deepseek/lists"}