{"id":28654097,"url":"https://github.com/thewebscrapingclub/ai-cursor-scraping-assistant","last_synced_at":"2025-06-13T07:10:07.003Z","repository":{"id":287605817,"uuid":"965247491","full_name":"TheWebScrapingClub/AI-Cursor-Scraping-Assistant","owner":"TheWebScrapingClub","description":"A powerful tool that leverages Cursor AI and MCP (Model Context Protocol) to easily generate web scrapers for various types of websites.","archived":false,"fork":false,"pushed_at":"2025-04-12T18:45:05.000Z","size":15,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-04-12T19:36:02.964Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TheWebScrapingClub.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-12T18:27:49.000Z","updated_at":"2025-04-12T18:45:09.000Z","dependencies_parsed_at":"2025-04-13T08:45:33.709Z","dependency_job_id":null,"html_url":"https://github.com/TheWebScrapingClub/AI-Cursor-Scraping-Assistant","commit_stats":null,"previous_names":["thewebscrapingclub/ai-cursor-scraping-assistant"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TheWebScrapingClub/AI-Cursor-Scraping-Assistant","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheWebScrapingClub%2FAI-Cursor-Scraping-Assistant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheWebScrapingClub%2FAI-Cursor-Scraping-Assistant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheWebScrapingClub%2FAI-Cursor-Scraping-Assistant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheWebScrapingClub%2FAI-Cursor-Scraping-Assistant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TheWebScrapingClub","download_url":"https://codeload.github.com/TheWebScrapingClub/AI-Cursor-Scraping-Assistant/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheWebScrapingClub%2FAI-Cursor-Scraping-Assistant/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259599328,"owners_count":22882357,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-13T07:10:06.279Z","updated_at":"2025-06-13T07:10:06.992Z","avatar_url":"https://github.com/TheWebScrapingClub.png","language":"Python","funding_links":[],"categories":["Browser Automation"],"sub_categories":["How to Submit"],"readme":"# AI-Cursor-Scraping-Assistant\n\nA powerful tool that leverages Cursor AI and MCP (Model Context Protocol) to easily generate web scrapers for various types of websites. This project helps you quickly analyze websites and generate proper Scrapy or Camoufox scrapers with minimal effort.\n\n## Project Overview\n\nThis project contains two main components:\n\n1. **Cursor Rules** - A set of rules that teach Cursor AI how to analyze websites and create different types of Scrapy spiders\n2. **MCP Tools** - A collection of Model Context Protocol tools that enhance Cursor's capabilities for web scraping tasks\n\n## Prerequisites\n\n- [Cursor AI](https://cursor.sh/) installed\n- Python 3.10+ installed\n- Basic knowledge of web scraping concepts\n\n## Installation\n\nClone this repository to your local machine:\n\n```bash\ngit clone https://github.com/TheWebScrapingClub/AI-Cursor-Scraping-Assistant.git\ncd AI-Cursor-Scraping-Assistant\n```\n\nInstall the required dependencies:\n\n```bash\npip install mcp camoufox scrapy\n```\n\nIf you plan to use Camoufox, you'll need to fetch its browser binary:\n\n```bash\npython -m camoufox fetch\n```\n\n## Setup\n\n### Setting Up MCP Server\n\nThe MCP server provides tools that help Cursor AI analyze web pages and generate XPath selectors. To start the MCP server:\n\n1. Navigate to the MCPfiles directory:\n   ```bash\n   cd MCPfiles\n   ```\n\n2. Update the `CAMOUFOX_FILE_PATH` in `xpath_server.py` to point to your local `Camoufox_template.py` file.\n\n3. Start the MCP server:\n   ```bash\n   python xpath_server.py\n   ```\n\n4. In Cursor, connect to the MCP server by configuring it in the settings or using the MCP panel.\n\n### Cursor Rules\n\nThe cursor-rules directory contains rules that teach Cursor AI how to analyze websites and create different types of scrapers. These rules are automatically loaded when you open the project in Cursor.\n\n## Detailed Cursor Rules Explanation\n\nThe `cursor-rules` directory contains a set of MDC (Markdown Configuration) files that guide Cursor's behavior when creating web scrapers:\n\n### `prerequisites.mdc`\nThis rule handles initial setup tasks before creating any scrapers:\n- Gets the full path of the current project using `pwd`\n- Stores the path in context for later use by other rules\n- Confirms the execution of preliminary actions before proceeding\n\n### `website-analysis.mdc`\nThis comprehensive rule guides Cursor through website analysis:\n- Identifies the type of Scrapy spider to build (PLP, PDP, etc.)\n- Fetches and stores homepage HTML and cookies\n- Strips CSS using the MCP tool to simplify HTML analysis\n- Checks cookies for anti-bot protection (Akamai, Datadome, PerimeterX, etc.)\n- For PLP scrapers: fetches category pages, analyzes structure, looks for JSON data\n- For PDP scrapers: fetches product pages, analyzes structure, looks for JSON data\n- Detects schema.org markup and modern frameworks like Next.js\n\n### `scrapy-step-by-step-process.mdc`\nThis rule provides the execution flow for creating scrapers:\n- Outlines the sequence of steps to follow\n- References other rule files in the correct order\n- Ensures prerequisite actions are completed before scraper creation\n- Guides Cursor to analyze the website before generating code\n\n### `scrapy.mdc`\nThis extensive rule contains Scrapy best practices:\n- Defines recommended code organization and directory structure\n- Details file naming conventions and module organization\n- Provides component architecture guidelines\n- Offers strategies for code splitting and reuse\n- Includes performance optimization recommendations\n- Covers security practices, error handling, and logging\n- Provides specific syntax examples and code snippets\n\n### `scraper-models.mdc`\nThis rule defines the different types of scrapers that can be created:\n- **E-commerce PLP**: Details the data structure, field definitions, and implementation steps\n- **E-commerce PDP**: Details the data structure, field definitions, and implementation steps\n- Field mapping guidelines for all scraper types\n- Step-by-step instructions for creating each type of scraper\n- Default settings recommendations\n- Anti-bot countermeasures for different protection systems\n\n## Usage\n\nHere's how to use the AI-Cursor-Scraping-Assistant:\n\n1. Open the project in Cursor AI\n2. Make sure the MCP server is running\n3. Ask Cursor to create a scraper with a prompt like:\n   ```\n   Write an e-commerce PLP scraper for the website gucci.com\n   ```\n\nCursor will then:\n1. Analyze the website structure\n2. Check for anti-bot protection\n3. Extract the relevant HTML elements\n4. Generate a complete Scrapy spider based on the website type\n\n## Available Scraper Types\n\nYou can request different types of scrapers:\n\n- **E-commerce PLP (Product Listing Page)** - Scrapes product catalogs/category pages\n- **E-commerce PDP (Product Detail Page)** - Scrapes detailed product information\n\nFor example:\n```\nWrite an e-commerce PDP scraper for nike.com\n```\n\n## Advanced Usage\n\n### Camoufox Integration\n\nThe project includes a Camoufox template for creating stealth scrapers that can bypass certain anti-bot measures. The MCP tools help you:\n\n1. Fetch page content using Camoufox\n2. Generate XPath selectors for the desired elements\n3. Create a complete Camoufox scraper based on the template\n\n### Custom Scrapers\n\nYou can extend the functionality by adding new scraper types to the cursor-rules files. The modular design allows for easy customization.\n\n## Project Structure\n\n```\nAI-Cursor-Scraping-Assistant/\n├── MCPfiles/\n│   ├── xpath_server.py     # MCP server with web scraping tools\n│   └── Camoufox_template.py # Template for Camoufox scrapers\n├── cursor-rules/\n│   ├── website-analysis.mdc    # Rules for analyzing websites\n│   ├── scrapy.mdc              # Best practices for Scrapy\n│   ├── scrapy-step-by-step-process.mdc # Guide for creating scrapers\n│   ├── scraper-models.mdc      # Templates for different scraper types\n│   └── prerequisites.mdc       # Setup requirements\n└── README.md\n```\n\n## TODO: Future Enhancements\n\nThe following features are planned for future development:\n\n### Proxy Integration\n- Add proxy support when requested by the operator\n- Implement proxy rotation strategies\n- Support for different proxy providers\n- Handle proxy authentication\n- Integrate with popular proxy services\n\n### Improved XPath Generation and Validation\n- Add validation mechanisms for generated XPath selectors\n- Implement feedback loop for selector refinement\n- Control flow management for reworking selectors\n- Auto-correction of problematic selectors\n- Handle edge cases like dynamic content and AJAX loading\n\n### Other Planned Features\n- Support for more scraper types (news sites, social media, etc.)\n- Integration with additional anti-bot bypass techniques\n- Enhanced JSON extraction capabilities\n- Support for more complex navigation patterns\n- Multi-page scraping optimizations\n\n## References\n\nThis project is based on articles from The Web Scraping Club:\n\n- [Claude \u0026 Cursor AI Scraping Assistant](https://substack.thewebscraping.club/p/claude-cursor-ai-scraping-assistant)\n- [Cursor MCP Web Scraping Assistant](https://substack.thewebscraping.club/p/cursor-mcp-web-scraping-assistant)\n\nFor more information on web scraping techniques and best practices, visit [The Web Scraping Club](https://thewebscrapingclub.com).\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details. ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthewebscrapingclub%2Fai-cursor-scraping-assistant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthewebscrapingclub%2Fai-cursor-scraping-assistant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthewebscrapingclub%2Fai-cursor-scraping-assistant/lists"}