{"id":46231650,"url":"https://github.com/oxylabs/ecommerce-category-scraper","last_synced_at":"2026-03-03T18:01:45.654Z","repository":{"id":317508689,"uuid":"1061858903","full_name":"oxylabs/ecommerce-category-scraper","owner":"oxylabs","description":"AI-Powered E-commerce Category Scraper with AI Studio","archived":false,"fork":false,"pushed_at":"2025-10-01T10:30:13.000Z","size":986,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-01T11:32:36.985Z","etag":null,"topics":["ai-studio","data-extraction","ecommerce","low-code","price-comparison","web-scraping"],"latest_commit_sha":null,"homepage":"https://aistudio.oxylabs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-22T13:33:52.000Z","updated_at":"2025-10-01T10:30:16.000Z","dependencies_parsed_at":"2025-10-01T11:32:39.666Z","dependency_job_id":"5bb60b12-5f58-4bc6-8672-a43293da7a72","html_url":"https://github.com/oxylabs/ecommerce-category-scraper","commit_stats":null,"previous_names":["oxylabs/ecommerce-category-scraper"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/oxylabs/ecommerce-category-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fecommerce-category-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fecommerce-category-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fecommerce-category-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fecommerce-category-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/ecommerce-category-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fecommerce-category-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30053990,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-03T17:46:22.538Z","status":"ssl_error","status_checked_at":"2026-03-03T17:46:22.036Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-studio","data-extraction","ecommerce","low-code","price-comparison","web-scraping"],"created_at":"2026-03-03T18:01:44.878Z","updated_at":"2026-03-03T18:01:45.644Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ecommerce Category Scraper\n\n[![AI-Studio Python (1)](https://github.com/oxylabs/ecommerce-category-scraper/blob/main/Ai-Studio2.png)](https://aistudio.oxylabs.io/?utm_source=877\u0026utm_medium=affiliate\u0026utm_campaign=ai_studio\u0026groupid=877\u0026utm_content=ai-studio-js-github\u0026transaction_id=102f49063ab94276ae8f116d224b67) \n\n\n[![](https://dcbadge.limes.pink/api/server/Pds3gBmKMH?style=for-the-badge\u0026theme=discord)](https://discord.gg/Pds3gBmKMH) [![YouTube](https://img.shields.io/badge/YouTube-Oxylabs-red?style=for-the-badge\u0026logo=youtube\u0026logoColor=white)](https://www.youtube.com/@oxylabs)\n\n## 🛒 E-commerce Category Scraper\n\nAI-Powered E-commerce Category Scraper with AI Studio\n\nThe E-commerce Category Scraper is an AI-powered, open-source tool built using Oxylabs AI Studio. It automates and streamlines data extraction from e-commerce websites, making it accessible to developers of all skill levels. This solution can also be adapted as a scalable price comparison tool, perfect for analyzing competitor pricing and market trends.\n\nWhat problems does this tool solve?\n- Scraping all products from ecommerce categories without writing custom code;\n- Automatically locating ecommerce categories and scraping their products;\n\n## 🚀 Key features\n\n- **Cost optimization**: AI Studio ensures task-based scalability.\n- **Pre-built solution**: A ready-to-use open-source tool for faster adoption and seamless integration.\n- **Low-code automation**: Create automated workflows for web scraping and parsing without needing advanced coding skills.\n- **AI-powered capabilities**: Extract structured web data with minimal effort using Oxylabs’ AI Studio apps.\n- **Enterprise-ready infrastructure**: Handle CAPTCHAs, bypass IP blocks, and navigate dynamic content effortlessly.\n- **Flexible scalability**: Perfect for small tasks using free AI Studio credits or scaling to enterprise-level projects.\n\n## 🤖 How it works\n\n- **Browser Agent**: Locates category in the website and collects all category pagination URLs.\n- **AI-Scraper**: Extracts all product URLs from category listing pages.\n- **AI-Scraper**: Extracts structured product data like pricing, titles, and stock availability. Based on user prompt or JSON schema.\n- **Final Output**: Clean, structured datasets ready for use in analytics, reporting, or pricing workflows. Which can be saved to JSON file or returned to the user programatically.\n\n\n## ✅ Prerequisites\n\nBefore you begin, make sure you have Oxylabs AI studio API key. Obtain your API key from [Oxylabs AI Studio](https://aistudio.oxylabs.io/settings/api-key). (1000 credits free).\n\n## 📦 Instalation \n\n- Open your terminal.\n- Install the uv package manager:\n  ```bash\n  # macOS and Linux\n  curl -LsSf https://astral.sh/uv/install.sh | sh\n  ```\n\n- Clone the repository:\n  ```bash\n  git clone https://github.com/oxylabs/ecommerce-category-scraper.git\n  ```\n\n- Navigate to the repository:\n  ```bash\n  cd ecommerce-category-scraper\n  ```\n\n- Install the dependencies:\n  ```bash\n  uv sync\n  ```\n- Enable the virtual environment:\n  ```bash\n  source .venv/bin/activate\n  ```\n\n## 🧪 Running Tests\n\nBoth tests scrape books from `books.toscrape.com` (Sequential art category), extracting book name, price, UPC code, and availability.\n\n- **Test 1:** Accepts a category URL directly and scrapes products from that specific category page.\n  Replace `\u003cyour-api-key\u003e` with your actual API key.\n  ```bash\n  python -m test.test_1 --oxylabs-ai-studio-api-key \u003cyour-api-key\u003e\n  ```\n  Results are saved to `test_1_results.json`. View with:\n  ```bash\n  cat test_1_results.json | python -m json.tool\n  ```\n\n- **Test 2:** Accepts an ecommerce domain URL and automatically searches for and identifies category pages before scraping.\n  Replace `\u003cyour-api-key\u003e` with your actual API key.\n  ```bash\n  python -m test.test_2 --oxylabs-ai-studio-api-key \u003cyour-api-key\u003e\n  ```\n  Results are saved to `test_2_results.json`. View with:\n  ```bash\n  cat test_2_results.json | python -m json.tool\n  ```\n\n**Note:** Modify tests for different domains and settings.\n\n## 🐍 Python Interface\n\nUse `scrape_category` function to integrate the scraper into your code.\n\n```python\nfrom ecommerce_category_scraper.process import scrape_category\n\nresult = await scrape_category(\n    api_key=\"your-api-key\",\n    category_url=\"https://example.com/category\",  # OR use ecommerce_domain_url\n    parsing_prompt=\"Extract product name, price, and rating\",\n    max_products=50,\n)\n```\n\n### Parameters\n\n**Required:**\n- `api_key` - Oxylabs AI Studio API key\n\n**Category selection (choose one):**\n- `category_url` - Direct category URL (starts gathering product URLs immediately)\n- `ecommerce_domain_url` + `category_description_prompt` - Domain URL and description of category to search for\n\n**Parsing (choose one):**\n- `parsing_prompt` - Text description of data to extract\n- `json_schema` - JSON schema for structured extraction (more reliable and deterministic)\n\n**Optional:**\n- `geo_location` - IP location in ISO2 format (e.g., `\"US\"`)\n- `render_javascript` - Enable JavaScript rendering (default: `False`)\n- `json_filepath` - Save results to file (if not provided, returns list)\n- `max_pages` - Maximum category pages to scrape (default: all)\n- `max_products` - Maximum products to scrape (default: all)\n\n## 📈 Practical use cases\n- **Price comparison tool**: Automate workflows to compare competitor prices by category or region.\n- **Price monitoring**: Regularly track competitor pricing trends and fluctuations.\n- **Market intelligence**: Collect data for competitive and industry analysis.\n- **E-commerce scraping**: Extract essential product details for AI applications or business intelligence.\n- **Product detail extraction**: Automate the retrieval of pricing, inventory, and product descriptions.\n\n## 📝 FAQ\n\n- **Can I scrape any website using this tool?**\n\n  This tool can scrape most websites, but scraping capabilities depend on adhering to the website's legal and technical restrictions.\n- **Is this tool free?**\n  Yes, the E-commerce Category Scraper is open-source and free to use. Smaller tasks are powered by AI Studio’s free credits, while flexible plans allow scaling for larger workflows.\n- **Do I need advanced coding skills to use this tool?**\n  Advanced coding skills are not required. AI-powered code editor simplifies integration, making it accessible for engineers with basic coding experience.\n- **Can I customize this scraper for my needs?**\n  Yes, the open-source solution can be fully customized to meet specific workflow or business requirements.\n- **What are AI Studio free credits?**\n  AI Studio offers free credits for smaller tasks. For scaling beyond free credits, users can subscribe to flexible plans.\n\n\n## 🎥 Showcased at Oxycon 2025\nThis E-commerce Category Scraper was featured live at Oxycon 2025. The presentation demonstrated how AI Studio can be used to easily build real-time price comparison tool while showcasing how developers can create scalable scraping workflows for various e-commerce tasks.\n\n## 📚 Learn more\nFor a deeper dive into features, integrations, and examples, and documentation, visit the [AI Studio](https://aistudio.oxylabs.io/) website.\n\n## 💬 Contact us\nIf you have questions or need support, reach out to us at hello@oxylabs.io, through [live chat](https://oxylabs.drift.click/oxybot), or join our [Discord community](https://discord.com/invite/Pds3gBmKMH).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fecommerce-category-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fecommerce-category-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fecommerce-category-scraper/lists"}