{"id":20710015,"url":"https://github.com/oxylabs/scraping-ecommerce-websites","last_synced_at":"2025-07-08T10:41:19.136Z","repository":{"id":216520899,"uuid":"741502249","full_name":"oxylabs/scraping-ecommerce-websites","owner":"oxylabs","description":"A guide to scraping e-commerce websites by overcoming bot protection and CAPTCHAs.","archived":false,"fork":false,"pushed_at":"2024-09-30T13:29:09.000Z","size":15,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-17T02:09:28.495Z","etag":null,"topics":["e-commerce-api","e-commerce-scraper","ecommerce-api","ecommerce-scraper","ecommerce-scraping","marketplace-api","marketplace-scraper","marketplace-scraper-api","scraper","scraper-api","scraper-python","web-scraping"],"latest_commit_sha":null,"homepage":"https://oxylabs.io/blog/scraping-ecommerce-websites","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2024-01-10T14:28:12.000Z","updated_at":"2024-11-11T02:35:57.000Z","dependencies_parsed_at":"2024-01-17T15:10:51.902Z","dependency_job_id":null,"html_url":"https://github.com/oxylabs/scraping-ecommerce-websites","commit_stats":null,"previous_names":["oxylabs/scraping-ecommerce-websites"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscraping-ecommerce-websites","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscraping-ecommerce-websites/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscraping-ecommerce-websites/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscraping-ecommerce-websites/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/scraping-ecommerce-websites/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234429243,"owners_count":18831240,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["e-commerce-api","e-commerce-scraper","ecommerce-api","ecommerce-scraper","ecommerce-scraping","marketplace-api","marketplace-scraper","marketplace-scraper-api","scraper","scraper-api","scraper-python","web-scraping"],"created_at":"2024-11-17T02:09:33.094Z","updated_at":"2025-07-08T10:41:19.111Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# How to Scrape E-Commerce Websites With Python\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.io/pages/gitoxy?utm_source=877\u0026utm_medium=affiliate\u0026groupid=877\u0026utm_content=scraping-ecommerce-websites-github\u0026transaction_id=102f49063ab94276ae8f116d224b67)\n\n\n[![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/Pds3gBmKMH)\n\nScraping e-commerce websites is easy with Python and Oxylabs [E-Commerce Scraper API](https://oxylabs.io/products/scraper-api/ecommerce) (now a part of Web Scraper API). Follow this quick guide to build your e-commerce scraper while utilizing a **1-week free API trial**.\n\nSee this [blog post](https://oxylabs.io/blog/scraping-ecommerce-websites) for a complete tutorial with detailed insights and images.\n\n- [1. Project setup](#1-project-setup)\n- [2. Install dependencies](#2-install-dependencies)\n- [3. Import libraries](#3-import-libraries)\n- [4. Retrieve API credentials](#4-retrieve-api-credentials)\n- [5. Prepare the payload](#5-prepare-the-payload)\n- [6. Send a POST request to the API](#6-send-a-post-request-to-the-api)\n- [7. Parse Data](#7-parse-data)\n  * [Title](#title)\n  * [Price](#price)\n  * [Availability](#availability)\n  * [All products](#all-products)\n- [Full source code](#full-source-code)\n- [Output](#output)\n\n## 1. Project setup\nFirst, you’ll have to install Python. Please download it from [here](https://www.python.org/downloads/). \n\n## 2. Install dependencies\n\n```bash\npip install bs4 requests\n```\n\n## 3. Import libraries\n\n```python\nimport requests\nfrom bs4 import BeautifulSoup\nfrom pprint import pprint\n```\n\n## 4. Retrieve API credentials\nNext, you’ll have to log in to your Oxylabs account to retrieve API credentials. If you don’t have an account yet, you can simply [sign up](https://dashboard.oxylabs.io/en/registration?_gl=1*1d3w2bu*_ga*MTYxNzk4NTY4OS4xNjkzMzg1MTgy*_ga_FC1FV63ZQ4*MTcwNDg4NjM5NS4xNDEuMS4xNzA0ODk3MDc3LjYwLjAuMA..*_gcl_aw*R0NMLjE3MDIwMjkzNjMuQ2p3S0NBaUFtc3VyQmhCdkVpd0E2ZS1XUEdPNkxWQkRrS2hUT0VVUGpNQzM2QURsQ1dzWElBbXVCQUVfbGFaMDFJT3lqdndkNEhOSkRCb0Nob0FRQXZEX0J3RQ..*_gcl_au*MTM0MTc1OTcxNy4xNzAxMDY5ODYxLjQ3MDAwMTI5MC4xNzA0ODczMjI5LjE3MDQ4NzU2ODM.) for a **free trial** and go to the dashboard. There, you’ll get the necessary credentials to replace the `USERNAME` and `PASSWORD` variables in the code below:\n\n```python\nusername, password = 'USERNAME', 'PASSWORD'\n```\n\n## 5. Prepare the payload\n\n```python\nurl = \"https://sandbox.oxylabs.io/products\"\npayload = {\n    'source': 'universal',\n    'render': 'html',\n    'url': url,\n}\n```\n## 6. Send a POST request to the API\n\n```python\nresponse = requests.post(\n    'https://realtime.oxylabs.io/v1/queries',\n    auth=(username, password),\n    json=payload,\n)\nprint(response.status_code)\n```\n## 7. Parse Data\n\n```python\ncontent = response.json()[\"results\"][0][\"content\"]\nsoup = BeautifulSoup(content, \"html.parser\")\n```\nLet's use CSS selectors to select specific elements in the HTML file of [https://sandbox.oxylabs.io/products](https://sandbox.oxylabs.io/products), which you can inspect via **Developer Tools** on your web browser:\n\n### Title\n\n```python\ntitle = soup.find('h4', {\"class\": \"title\"}).get_text(strip=True)\n```\n### Price\n\n```python\nprice = soup.find('div', {\"class\": \"price-wrapper\"}).get_text(strip=True)\n```\n### Availability\n\n```python\navailability = soup.find('p', {\"class\": [\"in-stock\", \"out-of-stock\"]}).get_text(strip=True)\n```\n### All products\n\n```python\ndata = []\nfor elem in soup.find_all(\"div\", {\"class\": \"product-card\"}):\n    title = elem.find('h4', {\"class\": \"title\"}).get_text(strip=True)\n    price = elem.find('div', {\"class\": \"price-wrapper\"}).get_text(strip=True)\n\n\n    availability = elem.find('p', {\"class\": [\"in-stock\", \"out-of-stock\"]}).get_text(strip=True)\n    data.append({\n        \"title\": title,\n        \"price\": price,\n        \"availability\": availability,\n    })\npprint(data)\n```\n\n## Full source code\n\n```python\nimport requests\nfrom bs4 import BeautifulSoup\nfrom pprint import pprint\n\n\nusername, password = 'USERNAME', 'PASSWORD'\nurl = \"https://sandbox.oxylabs.io/products\"\n\npayload = {\n    'source': 'universal',\n    'render': 'html',\n    'url': url,\n}\nresponse = requests.post(\n    'https://realtime.oxylabs.io/v1/queries',\n    auth=(username, password),\n    json=payload,\n)\nprint(response.status_code)\n\n\ncontent = response.json()[\"results\"][0][\"content\"]\nsoup = BeautifulSoup(content, \"html.parser\")\n\n\ndata = []\nfor elem in soup.find_all(\"div\", {\"class\": \"product-card\"}):\n    title = elem.find('h4', {\"class\": \"title\"}).get_text(strip=True)\n    price = elem.find('div', {\"class\": \"price-wrapper\"}).get_text(strip=True)\n\n\n    availability = elem.find('p', {\"class\": [\"in-stock\", \"out-of-stock\"]}).get_text(strip=True)\n    data.append({\n        \"title\": title,\n        \"price\": price,\n        \"availability\": availability,\n    })\npprint(data)\n```\nBy using the techniques described in this article, you can perform large-scale web scraping on websites that employ bot protection and CAPTCHAs.\n\n## Output\n\n```json\n200\n[\n  {\n    \"title\": \"The Legend of Zelda: Ocarina of Time\",\n    \"price\": \"91,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"Super Mario Galaxy\",\n    \"price\": \"91,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"Super Mario Galaxy 2\",\n    \"price\": \"91,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"Metroid Prime\",\n    \"price\": \"89,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"Super Mario Odyssey\",\n    \"price\": \"89,99 €\",\n    \"availability\": \"In stock\"\n  },\n  {\n    \"title\": \"Halo: Combat Evolved\",\n    \"price\": \"87,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"The House in Fata Morgana - Dreams of the Revenants Edition -\",\n    \"price\": \"83,99 €\",\n    \"availability\": \"In stock\"\n  },\n  {\n    \"title\": \"NFL 2K1\",\n    \"price\": \"62,99 €\",\n    \"availability\": \"In stock\"\n  },\n  {\n    \"title\": \"Uncharted 2: Among Thieves\",\n    \"price\": \"88,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"Tekken 3\",\n    \"price\": \"91,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"The Legend of Zelda: The Wind Waker\",\n    \"price\": \"90,99 €\",\n    \"availability\": \"In stock\"\n  },\n  {\n    \"title\": \"Gran Turismo\",\n    \"price\": \"86,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"Metal Gear Solid 2: Sons of Liberty\",\n    \"price\": \"88,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"Grand Theft Auto Double Pack\",\n    \"price\": \"81,99 €\",\n    \"availability\": \"In stock\"\n  },\n  {\n    \"title\": \"Baldur's Gate II: Shadows of Amn\",\n    \"price\": \"91,99 €\",\n    \"availability\": \"In stock\"\n  },\n  {\n    \"title\": \"Tetris Effect: Connected\",\n    \"price\": \"88,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"The Legend of Zelda Collector's Edition\",\n    \"price\": \"89,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"Gran Turismo 3: A-Spec\",\n    \"price\": \"84,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"The Legend of Zelda: A Link to the Past\",\n    \"price\": \"90,99 €\",\n    \"availability\": \"In stock\"\n  },\n  {\n    \"title\": \"The Legend of Zelda: Majora's Mask\",\n    \"price\": \"91,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"The Last of Us\",\n    \"price\": \"92,99 €\",\n    \"availability\": \"In stock\"\n  },\n  {\n    \"title\": \"Persona 5 Royal\",\n    \"price\": \"84,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"The Last of Us Remastered\",\n    \"price\": \"92,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"The Legend of Zelda: Ocarina of Time 3D\",\n    \"price\": \"90,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"Chrono Cross\",\n    \"price\": \"88,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"Gears of War\",\n    \"price\": \"84,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"Sid Meier's Civilization II\",\n    \"price\": \"88,99 €\",\n    \"availability\": \"In stock\"\n  },\n  {\n    \"title\": \"Halo 3\",\n    \"price\": \"81,99 €\",\n    \"availability\": \"In stock\"\n  },\n  {\n    \"title\": \"Ninja Gaiden Black\",\n    \"price\": \"88,99 €\",\n    \"availability\": \"In stock\"\n  },\n  {\n    \"title\": \"Super Mario Advance 4: Super Mario Bros. 3\",\n    \"price\": \"89,99 €\",\n    \"availability\": \"Out of Stock\"\n  },\n  {\n    \"title\": \"Jet Grind Radio\",\n    \"price\": \"83,99 €\",\n    \"availability\": \"In stock\"\n  },\n  {\n    \"title\": \"Grim Fandango\",\n    \"price\": \"91,99 €\",\n    \"availability\": \"Out of Stock\"\n  }\n]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fscraping-ecommerce-websites","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fscraping-ecommerce-websites","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fscraping-ecommerce-websites/lists"}