{"id":20709907,"url":"https://github.com/oxylabs/how-to-scrape-amazon-product-data","last_synced_at":"2025-04-14T04:10:20.481Z","repository":{"id":257400836,"uuid":"851644708","full_name":"oxylabs/how-to-scrape-amazon-product-data","owner":"oxylabs","description":"The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions. ","archived":false,"fork":false,"pushed_at":"2025-02-10T12:49:08.000Z","size":2528,"stargazers_count":32,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-27T18:03:17.197Z","etag":null,"topics":["amazon","amazon-scraper","python","web-scraper","web-scraping","web-scraping-python"],"latest_commit_sha":null,"homepage":"https://oxylabs.io/products/scraper-api/ecommerce/amazon","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-03T13:22:59.000Z","updated_at":"2025-03-23T06:15:59.000Z","dependencies_parsed_at":"2024-09-16T14:04:56.101Z","dependency_job_id":"bac47662-33c9-40e3-8049-38a1d4e3381a","html_url":"https://github.com/oxylabs/how-to-scrape-amazon-product-data","commit_stats":null,"previous_names":["oxylabs/how-to-scrape-amazon-product-data"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-amazon-product-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-amazon-product-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-amazon-product-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-amazon-product-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/how-to-scrape-amazon-product-data/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248819404,"owners_count":21166477,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon","amazon-scraper","python","web-scraper","web-scraping","web-scraping-python"],"created_at":"2024-11-17T02:08:46.522Z","updated_at":"2025-04-14T04:10:20.459Z","avatar_url":"https://github.com/oxylabs.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/amazon-scraper/refs/heads/main/Amazon-Scraper-API-1090x275.png)](https://oxylabs.go2cloud.org/aff_c?offer_id=7\u0026aff_id=877\u0026url_id=112)\n\n[![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/GbxmdGhZjq)\n\n# Scraping Amazon Product Data With Python\n\nYou can find an extended version of this guide on our [blog](https://oxylabs.io/blog/scrape-amazon-product-data).\n\nThis guide uses Python to scrape the following data points from Amazon:\n\n- Product name\n- Product rating\n- Product price\n- Product images\n- Product description\n\n## Contents\n\n- [Setting up](#setting-up)\n  + [Installing packages](#installing-packages)\n- [Scraping product data](#scraping-product-data)\n  + [1. Sending a GET request with custom headers](#1.-sending-a-get-request-with-custom-headers)\n  + [2. Locating and scraping product name](#2.-locating-and-scraping-product-name)\n  + [3. Locating and scraping product rating](#3.-locating-and-scraping-product-rating)\n  + [4. Locating and scraping product price](#4.-locating-and-scraping-product-price)\n  + [5. Locating and scraping product image](#5.-locating-and-scraping-product-image)\n  + [6. Locating and scraping product description](#6.-locating-and-scraping-product-description)\n  + [7. Handling product listing](#7.-handling-product-listing)\n  + [8. Exporting scraped product data to a CSV file](#8.-exporting-scraped-product-data-to-a-CSV-file)\n- [Reviewing the final script](#reviewing-the-final-script)\n- [An easier solution to extract Amazon data](#an-easier-solution-to-extract-Amazon-data)\n  + [Scraping products from search results](#scraping-products-from-search-results)\n  + [Extracting product details](#extracting-product-details)\n  + [Scraping products by ASIN](#scraping-products-by-ASIN)\n\n## Setting up\n\nCreate a folder to save your code files. Also, creating a virtual environment is generally a good practice.\n\nThe following commands work on macOS and Linux. The commands will create a virtual environment and activate it:\n\n```\npython3 -m venv .env\nsource .env/bin/activate\n```\n\nIf you are on Windows, these commands will vary a little:\n\n```\npython -m venv .env\n.env\\scripts\\activate\n```\n### Installing packages\n\n```\npython3 -m pip install requests beautifulsoup4 lxml pandas\n```\n\nFor Windows, use Python instead of Python3:\n\n```\npython -m pip install requests beautifulsoup4 lxml pandas\n```\n\nTo try the Requests library, create a new file with the name amazon.py and enter the following:\n\n```\nimport requests\nurl = 'https://www.amazon.com/Bose-QuietComfort-45-Bluetooth-Canceling-Headphones/dp/B098FKXT8L'\n\nresponse = requests.get(url)\n\nprint(response.text)\n```\n\nSave the file and run it from the terminal:\n\n```\npython3 amazon.py\n```\n\nIn most cases, you cannot view the desired HTML. Amazon will block this request, and you will see the following text in the response:\n\n```\nTo discuss automated access to Amazon data, please contact api-services-support@amazon.com.\n```\n\nIf you print the `response.status_code`, you will see that instead of getting 200, which means success, you may get 503, which means an error.\n\nAmazon knows this request was not using a browser and thus blocks it.\n\nMany websites employ this practice. Amazon will block your requests and return an error code beginning with 500 or sometimes even 400.\n\nThe solution is simple in most cases. You can send HTTP headers along with your request just like an actual browser.\n\nSometimes, sending only the `user-agent` is enough. At other times, you may need to send more headers. A good example is sending the `accept-language` header.\n\nTo identify the user-agent sent by your browser, press F12 and open the Network tab. Reload the page. Select the first request and examine Request Headers.\n\n![](https://raw.githubusercontent.com/oxylabs/how-to-scrape-amazon-product-data/main/images/Amazon%20(1).jpg?token=GHSAT0AAAAAACW62VSTLRWZD5SJMWB7ZKI6ZWYLXUA)\n\n![](https://oxylabs.io/_next/image?url=https%3A%2F%2Foxylabs.io%2Foxylabs-web%2FZpBeQh5LeNNTxEWk_ZmK9sZm069VX1icx_Amazon-1-.jpg%3Fauto%3Dformat%2Ccompress\u0026w=1200\u0026q=75)\n\nYou can copy this user-agent and create a dictionary for the headers. \n\nThe following shows a dictionary with the `user-agent` and `accept-language` headers:\n\n```\ncustom_headers = {\n    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',\n    'accept-language': 'en-GB,en;q=0.9',\n}\n```\n\nYou can send this dictionary to the optional parameter of the `get` method as follows:\n\n```\nresponse = requests.get(url, headers= custom_headers)\n```\n\nExecuting the code with these changes may show the expected HTML with the product details.\n\nYou will not need Javascript rendering if you send as many headers as possible. If you need rendering, you will have to use tools like Playwright or Selenium. If the `User-Agent` and `Accept-Language` strings still bring you the `503` error, you can try to use the following headers:\n\n```\ncustom_headers = {\n    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15',\n    'Accept-Language': 'da, en-gb, en',\n    'Accept-Encoding': 'gzip, deflate, br',\n    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',\n    'Referer': 'https://www.google.com/'\n}\n```\n\nIt’s also a good idea to rotate different `User-Agent` strings and try your requests again to overcome the `503` error.\n\n## Scraping product data\n\nWhen scraping Amazon products, typically, you would work with two categories of pages — the category page and the product details page.\n\nFor example, open [this](https://www.amazon.com/b?node=12097479011) or search for Over-Ear Headphones on Amazon. The page that shows the search results is the category page.\n\nThe category page displays the product title, product image, product rating, product price, and, most importantly, the product URLs page. If you want more details, such as product descriptions, you will get them only from the product details page.\n\nLet's examine the structure of the product details page.\n\nOpen a product URL, such as [this](https://www.amazon.com/Bose-QuietComfort-45-Bluetooth-Canceling-Headphones/dp/B098FKXT8L), in Chrome or any other modern browser, right-click the product title, and select Inspect. You will see that the HTML markup of the product title is highlighted.\n\n![](https://oxylabs.io/_next/image?url=https%3A%2F%2Foxylabs.io%2Foxylabs-web%2FZpBeQx5LeNNTxEWl_ZmK9xZm069VX1ic0_Amazon-2-.jpg%3Fauto%3Dformat%2Ccompress\u0026w=1200\u0026q=75)\n\nYou will see that it is a span tag with its id attribute set to `productTitle`.\n\nSimilarly, if you right-click the price and select Inspect, you will see the HTML markup of the price.\n\n![](https://oxylabs.io/_next/image?url=https%3A%2F%2Foxylabs.io%2Foxylabs-web%2FZpBeRR5LeNNTxEWm_ZmK905m069VX1ic4_Amazon-3-.jpg%3Fauto%3Dformat%2Ccompress\u0026w=1200\u0026q=75)\n\nYou can see that the dollar component of the price is in a span tag with the class `a-price-whole`, and the cents component is in another span tag with the class set to `a-price-fraction`.\n\nSimilarly, you can locate the rating, image, and description.\n\n### 1. Sending a GET request with custom headers\n\n```\nfrom bs4 import BeautifulSoup\n\nresponse = requests.get(url, headers=custom_headers)\nsoup = BeautifulSoup(response.text, 'lxml')\n```\n\nThis guide uses CSS selectors. You can now use the `Soup` object to query for specific information.\n\n### 2. Locating and scraping product name\n\nThe product name or title is located in a `span` element with its id `productTitle`. It's easy to select elements using a unique ID.\n\n```\ntitle_element = soup.select_one('#productTitle')\n```\n\nSend the CSS selector to the `select_one` method, which returns an element instance. You can extract information from the text using the `text` attribute.\n\n```\ntitle = title_element.text\n```\n\nUpon printing, you will see that there are few white spaces. To fix that, add `.strip()` function call as follows:\n\n```\ntitle = title_element.text.strip()\n```\n\n### 3. Locating and scraping product rating\n\nCreate a selector for rating:\n\n```\n#acrPopover\n```\n\nThe following statement can select the element that contains the rating:\n\n```\nrating_element = soup.select_one('#acrPopover')\n```\n\nNote that the rating value is actually in the title attribute:\n\n```\nrating_text = rating_element.attrs.get('title')\nprint(rating_text)\n# prints '4.6 out of 5 stars'\n```\n\nLastly, use the `replace` method to get the number:\n\n```\nrating = rating_text.replace('out of 5 stars','')\n```\n\n### 4. Locating and scraping product price\n\nThe product price is located in two places: below the product title and on the Buy Now box. You can use either of these tags.\n\nCreate a CSS selector for the price:\n\n```\nspan.a-offscreen\n```\n\nThe CSS selector can be passed to the `select_one` method of BeautifulSoup as follows:\n\n```\nprice_element = soup.select_one('span.a-offscreen')\n```\n\nYou can now print the price:\n\n```\nprint(price_element.text)\n```\n\n### 5. Locating and scraping product image\n\nLet's scrape the default image. This image has the CSS selector as `#landingImage`. Write the following to get the image URL from the `src` attribute:\n\n```\nimage_element = soup.select_one('#landingImage')\nimage = image_element.attrs.get('src')\n```\n\n### 6. Locating and scraping product description\n\nThe methodology remains the same — create a CSS selector and use the `select_one` method.\n\n```\n#productDescription\n```\n\nYou can extract the element as follows:\n\n```\ndescription_element = soup.select_one('#productDescription').text.strip()\nprint(description_element)\n```\n\n### 7. Handling product listing\n\nTo reach the product information, begin with product listing or category pages.\n\nFor example, [here](https://www.amazon.com/b?node=12097479011) is the category page for over-ear headphones.\n\nNotice that all the products are contained in a `div` with the special attribute `[data-asin]`. In the `div`, all the product links are in an `h2` tag.\n\nThe CSS Selector is as follows:\n\n```\n[data-asin] h2 a\n```\n\nYou can read the `href` attribute of this selector and run a loop. However, note that the links will be relative. You would need to use the `urljoin` method to parse these links.\n\n```\nfrom urllib.parse import urljoin\n\ndef parse_listing(listing_url):\n    global visited_urls\n    response = requests.get(listing_url, headers=custom_headers)\n    print(response.status_code)\n    soup_search = BeautifulSoup(response.text, \"lxml\")\n    link_elements = soup_search.select(\"[data-asin] h2 a\")\n    page_data = []\n\n    for link in link_elements:\n        full_url = urljoin(listing_url, link.attrs.get(\"href\"))\n        if full_url not in visited_urls:\n            visited_urls.add(full_url)\n            print(f\"Scraping product from {full_url[:100]}\", flush=True)\n            product_info = get_product_info(full_url)\n            if product_info:\n                page_data.append(product_info)\n```\n\n#### Handling pagination\n\nThe link to the next page contains the text \"Next\". Look for this link using the contains operator of CSS as follows:\n\n```\n    next_page_el = soup_search.select_one('a.s-pagination-next')\n    if next_page_el:\n        next_page_url = next_page_el.attrs.get('href')\n        next_page_url = urljoin(listing_url, next_page_url)\n        print(f'Scraping next page: {next_page_url}', flush=True)\n        page_data += parse_listing(next_page_url)\n\n    return page_data\n```\n\n### 8. Exporting scraped product data to a CSV file\n\nThe scraped data is being returned as a dictionary. It is intentional. \n\nYou can create a list that contains all the scraped products:\n\n```\ndef main():\n    data = []\n    search_url = \"https://www.amazon.com/s?k=bose\u0026rh=n%3A12097479011\u0026ref=nb_sb_noss\"\n    data = parse_listing(search_url)\n```\n\nThis `page_data` can then be used to create a Pandas `DataFrame` object:\n\n```\n    df = pd.DataFrame(data)\n    df.to_csv(\"headphones.csv\", index=False)\n```\n\n## Reviewing the final script\n\nPutting together everything, here is the final script:\n\n```\nimport requests\nfrom bs4 import BeautifulSoup\nfrom urllib.parse import urljoin\nimport pandas as pd\n\ncustom_headers = {\n    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15',\n    'Accept-Language': 'da, en-gb, en',\n    'Accept-Encoding': 'gzip, deflate, br',\n    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',\n    'Referer': 'https://www.google.com/'\n}\n\nvisited_urls = set()\n\ndef get_product_info(url):\n    response = requests.get(url, headers=custom_headers)\n    if response.status_code != 200:\n        print(f\"Error in getting webpage: {url}\")\n        return None\n\n    soup = BeautifulSoup(response.text, \"lxml\")\n\n    title_element = soup.select_one(\"#productTitle\")\n    title = title_element.text.strip() if title_element else None\n\n    price_element = soup.select_one('span.a-offscreen')\n    price = price_element.text if price_element else None\n\n    rating_element = soup.select_one(\"#acrPopover\")\n    rating_text = rating_element.attrs.get(\"title\") if rating_element else None\n    rating = rating_text.replace(\"out of 5 stars\", \"\") if rating_text else None\n\n    image_element = soup.select_one(\"#landingImage\")\n    image = image_element.attrs.get(\"src\") if image_element else None\n\n    description_element = soup.select_one(\"#productDescription\")\n    description = description_element.text.strip() if description_element else None\n\n    return {\n        \"title\": title,\n        \"price\": price,\n        \"rating\": rating,\n        \"image\": image,\n        \"description\": description,\n        \"url\": url\n    }\n\n\ndef parse_listing(listing_url):\n    global visited_urls\n    response = requests.get(listing_url, headers=custom_headers)\n    print(response.status_code)\n    soup_search = BeautifulSoup(response.text, \"lxml\")\n    link_elements = soup_search.select(\"[data-asin] h2 a\")\n    page_data = []\n\n    for link in link_elements:\n        full_url = urljoin(listing_url, link.attrs.get(\"href\"))\n        if full_url not in visited_urls:\n            visited_urls.add(full_url)\n            print(f\"Scraping product from {full_url[:100]}\", flush=True)\n            product_info = get_product_info(full_url)\n            if product_info:\n                page_data.append(product_info)\n\n    next_page_el = soup_search.select_one('a.s-pagination-next')\n    if next_page_el:\n        next_page_url = next_page_el.attrs.get('href')\n        next_page_url = urljoin(listing_url, next_page_url)\n        print(f'Scraping next page: {next_page_url}', flush=True)\n        page_data += parse_listing(next_page_url)\n\n    return page_data\n\n\ndef main():\n    data = []\n    search_url = \"https://www.amazon.com/s?k=bose\u0026rh=n%3A12097479011\u0026ref=nb_sb_noss\"\n    data = parse_listing(search_url)\n    df = pd.DataFrame(data)\n    df.to_csv(\"headphones.csv\", orient='records')\n\n\nif __name__ == '__main__':\n    main()\n```\n\n## An easier solution to extract Amazon data\n\nYou can simplify the whole process with Oxylabs [Amazon Scraper](https://oxylabs.io/products/scraper-api/ecommerce/amazon) (a free trial is available).\n\n### Scraping products from search results\n\nExtract product data with the following code:\n\n```\nimport requests\nfrom pprint import pprint\n\n# Structure payload.\npayload = {\n    'source': 'amazon_search',\n    'query': 'bose',  # Search for \"bose\"\n    'start_page': 1,\n    'pages': 10,\n    'parse': True,\n    'context': [\n        {'key': 'category_id', 'value': 12097479011}  # category id for headphones\n    ],\n}\n\n# Get response\nresponse = requests.request(\n    'POST',\n    'https://realtime.oxylabs.io/v1/queries',\n    auth=('USERNAME', 'PASSWORD'),\n    json=payload,\n)\n\n# Print prettified response to stdout.\npprint(response.json())\n```\n\nNotice how it requests 10 pages beginning with the page 1. Also, we limit the search to category ID 12097479011, which is Amazon's category ID for headphones. You’ll get the data in JSON format:\n\n![](https://oxylabs.io/_next/image?url=https%3A%2F%2Foxylabs.io%2Foxylabs-web%2FZpBeRh5LeNNTxEWn_0dcb25ef-f532-49c2-8ef5-5960d9773bd3_amazon_product_search.png%3Fauto%3Dformat%2Ccompress\u0026w=1200\u0026q=75)\n\n### Extracting product details\n\nYou only need the product URL, regardless of the country where the Amazon store is located. The only code change is the payload.\n\nThe following payload extracts details, such as name, price, stock availability, description, and more, for the Bose QC 45:\n\n```\npayload = {\n    'source': 'amazon',\n    'url': 'https://www.amazon.com/dp/B098FKXT8L',\n    'parse': True\n}\n```\n\nThe output:\n\n![](https://oxylabs.io/_next/image?url=https%3A%2F%2Foxylabs.io%2Foxylabs-web%2FZpBeRx5LeNNTxEWo_fddcfa94-6d5c-4a61-b9ff-7035108bf36d_amazon_product_details.png%3Fauto%3Dformat%2Ccompress\u0026w=1200\u0026q=75)\n\n### Scraping products by ASIN\n\nAnother way to get data is by the ASIN of a product. You need to modify the payload:\n\n```\npayload = {\n    'source': 'amazon_product',\n    'domain': 'co.uk',\n    'query': 'B098FKXT8L',\n    'parse': True,\n    'context': [\n        {'key': 'autoselect_variant', 'value': True}\n    ]\n}\n```\n\nNote the optional parameter `domain`. Use this parameter to get Amazon data from any domain, such as amazon.co.uk.\n\nLooking to scrape more other Amazon data? [Amazon Review Scraper](https://github.com/oxylabs/amazon-review-scraper), [Amazon ASIN Scraper](https://github.com/oxylabs/amazon-asin-scraper), [Bypass Amazon CAPTCHA](https://github.com/oxylabs/how-to-bypass-amazon-captcha), [How to Scrape Amazon Prices](https://github.com/oxylabs/how-to-scrape-amazon-prices)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fhow-to-scrape-amazon-product-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fhow-to-scrape-amazon-product-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fhow-to-scrape-amazon-product-data/lists"}