{"id":20709918,"url":"https://github.com/oxylabs/scrape-google-python","last_synced_at":"2025-09-21T11:14:06.297Z","repository":{"id":250833370,"uuid":"835593102","full_name":"oxylabs/scrape-google-python","owner":"oxylabs","description":"In this tutorial, we showcase how to scrape public Google data with Python and Oxylabs API.","archived":false,"fork":false,"pushed_at":"2025-04-09T08:38:54.000Z","size":37,"stargazers_count":47,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-14T04:12:15.969Z","etag":null,"topics":["google","python","python-scraping","python-web-scraper","scrape","scrape-google","scraper-api","scraping","scraping-api","web-scraping"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-30T06:38:36.000Z","updated_at":"2025-04-09T08:38:58.000Z","dependencies_parsed_at":"2024-07-30T10:40:00.092Z","dependency_job_id":"cd0ac848-bd79-4f2c-80fa-fbde65c038de","html_url":"https://github.com/oxylabs/scrape-google-python","commit_stats":null,"previous_names":["oxylabs/scrape-google-python"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oxylabs/scrape-google-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrape-google-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrape-google-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrape-google-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrape-google-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/scrape-google-python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrape-google-python/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259695307,"owners_count":22897514,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google","python","python-scraping","python-web-scraper","scrape","scrape-google","scraper-api","scraping","scraping-api","web-scraping"],"created_at":"2024-11-17T02:09:01.290Z","updated_at":"2025-09-21T11:14:06.276Z","avatar_url":"https://github.com/oxylabs.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# How to Scrape Google Search Results: Python Tutorial\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/how-to-scrape-google-scholar/refs/heads/main/Google-Scraper-API-1090x275.png)](https://oxylabs.io/products/scraper-api/serp/google?utm_source=877\u0026utm_medium=affiliate\u0026groupid=877\u0026utm_content=scrape-google-python-github\u0026transaction_id=102c8d36f7f0d0e5797b8f26152160)\n\n[![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/Pds3gBmKMH)\n\nIn this tutorial, we showcase how to scrape public Google data with Python and Oxylabs [SERP Scraper API](https://oxylabs.io/products/scraper-api/serp) (a part of Web Scraper API), which requires a subscription or **a free trial**.\n\n  * [What is a Google SERP?](#what-is-a-google-serp)\n  * [Is it legal to scrape Google results?](#is-it-legal-to-scrape-google-results)\n  * [Scraping public Google data with Python and Oxylabs Scraper API](#scraping-public-google-data-with-python-and-oxylabs-scraper-api)\n  * [Set up a payload and send a POST request](#set-up-a-payload-and-send-a-post-request)\n    + [Customizing query parameters](#customizing-query-parameters)\n    + [Basic parameters](#basic-parameters)\n  * [Location query parameters](#location-query-parameters)\n    + [Controlling the number of results](#controlling-the-number-of-results)\n    + [Python code for scraping Google search data](#python-code-for-scraping-google-search-data)\n    + [Export scraped data to a CSV](#export-scraped-data-to-a-csv)\n    + [Handling errors and exceptions](#handling-errors-and-exceptions)\n\n## What is a Google SERP? \n\nUpon any discussion of scraping Google search results, you’ll likely run into the “SERP” abbreviation. SERP stands for Search Engine Results Page; it’s the page you get after entering a query into the search bar. SERPs contain various features and elements, such as:\n\n1. Featured snippets\n2. Paid ads\n3. Video carousel\n4. People also ask\n5. Local pack\n6. Related searches\n\n## Is it legal to scrape Google results? \nThe legality of scraping Google search data is largely discussed in the scraping field. As a matter of fact, scraping publicly available data on the internet – including Google SERP data – is legal. However, it may vary from one situation to another, so it’s best to seek legal advice about your specific case. \n\n## Scraping public Google data with Python and Oxylabs Scraper API \n\n1. Install required Python libraries\nTo follow this guide on scraping Google search results, you’ll need the following: \n\n- Credentials for Oxylabs' [SERP Scraper API](https://oxylabs.io/products/scraper-api/serp) – you can get a 7-day free trial by registering on the [dashboard](https://dashboard.oxylabs.io/);\n- Python;\n- Requests library.\n\nFirst, sign up for Oxylabs' Google Search Results API and save your `username` and `password`. \n\nThen, download and install Python 3.8 or above from the [python.org](https://www.python.org/) website. Finally, install the [Request library](https://pypi.org/project/requests/) by using the following command: \n\n`$python3 -m pip install requests`\n\nIf you’re using Windows, choose Python instead of Python3. The rest of the command remains the same: \n\n`d:\\amazon\u003epython -m pip install requests`\n\n## Set up a payload and send a POST request\n\nCreate a new file and enter the following code:\n\n```\nimport requests\nfrom pprint import pprint\n\npayload = {\n    'source': 'google',\n    'url': 'https://www.google.com/search?hl=en\u0026q=newton'  # search for newton\n}\n\nresponse = requests.request(\n    'POST',\n    'https://realtime.oxylabs.io/v1/queries',\n    auth=('USERNAME', 'PASSWORD'),\n    json=payload,\n)\n\npprint(response.json())\n```\n\nHere’s what the result should look like:\n\n```\n{\n    \"results\": [\n        {\n            \"content\": \"\u003c!doctype html\u003e\u003chtml\u003e...\u003c/html\u003e\",\n            \"created_at\": \"YYYY-DD-MM HH:MM:SS\",\n            \"updated_at\": \"YYYY-DD-MM HH:MM:SS\",\n            \"page\": 1,\n            \"url\": \"https://www.google.com/search?hl=en\u0026q=newton\",\n            \"job_id\": \"1234567890123456789\",\n            \"status_code\": 200\n        }\n    ]\n}\n```\n\nNotice how the `url` in the payload dictionary is a Google search results page. In this example, the keyword is `newton`.\n\nAs you can see, the query is executed and the page result in HTML is returned in the content key of the response.\n\n### Customizing query parameters\nLet's review the payload dictionary from the above example for scraping Google search data.\n\n```\npayload = {\n    'source': 'google',\n    'url': 'https://www.google.com/search?hl=en\u0026q=newton'\n}\n```\n\nThe dictionary keys are parameters used to inform Google Scraper API about required customization. \n\nThe first parameter is the `source`, which is really important because it sets the scraper we’re going to use. \n\nThe default value is `Google` – when you use it, you can set the url as any Google search page, and all the other parameters will be extracted from the URL. \n\nAlthough in this guide we’ll be using the `google_search` parameter, there's many others: `google_ads`, `google_hotels`, `google_lens`, `google_suggest`, and more (full list [here](https://developers.oxylabs.io/scraper-apis/web-scraper-api/google).\n \nKeep in mind that if you set the source as `google_search`, you cannot use the `url` parameter. Luckily, you can use several different parameters for acquiring public Google SERP data without having to create multiple URLs (more on that in the next paragraph.) \n\n### Basic parameters\nWe’ll build the payload by adding the parameters one by one. First, begin with setting the source as `google_search`.\n\n```\npayload = {\n    'source': 'google_search',\n}\n```\n\nNow, let’s add `query` – a crucial parameter that determines what search results you’ll be retrieving. In our example, we’ll use `newton` as our search query. At this stage, the payload dictionary looks like this:\n\n```\npayload = {\n    'source': 'google_search',\n    'query': 'newton',\n}\n```\n\nThat said, `google_search` and query are the two essential parameters for scraping public Google search data. If you want the API to return Google search results at this stage, you can use `payload`. Now, let’s move to the next parameter. \n\n## Location query parameters\nYou can work with a domain parameter if you want to use a localized domain – for example, `'domain':'de'` will fetch results from google.de. If you want to see the results from Germany, use the `geo_location` parameter— `'geo_location':'Germany'`. See the [documentation](https://developers.oxylabs.io/scraper-apis/web-scraper-api/features/geo-location#google) for the `geo_location` parameter to learn more about the correct values.\n\nAlso, here’s what changing the locale parameter looks like: \n\n```\npayload = {\n'source':'google_search',\n'query':'newton',\n'domain':'de' ,\n'geo_location': 'Germany',\n'locale' : 'en-us'\n}\n```\n\nTo learn more about the potential values of the locale parameter, check the [documentation](https://developers.oxylabs.io/scraper-apis/web-scraper-api/features/domain-locale-results-language#locale-1), as well.  \n\nIf you send the above payload, you’ll receive search results in American English from google.de, just like anyone physically located in Germany would.\n\n### Controlling the number of results\n\nBy default, you’ll see the first ten results from the first page. If you want to customize this, you can use these parameters: `start_page`, `pages`, and `limit`.\n\nThe `start_page` parameter determines which page of search results to return. The `pages` parameter specifies the number of pages. Finally, the `limit parameter` sets the number of results on each page.\n\nFor example, the following set of parameters fetch results from pages 11 and 12 of the search engine results, with 20 results on each page:\n\n```\npayload = {\n    'start_page': 11,\n    'pages': 2,\n    'limit': 20,\n    ...  # other parameters\n}\n```\n\nApart from the search parameters we’ve covered so far, there are a few more you can use to fine-tune your results – see our [documentation](https://developers.oxylabs.io/scraper-apis/web-scraper-api/google/search#request-parameter-values) on collecting public Google Search data. \n\n### Python code for scraping Google search data\n\nNow, let’s put together everything we’ve learned so far – here’s what the final script with the shoes keyword looks like:\n\n```\nimport requests\nfrom pprint import pprint\npayload = {\n    'source': 'google_search',\n    'query': 'shoes',\n    'domain': 'de',\n    'geo_location': 'Germany',\n    'locale': 'en-us',\n    'parse': True,\n    'start_page': 1,\n    'pages': 5,\n    'limit': 10,\n}\n\n\n# Get response.\nresponse = requests.request(\n    'POST',\n    'https://realtime.oxylabs.io/v1/queries',\n    auth=('USERNAME', 'PASSWORD'),\n    json=payload,\n)\n\n\nif response.status_code != 200:\n    print(\"Error - \", response.json())\n    exit(-1)\n\n\npprint(response.json())\n```\n\n### Export scraped data to a CSV\nOne of the best Google Scraper API features is the ability to parse an HTML page into JSON. For that, you don't need to use BeautifulSoup or any other library – just send the parse parameter as True. \n\nHere is a sample payload:\n\n```\npayload = {\n    'source': 'google_search',\n    'query': 'adidas',\n    'parse': True,\n}\n```\n\nWhen sent to the Google Scraper API, this payload will return the results in JSON. To see a detailed JSON data structure, see our [documentation](https://developers.oxylabs.io/scraper-apis/web-scraper-api/google/search#structured-data). \n\nThe key highlights: \n\n- The results are in the dedicated results list. Here, each page gets a new entry.\n- Each result contains the content in a dictionary key named content. \n- The actual results are in the results key.\n\nNote that there’s a `job_id` in the results.\n\nThe easiest way to save the data is by using the Pandas library, since it can normalize JSON quite effectively.\n\n```\nimport pandas as pd\n...\ndata = response.json()\ndf = pd.json_normalize(data['results'])\ndf.to_csv('export.csv', index=False)\n```\n\nAlternatively, you can also take note of the `job_id` and send a GET request to the following URL, along with your credentials.\n\n```\nhttp://data.oxylabs.io/v1/queries/{job_id}/results/normalized?format=csv\n```\n\nIn case you need multiple data formats, you can utilize the [multi-format output](https://developers.oxylabs.io/scraping-solutions/web-scraper-api/features/result-processing-and-storage/output-types/multi-format-output) feature to get all result types in a single API response. For example, the following will fetch parsed data, a screenshot of the page, and the raw HTML document:\n\n```\nhttps://data.oxylabs.io/v1/queries/{job_id}/results?type=parsed,png,raw\n```\n\n\n\n### Handling errors and exceptions\nWhen scraping Google, you can run into several challenges: network issues,  invalid query parameters, or API quota limitations.\n\nTo handle these, you can use try-except blocks in your code. For example, if an error occurs when sending the API request, you can catch the exception and print an error message:\n\n```\ntry:\n    response = requests.request(\n        'POST',\n        'https://realtime.oxylabs.io/v1/queries',\n        auth=('USERNAME', 'PASSWORD'),\n        json=payload,\n    )\nexcept requests.exceptions.RequestException as e:\n    print(\"Error:\", e)\n```\n\nIf you send an invalid parameter, Google Scraper API will return the 400 response code.\n\nTo catch these errors, check the status code: \n\n```\nif response.status_code != 200:\n    print(\"Error - \", response.json())\n```\n\nLooking to scrape data from other Google sources? [Google Sheets for Basic Web Scraping](https://github.com/oxylabs/web-scraping-google-sheets), [How to Scrape Google Shopping Results](https://github.com/oxylabs/scrape-google-shopping), [Google Play Scraper](https://github.com/oxylabs/google-play-scraper), [How To Scrape Google Jobs](https://github.com/oxylabs/how-to-scrape-google-jobs), [Google News Scrpaer](https://github.com/oxylabs/google-news-scraper), [How to Scrape Google Scholar](https://github.com/oxylabs/how-to-scrape-google-scholar), [How to Scrape Google Flights with Python](https://github.com/oxylabs/how-to-scrape-google-flights),  [Scrape Google Search Results](https://github.com/oxylabs/scrape-google-python), [Scrape Google Trends](https://github.com/oxylabs/how-to-scrape-google-trends)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fscrape-google-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fscrape-google-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fscrape-google-python/lists"}