{"id":20709920,"url":"https://github.com/oxylabs/how-to-scrape-google-jobs","last_synced_at":"2025-10-24T07:07:55.106Z","repository":{"id":221656790,"uuid":"754551377","full_name":"oxylabs/how-to-scrape-google-jobs","owner":"oxylabs","description":"Learn how to build your own Google Jobs scraper that simultaneously scrapes Google Jobs for multiple search queries and geo-locations with Python and Oxylabs’ Google Jobs Scraper API. https://oxylabs.io/blog/how-to-scrape-google-jobs","archived":false,"fork":false,"pushed_at":"2025-04-09T08:39:03.000Z","size":128,"stargazers_count":231,"open_issues_count":0,"forks_count":4,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-13T04:59:08.091Z","etag":null,"topics":["google-scraping","google-search-api","google-search-scraper","python","scraper","scraper-api","scraper-python","serp-scraper-api","serp-scraping","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-08T09:51:01.000Z","updated_at":"2025-04-10T14:00:18.000Z","dependencies_parsed_at":"2024-02-09T09:45:35.843Z","dependency_job_id":"8bd7eed6-7515-47fa-a724-b7e8b2b27881","html_url":"https://github.com/oxylabs/how-to-scrape-google-jobs","commit_stats":null,"previous_names":["oxylabs/how-to-scrape-google-jobs"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-google-jobs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-google-jobs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-google-jobs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-google-jobs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/how-to-scrape-google-jobs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248665758,"owners_count":21142123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google-scraping","google-search-api","google-search-scraper","python","scraper","scraper-api","scraper-python","serp-scraper-api","serp-scraping","web-scraping"],"created_at":"2024-11-17T02:09:01.368Z","updated_at":"2025-10-24T07:07:55.093Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# How To Scrape Google Jobs\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/how-to-scrape-google-scholar/refs/heads/main/Google-Scraper-API-1090x275.png)](https://oxylabs.io/products/scraper-api/serp/google?utm_source=877\u0026utm_medium=affiliate\u0026groupid=877\u0026utm_content=how-to-scrape-google-jobs-github\u0026transaction_id=102c8d36f7f0d0e5797b8f26152160)\n\n[![](https://dcbadge.limes.pink/api/server/Pds3gBmKMH?style=for-the-badge\u0026theme=discord)](https://discord.gg/Pds3gBmKMH) [![YouTube](https://img.shields.io/badge/YouTube-Oxylabs-red?style=for-the-badge\u0026logo=youtube\u0026logoColor=white)](https://www.youtube.com/@oxylabs)\n\nIn this two-part tutorial, we're going to show how to scrape Google Jobs data. First, we'll show how to do it for free but if you need data at scale, please refer to the second part of the tutorial. There, we'll demonstrate how to gather large-scale data with Oxylabs API. \n\n  * [Free Google Jobs Scraper](#free-google-jobs-scraper)\n    + [Prerequisites](#prerequisites)\n    + [Installation](#installation)\n    + [Scraping Google Jobs](#scraping-google-jobs)\n    + [Notes](#notes)\n  * [Scraping Google Jobs with Oxylabs API](#scraping-google-jobs-with-oxylabs-api)\n  * [Google Jobs website overview](#google-jobs-website-overview)\n  * [1. Get a free trial and send a request](#1-get-a-free-trial-and-send-a-request)\n    + [Install Python](#install-python)\n    + [Send a request for testing](#send-a-request-for-testing)\n  * [2. Install and import libraries](#2-install-and-import-libraries)\n  * [3. Add your API user credentials](#3-add-your-api-user-credentials)\n  * [4. Set up queries and locations](#4-set-up-queries-and-locations)\n    + [URL parameters](#url-parameters)\n    + [Locations](#locations)\n  * [5. Prepare the API payload with parsing instructions](#5-prepare-the-api-payload-with-parsing-instructions)\n    + [Job title](#job-title)\n    + [Company name](#company-name)\n    + [Location](#location)\n    + [Date](#date)\n    + [Salary](#salary)\n    + [Job posted via](#job-posted-via)\n    + [URL](#url)\n  * [6. Define functions](#6-define-functions)\n    + [Submit job](#submit-job)\n    + [Check job status](#check-job-status)\n    + [Get job results](#get-job-results)\n    + [Save data to a CSV file](#save-data-to-a-csv-file)\n    + [Scrape Google Jobs](#scrape-google-jobs)\n  * [7. Create the main() function](#7-create-the-main-function)\n  * [8. Run the complete code](#8-run-the-complete-code)\n\n## Free Google Jobs Scraper\n\nA free tool used to get data about jobs from Google Jobs for a provided search query.\n\n### Prerequisites\n\nTo run this tool, you need to have Python 3.11 installed in your system.\n\n### Installation\n\nOpen up a terminal window, navigate to this repository and run this command:\n\n```make install```\n\n### Scraping Google Jobs\n\nTo scrape jobs from Google Jobs, simply run this command in your terminal with a search query for a job that you need data for:\n\n```make scrape QUERY=\"\u003cjob_query\u003e\"```\n\nFor this example, let's try scraping for designer jobs. The command should look something like this:\n\n```make scrape QUERY=\"designer\"```\n\nMake sure to enclose your query in quotation marks, otherwise the tool might have trouble parsing it.\n\nAfter running the command, your terminal should look something like this:\n\n\u003cimg width=\"701\" alt=\"image\" src=\"https://github.com/user-attachments/assets/7958807d-c30b-44fc-bb0a-cd3140ceb7d8\"\u003e\n\nAfter the tool has finished running, you should see a file named `jobs.csv` in your current directory.\n\nThe CSV file contains available jobs in your area for the position you entered in the query. \nThe jobs are listed with these attributes:\n\n- `title` - The title of position.\n- `company` - The name of the company.\n- `location` - The location of the position.\n- `url` - The URL of the Google Jobs page for that job.\n\nHere's an example of how the data can look like:\n\n\u003cimg width=\"854\" alt=\"image\" src=\"https://github.com/user-attachments/assets/cfdb3e6e-c57f-46ba-943c-6f58c0e8e9c2\"\u003e\n\n### Notes\n\nIn case the code doesn't work or your project is of bigger scale, please refer to the second part of the tutorial. There, we showcase how to scrape public data with Oxylabs API.\n\n## Scraping Google Jobs with Oxylabs API\n\n## Google Jobs website overview\n\nOnce you visit the Google Jobs page, you'll see that all job listings for a query are displayed on the left side. Looking at the HTML structure, you can see that each listing is enclosed in the ```\u003cli\u003e``` tag and collectively wrapped within the ```\u003cul\u003e``` tag:\n\nIn this guide, let’s scrape Google Jobs results asynchronously and extract the following publicly available data:\n\n1. Job title\n2. Company name\n3. Job location\n4. Job posted via *[platform]*\n5. Job listing date\n6. Salary\n\nIf you want to extract even more public data, such as job highlights, job description, and similar jobs, expand the code shown in this article to make additional API calls to the scraped job URLs.\n\n### 1. Get a free trial and send a request\nVisit the Oxylabs dashboard and create an account to claim your 1-week free trial for Google Jobs API. It’s equipped with proxy servers, Headless Browser, Custom Parser, and other advanced features that’ll help you overcome blocks and fingerprinting. See this short guide that shows how to navigate the dashboard and get the free trial.\n\n### Install Python\nIf you don’t have Python installed yet, you can download it from the official Python website. This tutorial is written with Python 3.12.0, so ensure that you have a compatible version.\n\n### Send a request for testing\nAfter creating an API user, copy and save your API user credentials, which you’ll use for authentication. Next, open your terminal and install the requests library:\n\n```bash\npip install requests\n```\n\nThen run the following code that scrapes Google Jobs results and retrieves the entire HTML file:\n\n```python\nimport requests\n\npayload = {\n    \"source\": \"google\",\n    \"url\": \"https://www.google.com/search?q=developer\u0026ibp=htl;jobs\u0026hl=en\u0026gl=us\",\n    \"render\": \"html\"\n}\n\nresponse = requests.post(\n    \"https://realtime.oxylabs.io/v1/queries\",\n    auth=(\"USERNAME\", \"PASSWORD\"),\t# Replace with your API user credentials\n    json=payload\n)\nprint(response.json())\nprint(response.status_code)\n```\n\nOnce it finishes running, you should see a JSON response with HTML results and a status code of your request. If everything works correctly, the status code should be ```200```.\n\n## 2. Install and import libraries\n\nFor this project, let’s use the ```asyncio``` and ```aiohttp``` libraries to make asynchronous requests to the API. Additionally, the ```json``` and ```pandas``` libraries will help you deal with JSON and CSV files. \n\nOpen your terminal and run the following command to install the necessary libraries:\n\n```bash\npip install asyncio aiohttp pandas\n```\n\nThen, import them into your Python file:\n\n```python\nimport asyncio, aiohttp, json, pandas as pd\nfrom aiohttp import ClientSession, BasicAuth\n```\n\n## 3. Add your API user credentials\n\nCreate the API user ```credentials``` variable and use ```BasicAuth```, as ```aiohttp``` requires this for authentication:\n\n```python\ncredentials = BasicAuth(\"USERNAME\", \"PASSWORD\") # Replace with your API user credentials\n```\n\n## 4. Set up queries and locations\n\nYou can easily form Google Jobs URLs for different queries by manipulating the q= parameter:\n\n```\nhttps://www.google.com/search?q=developer\u0026ibp=htl;jobs\u0026hl=en\u0026gl=us\n```\n\nThis enables you to scrape job listings for as many search queries as you want. \n\n**Note** that the ```q=```, ```ibp=htl;jobs```, ```hl=```, and ```gl=``` parameters are mandatory for the URL to work.\n\nAdditionally, you could set the UULE parameter for geo-location targeting yourself, but that’s unnecessary since the ```geo_location``` parameter of Google Jobs Scraper API does that by default.\n\n### URL parameters\n\nCreate the URL_parameters list to store your search queries:\n\n```python\nURL_parameters = [\"developer\", \"chef\", \"manager\"]\n``` \n\n### Locations\n\nThen, create the ```locations``` dictionary where the key refers to the country, and the value is a list of geo-location parameters. This dictionary will be used to dynamically form the API payload and localize Google Jobs results for the specified location. The two-letter country code will be used to modify the ```gl=``` parameter in the Google Jobs URL:\n\n```python\nlocations = {\n    \"US\": [\"California,United States\", \"Virginia,United States\", \"New York,United States\"],\n    \"GB\": [\"United Kingdom\"],\n    \"DE\": [\"Germany\"]\n}\n```\n\nVisit our [documentation](https://developers.oxylabs.io/scraper-apis/web-scraper-api/features/geo-location#google) for more details about geo-locations.\n\n## 5. Prepare the API payload with parsing instructions\n\nGoogle Jobs Scraper API takes web scraping instructions from a ```payload``` dictionary, making it the most important configuration to fine-tune. The ```url``` and ```geo_location``` keys are set to ```None```, as the scraper will pass these values dynamically for each search query and location. The ```\"render\": \"html\"``` parameter enables JavaScript rendering and returns the rendered HTML file:\n\n```python\npayload = {\n    \"source\": \"google\",\n    \"url\": None,\n    \"geo_location\": None,\n    \"user_agent_type\": \"desktop\",\n    \"render\": \"html\"\n}\n```\n\nNext, use [Custom Parser](https://developers.oxylabs.io/scraper-apis/web-scraper-api/features/custom-parser) to define your own parsing logic with ```xPath``` or ```CSS``` selectors and retrieve only the data you need. Remember that you can create as many functions as you want and extract even more data points than shown in this guide. Head to this Google Jobs URL in your browser and open Developer Tools by pressing Ctrl+Shift+I (Windows) or Option + Command + I (macOS). Use Ctrl+F or Command+F to open a search bar and test selector expressions.\n\nAs mentioned previously, the job listings are within the \u003cli\u003e tags, which are wrapped with the \u003cul\u003e tag. \n\nAs there is more than one ```\u003cul\u003e``` list on the Google Jobs page, you can form an xPath selector by specifying the ```div``` element that contains the targeted list.\n\n```//div[@class='nJXhWc']//ul/li```\n\nYou can use this selector to specify the location of all job listings in the HTML file. In the ```payload``` dictionary, set the ```parse``` key to ```True``` and create the ```parsing_instructions``` parameter with the jobs function:\n\n```python\npayload = {\n    \"source\": \"google\",\n    \"url\": None,\n    \"geo_location\": None,\n    \"user_agent_type\": \"desktop\",\n    \"render\": \"html\",\n    \"parse\": True,\n    \"parsing_instructions\": {\n        \"jobs\": {\n            \"_fns\": [\n                {\n                    \"_fn\": \"xpath\",\n                    \"_args\": [\"//div[@class='nJXhWc']//ul/li\"]\n                }\n            ],\n        }\n    }\n}\n```\n\nNext, create the ```_items``` iterator that will loop over the jobs list and extract details for each listing:\n\n```python\npayload = {\n    \"source\": \"google\",\n    \"url\": None,\n    \"geo_location\": None,\n    \"user_agent_type\": \"desktop\",\n    \"render\": \"html\",\n    \"parse\": True,\n    \"parsing_instructions\": {\n        \"jobs\": {\n            \"_fns\": [\n                {\n                    \"_fn\": \"xpath\", # You can use CSS or xPath\n                    \"_args\": [\"//div[@class='nJXhWc']//ul/li\"]\n                }\n            ],\n            \"_items\": {\n                \"data_point_1\": {\n                    \"_fns\": [\n                        {\n                            \"_fn\": \"selector_type\",  # You can use CSS or xPath\n                            \"_args\": [\"selector\"]\n                        }\n                    ]\n                },\n                \"data_point_2\": {\n                    \"_fns\": [\n                        {\n                            \"_fn\": \"selector_type\",\n                            \"_args\": [\"selector\"]\n                        }\n                    ]\n                },\n            }\n        }\n    }\n}\n```\n\nFor each data point, you can create a separate function within the ```_items``` iterator. Let’s see how ```xPath``` selectors should look like for each Google Jobs data point:\n\n### Job title\n\n```\n.//div[@class='BjJfJf PUpOsf']/text()\n```\n\n### Company name\n\n```\n.//div[@class='vNEEBe']/text()\n```\n\n### Location\n\n```\n.//div[@class='Qk80Jf'][1]/text()\n```\n\n### Date\n\n\n```\n.//div[@class='PuiEXc']//span[@class='LL4CDc' and contains(@aria-label, 'Posted')]/span/text()\n```\n\n### Salary\n\n```\n.//div[@class='PuiEXc']//div[@class='I2Cbhb bSuYSc']//span[@aria-hidden='true']/text()\n```\n\n### Job posted via\n\n```\n.//div[@class='Qk80Jf'][2]/text()\n```\n\n### URL\n\n```\n.//div[@data-share-url]/@data-share-url\n```\n\n**Please be aware** that you can only access this job listing URL in your browser with an IP address from the same country used during web scraping. If you’ve used a United States proxy, make sure to use a US IP address in your browser.\n\nIn the end, you should have a ```payload``` that looks like shown below. Save it to a separate JSON file and ensure that the ```None``` and ```True``` parameter values are converted to respective JSON values: ```null``` and ```true```:\n\n```python\nimport json\n\npayload = {\n    \"source\": \"google\",\n    \"url\": None,\n    \"geo_location\": None,\n    \"user_agent_type\": \"desktop\",\n    \"render\": \"html\",\n    \"parse\": True,\n    \"parsing_instructions\": {\n        \"jobs\": {\n            \"_fns\": [\n                {\n                    \"_fn\": \"xpath\",\n                    \"_args\": [\"//div[@class='nJXhWc']//ul/li\"]\n                }\n            ],\n            \"_items\": {\n                \"job_title\": {\n                    \"_fns\": [\n                        {\n                            \"_fn\": \"xpath_one\",\n                            \"_args\": [\".//div[@class='BjJfJf PUpOsf']/text()\"]\n                        }\n                    ]\n                },\n                \"company_name\": {\n                    \"_fns\": [\n                        {\n                            \"_fn\": \"xpath_one\",\n                            \"_args\": [\".//div[@class='vNEEBe']/text()\"]\n                        }\n                    ]\n                },\n                \"location\": {\n                    \"_fns\": [\n                        {\n                            \"_fn\": \"xpath_one\",\n                            \"_args\": [\".//div[@class='Qk80Jf'][1]/text()\"]\n                        }\n                    ]\n                },\n                \"date\": {\n                    \"_fns\": [\n                        {\n                            \"_fn\": \"xpath_one\",\n                            \"_args\": [\".//div[@class='PuiEXc']//span[@class='LL4CDc' and contains(@aria-label, 'Posted')]/span/text()\"]\n                        }\n                    ]\n                },\n                \"salary\": {\n                    \"_fns\": [\n                        {\n                            \"_fn\": \"xpath_one\",\n                            \"_args\": [\".//div[@class='PuiEXc']//div[@class='I2Cbhb bSuYSc']//span[@aria-hidden='true']/text()\"]\n                        }\n                    ]\n                },\n                \"posted_via\": {\n                    \"_fns\": [\n                        {\n                            \"_fn\": \"xpath_one\",\n                            \"_args\": [\".//div[@class='Qk80Jf'][2]/text()\"]\n                        }\n                    ]\n                },\n                \"URL\": {\n                    \"_fns\": [\n                        {\n                            \"_fn\": \"xpath_one\",\n                            \"_args\": [\".//div[@data-share-url]/@data-share-url\"]\n                        }\n                    ]\n                }\n            }\n        }\n    }\n}\n\nwith open(\"payload.json\", \"w\") as f:\n    json.dump(payload, f, indent=4)\n```\n\nThis allows you to import the payload and make the scraper code much shorter:\n\n```python\npayload = {}\nwith open(\"payload.json\", \"r\") as f:\n    payload = json.load(f)\n```\n\n## 6. Define functions \n\nThere are several ways you can [integrate](https://developers.oxylabs.io/scraper-apis/web-scraper-api/integration-methods) Oxylabs API, namely Realtime, [Push-Pull](https://developers.oxylabs.io/scraper-apis/web-scraper-api/integration-methods/push-pull) (supports [batch requests](https://developers.oxylabs.io/scraper-apis/web-scraper-api/integration-methods/push-pull-batch)), and Proxy endpoint. For this guide, let’s use Push-Pull with batch requests, as you won’t have to keep your connection open after submitting a scraping job to the API. The API endpoint to use in this scenario is https://data.oxylabs.io/v1/queries.\n\nYou could also use another endpoint to submit batches of up to 5000 URLs or queries. Keep in mind that making this choice will require you to modify the code shown in this tutorial. Read up about batch queries in our documentation.\n\n### Submit job\nDefine an ```async``` function called ```submit_job``` and pass the ```session: ClientSession``` together with the ```payload``` to submit a web scraping job to the Oxylabs API using the ```POST``` method. This will return the ID number of the submitted job:\n\n```python\nasync def submit_job(session: ClientSession, payload):\n    async with session.post(\n        \"https://data.oxylabs.io/v1/queries\",\n        auth=credentials,\n        json=payload\n    ) as response:\n        return (await response.json())[\"id\"]\n```\n\n### Check job status\n\nThen, create another ```async``` function that passes the ```job_id``` (this will be defined later) and returns the ```status``` of the scraping job from the response:\n\n```python\nasync def check_job_status(session: ClientSession, job_id):\n    async with session.get(f\"https://data.oxylabs.io/v1/queries/{job_id}\", auth=credentials) as response:\n        return (await response.json())[\"status\"]\n```\n\n### Get job results\nNext, create an ```async``` function that retrieves the scraped and parsed jobs results. Note that the response is a JSON string that contains the API job details and the scraped content that you can access by parsing nested JSON properties:\n\n```python\nasync def get_job_results(session: ClientSession, job_id):\n    async with session.get(f\"https://data.oxylabs.io/v1/queries/{job_id}/results\", auth=credentials) as response:\n        return (await response.json())[\"results\"][0][\"content\"][\"jobs\"]\n```\n### Save data to a CSV file\n\nDefine another ```async``` function that saves the scraped and parsed data to a CSV file. Later on, we’ll create the four parameters that are passed to the function. As the ```pandas``` library is synchronous, you must use ```asyncio.to_thread()``` to run the ```df.to_csv``` asynchronously in a separate thread:\n\n```python\nasync def save_to_csv(job_id, query, location, results):\n    print(f\"Saving data for {job_id}\")\n    data = []\n    for job in results:\n        data.append({\n            \"Job title\": job[\"job_title\"],\n            \"Company name\": job[\"company_name\"],\n            \"Location\": job[\"location\"],\n            \"Date\": job[\"date\"],\n            \"Salary\": job[\"salary\"],\n            \"Posted via\": job[\"posted_via\"],\n            \"URL\": job[\"URL\"]\n        })\n\n    df = pd.DataFrame(data)\n    filename = f\"{query}_jobs_{location.replace(',', '_').replace(' ', '_')}.csv\"\n    await asyncio.to_thread(df.to_csv, filename, index=False)\n```\n\n### Scrape Google Jobs\n\nMake another ```async``` function that passes parameters to form the Google Jobs ```URL``` and the payload dynamically. Create a variable ```job_id``` and then call the ```submit_job``` function to submit the request to the API and create a ```while True``` loop by calling the ```check_job_status``` function to keep checking whether the API has finished web scraping. At the end, initiate the ```get_job_results``` and ```save_to_csv``` functions:\n\n```python\nasync def scrape_jobs(session: ClientSession, query, country_code, location):\n    URL = f\"https://www.google.com/search?q={query}\u0026ibp=htl;jobs\u0026hl=en\u0026gl={country_code}\"\n\n    payload[\"url\"] = URL\n    payload[\"geo_location\"] = location\n\n    job_id = await submit_job(session, payload)\n\n    await asyncio.sleep(15)\n\n    print(f\"Checking status for {job_id}\")\n\n    while True:\n        status = await check_job_status(session, job_id)\n        if status == \"done\":\n            print(f\"Job {job_id} done. Retrieving {query} jobs in {location}.\")\n            break\n        elif status == \"failed\":\n            print(f\"Job {job_id} encountered an issue. Status: {status}\")\n            return\n        \n        await asyncio.sleep(5)\n\n    results = await get_job_results(session, job_id)\n    await save_to_csv(job_id, query, location, results)\n```\n\n## 7. Create the main() function\n\nYou’ve written most of the code, what’s left is to pull everything together by defining an ```async``` function called ```main()``` that creates an ```aiohttp session```. It makes a list of tasks to scrape jobs for each combination of ```location``` and ```query``` and executes each task concurrently using ```asyncio.gather()```:\n\n```python\nasync def main():\n    async with aiohttp.ClientSession() as session:\n        tasks = []\n\n        for country_code, location_list in locations.items():\n            for location in location_list:\n                for query in URL_parameters:\n                    task = asyncio.ensure_future(scrape_jobs(session, query, country_code, location))\n                    tasks.append(task)\n\n        await asyncio.gather(*tasks)\n```\n\nIf you run into an SSL certificate verification failure, you can bypass SSL by using the ```TCPConnector```:\n\n```python\nasync def main():\n    async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(ssl=False)) as session:\n\t# Remaining code...\n```\n\nLastly, initialize the event loop and call the ```main()``` function:\n\n```python\nif __name__ == \"__main__\":\n    loop = asyncio.new_event_loop()\n    asyncio.set_event_loop(loop)\n    loop.run_until_complete(main())\n    print(\"Completed!\")\n```\n## 8. Run the complete code\nHere’s the full Python code that scrapes Google Jobs listings for each query and location asynchronously:\n\n```python\nimport asyncio, aiohttp, json, pandas as pd\nfrom aiohttp import ClientSession, BasicAuth\n\n\ncredentials = BasicAuth(\"USERNAME\", \"PASSWORD\") # Replace with your API user credentials\n\nURL_parameters = [\"developer\", \"chef\", \"manager\"]\n\nlocations = {\n    \"US\": [\"California,United States\", \"Virginia,United States\", \"New York,United States\"],\n    \"GB\": [\"United Kingdom\"],\n    \"DE\": [\"Germany\"]\n}\n\npayload = {}\nwith open(\"payload.json\", \"r\") as f:\n    payload = json.load(f)\n\nasync def submit_job(session: ClientSession, payload):\n    async with session.post(\n        \"https://data.oxylabs.io/v1/queries\",\n        auth=credentials,\n        json=payload\n    ) as response:\n        return (await response.json())[\"id\"]\n\n\nasync def check_job_status(session: ClientSession, job_id):\n    async with session.get(f\"https://data.oxylabs.io/v1/queries/{job_id}\", auth=credentials) as response:\n        return (await response.json())[\"status\"]\n\n\nasync def get_job_results(session: ClientSession, job_id):\n    async with session.get(f\"https://data.oxylabs.io/v1/queries/{job_id}/results\", auth=credentials) as response:\n        return (await response.json())[\"results\"][0][\"content\"][\"jobs\"]\n\n\nasync def save_to_csv(job_id, query, location, results):\n    print(f\"Saving data for {job_id}\")\n    data = []\n    for job in results:\n        data.append({\n            \"Job title\": job[\"job_title\"],\n            \"Company name\": job[\"company_name\"],\n            \"Location\": job[\"location\"],\n            \"Date\": job[\"date\"],\n            \"Salary\": job[\"salary\"],\n            \"Posted via\": job[\"posted_via\"],\n            \"URL\": job[\"URL\"]\n        })\n\n    df = pd.DataFrame(data)\n    filename = f\"{query}_jobs_{location.replace(',', '_').replace(' ', '_')}.csv\"\n    await asyncio.to_thread(df.to_csv, filename, index=False)\n\n\nasync def scrape_jobs(session: ClientSession, query, country_code, location):\n    URL = f\"https://www.google.com/search?q={query}\u0026ibp=htl;jobs\u0026hl=en\u0026gl={country_code}\"\n\n    payload[\"url\"] = URL\n    payload[\"geo_location\"] = location\n\n    job_id = await submit_job(session, payload)\n\n    await asyncio.sleep(15)\n\n    print(f\"Checking status for {job_id}\")\n\n    while True:\n        status = await check_job_status(session, job_id)\n        if status == \"done\":\n            print(f\"Job {job_id} done. Retrieving {query} jobs in {location}.\")\n            break\n        elif status == \"failed\":\n            print(f\"Job {job_id} encountered an issue. Status: {status}\")\n            return\n        \n        await asyncio.sleep(5)\n\n    results = await get_job_results(session, job_id)\n    await save_to_csv(job_id, query, location, results)\n\n\nasync def main():\n    async with aiohttp.ClientSession() as session:\n        tasks = []\n\n        for country_code, location_list in locations.items():\n            for location in location_list:\n                for query in URL_parameters:\n                    task = asyncio.ensure_future(scrape_jobs(session, query, country_code, location))\n                    tasks.append(task)\n\n        await asyncio.gather(*tasks)\n\n\nif __name__ == \"__main__\":\n    loop = asyncio.new_event_loop()\n    asyncio.set_event_loop(loop)\n    loop.run_until_complete(main())\n    print(\"Completed!\")\n```\n\nAfter the scraper finishes running, you’ll see all the CSV files saved in your local directory.\n\nRefer to this blog [blog post](https://oxylabs.io/blog/how-to-scrape-google-jobs) for a full guide featuring detailed insights and images.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fhow-to-scrape-google-jobs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fhow-to-scrape-google-jobs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fhow-to-scrape-google-jobs/lists"}