{"id":20709915,"url":"https://github.com/oxylabs/how-to-scrape-google-flights","last_synced_at":"2025-05-15T11:08:59.847Z","repository":{"id":234832027,"uuid":"784261097","full_name":"oxylabs/how-to-scrape-google-flights","owner":"oxylabs","description":"Scrapes flight data from Google Flights and analyzes flight prices with Python and Oxylabs’ Google Flights API.","archived":false,"fork":false,"pushed_at":"2025-02-10T12:44:28.000Z","size":72,"stargazers_count":626,"open_issues_count":2,"forks_count":3,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-08T06:34:33.222Z","etag":null,"topics":["google-flight-api","google-flights-api","google-flights-api-python","google-flights-python","google-flights-scraper","how-to-scrape-google-flights","scrape-google-flights"],"latest_commit_sha":null,"homepage":"https://oxylabs.io/blog/how-to-scrape-google-flights","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-09T13:59:36.000Z","updated_at":"2025-04-07T12:06:02.000Z","dependencies_parsed_at":"2024-08-05T13:19:41.120Z","dependency_job_id":"8a6a8a6d-8547-4303-94d1-4c5e42252044","html_url":"https://github.com/oxylabs/how-to-scrape-google-flights","commit_stats":null,"previous_names":["oxylabs/how-to-scrape-google-flights"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-google-flights","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-google-flights/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-google-flights/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-google-flights/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/how-to-scrape-google-flights/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254328386,"owners_count":22052632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google-flight-api","google-flights-api","google-flights-api-python","google-flights-python","google-flights-scraper","how-to-scrape-google-flights","scrape-google-flights"],"created_at":"2024-11-17T02:09:00.279Z","updated_at":"2025-05-15T11:08:59.828Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# How to Scrape Google Flights with Python\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/how-to-scrape-google-scholar/refs/heads/main/Google-Scraper-API-1090x275.png)](https://oxylabs.go2cloud.org/aff_c?offer_id=7\u0026aff_id=877\u0026url_id=112)\n\n[![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/GbxmdGhZjq)\n\n  * [Free Google Flights Scraper](#free-google-flights-scraper)\n    + [Prerequisites](#prerequisites)\n    + [Installation](#installation)\n    + [Getting the URL for a Google Flights page](#getting-the-url-for-a-google-flights-page)\n    + [Scraping Google Flights](#scraping-google-flights)\n    + [Notes](#notes)\n  * [Scrape public Google Flights data with Oxylabs API](#scrape-public-google-flights-data-with-oxylabs-api)\n    + [Installing prerequisite libraries](#installing-prerequisite-libraries)\n    + [Creating core structure](#creating-core-structure)\n    + [Getting the price](#getting-the-price)\n     + [Getting the flight time](#getting-the-flight-time)\n     + [Getting the airline name](#getting-the-airline-name)\n     + [Final result](#final-result)\n\n\n## Free Google Flights Scraper\n\nA free tool used to get data from Google Flights for a provided Google Flights URL.\n\n### Prerequisites\n\nTo run this tool, you need to have Python 3.11 installed in your system.\n\n### Installation\n\nOpen up a terminal window, navigate to this repository and run this command:\n\n```make install```\n\n### Getting the URL for a Google Flights page\n\nFirst of all, go to [Google Flights](https://www.google.com/travel/flights) in your browser and select your desired departure and arrival locations along with the dates.\n\nFor this example, we'll be scraping flights from Berlin to Paris, two weeks apart. \n\nAfter clicking **Search**, you should see something like this.\n\n\u003cimg width=\"1094\" alt=\"image\" src=\"https://github.com/user-attachments/assets/b7ac594c-e1c0-47e7-9c52-66ae06619f24\"\u003e\n\nCopy and save the URL from your browser, it will be used for scraping the data for this flight configuration.\n\nHere's the URL for this example\n```https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI0LTA5LTA0agwIAxIIL20vMDE1NnFyDAgDEggvbS8wNXF0ahooEgoyMDI0LTA5LTE4agwIAxIIL20vMDVxdGpyDAgDEggvbS8wMTU2cUABSAFwAYIBCwj___________8BmAEB```\n\n### Scraping Google Flights\n\nTo get results from Google Flights, simply run this command in your terminal:\n\n```make scrape URL=\"\u003cyour_google_flights_url\u003e\"```\n\nWith the URL we retrieved earlier, the command would look like this:\n\n```make scrape URL=\"https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI0LTA5LTA0agwIAxIIL20vMDE1NnFyDAgDEggvbS8wNXF0ahooEgoyMDI0LTA5LTE4agwIAxIIL20vMDVxdGpyDAgDEggvbS8wMTU2cUABSAFwAYIBCwj___________8BmAEB\"```\n\nMake sure to surround the URL with quotation marks, otherwise the tool might have trouble parsing it.\n\nAfter running the command, your terminal should look something like this:\n\n\u003cimg width=\"1143\" alt=\"image\" src=\"https://github.com/user-attachments/assets/c43eb93c-176a-4885-81cd-32dc4efb3b0f\"\u003e\n\nAfter the tool has finished running, you should notice that a `flights.csv` file appeared in your current directory.\n\nThis data in this file has these columns for the Google Flights data based on your provided URL:\n\n- `price` - The full price of a flight.\n- `departure_time` - The departure time of a flight.\n- `arrival_time` - The arrival time of a flight.\n- `airline` - The airline, or multiple airlines that operate the flight.\n- `stops` - The number of stops between the departure and arrival locations.\n- `full_detail` - The full detail of the flight in plain text.\n  \nHere's an example of how the data can look like:\n\n\u003cimg width=\"997\" alt=\"image\" src=\"https://github.com/user-attachments/assets/c1803749-6fa7-4828-8c59-a1bbaa803144\"\u003e\n\n### Notes\n\nIn case the code doesn't work or your project is of bigger scale, please refer to the second part of the tutorial. There, we showcase how to scrape public data with Oxylabs Scraper API.\n\n## Scrape public Google Flights data with Oxylabs API \n\nIn case you were not able to carry out your project with the free scraper, you may use Oxylabs API instead. \n\nIn this section of the guide, we’re going to demonstrate how to scrape public data from **flight pages** and generate **search results** with Python and [Oxylabs Google Flights API](https://oxylabs.io/products/scraper-api/serp/google/flights) (a part of Web Scraper API). To use the Oxylabs API, you'll need an **active subscription** – you can get a **free trial** by signing up [via the self-service dashboard](https://dashboard.oxylabs.io/).\n\nWe’ll gather all sorts of data, including **price**, **flight time**, and **airline name**.\n\nHead to our blog to see the [complete article](https://oxylabs.io/blog/how-to-scrape-google-flights) with in-depth explanations and images for a visual reference.\n\n- [1. Installing prerequisite libraries](#1-installing-prerequisite-libraries)\n- [2. Creating core structure](#2-creating-core-structure)\n- [3. Getting the price](#3-getting-the-price)\n- [4. Getting the flight time](#4-getting-the-flight-time)\n- [5. Getting the airline name](#5-getting-the-airline-name)\n- [6. Final result](#6-final-result)\n\n## Installing prerequisite libraries\n\n```bash\npip install bs4\n```\n\n## Creating core structure\n\nTo start off, let’s create a function that will take a URL as a parameter, scrape that URL with [Google Flights API](https://oxylabs.io/products/scraper-api/serp/google/flights) (you can get **a free 7-day trial** for it) and return the scraped HTML:\n\n```python\ndef get_flights_html(url):\n    payload = {\n        'source': 'google',\n        'render': 'html',\n        'url': url,\n    }\n\n    response = requests.request(\n        'POST',\n        'https://realtime.oxylabs.io/v1/queries',\n        auth=('username', 'password'),\n        json=payload,\n    )\n\n    response_json = response.json()\n\n    html = response_json['results'][0]['content']\n\n    return html\n```\n\nMake sure to change up `USERNAME` and `PASSWORD` with your actual Oxylabs credentials.\n\nNext up, we’ll create a function that accepts a `BeautifulSoup` object created from the HTML of the whole page. This function will create and return an array of objects containing information from individual flight listings. Let’s try to form the function in such a way that makes it easily extendible if required:\n\n```python\ndef extract_flight_information_from_soup(soup_of_the_whole_page):\n    flight_listings = soup_of_the_whole_page.find_all('li','pIav2d')\n\n    flights = []\n\n    for listing in flight_listings:\n        if listing is not None:\n            # Add some specific data extraction here\n\n            flight = {}\n\n            flights.append(flight)\n\n    return flights\n```\n\nNow that we can get the HTML and have a function to hold our information extraction, we can organize both of those into one:\n\n```python\ndef extract_flights_data_from_urls(urls):\n    constructed_flight_results = []\n\n    for url in urls:\n        html = get_flights_html(url)\n\n        soup = BeautifulSoup(html,'html.parser')\n\n        flights = extract_flight_information_from_soup(soup)\n\n        constructed_flight_results.append({\n            'url': url,\n            'flight_data': flights\n        })\n\n    return constructed_flight_results\n```\n\nThis function will take an array of URLs as a parameter and return an object of extracted flight data.\n\nOne thing left for our core is a function that takes this data and saves it as a file:\n\n```python\ndef save_results(results, filepath):\n    with open(filepath, 'w', encoding='utf-8') as file:\n        json.dump(results, file, ensure_ascii=False, indent=4)\n\n    return\n```\n\nWe can finish by creating a simple `main` function to invoke all that we’ve created so far:\n```python\ndef main():\n    results_file = 'data.json'\n\n    urls = [\n        'https://www.google.com/travel/flights?tfs=CBsQAhooEgoyMDI0LTA3LTI4agwIAxIIL20vMDE1NnFyDAgCEggvbS8wNGpwbBooEgoyMDI0LTA4LTAxagwIAhIIL20vMDRqcGxyDAgDEggvbS8wMTU2cUABSAFSA0VVUnABemxDalJJTkRCNVRGbDBOMU5UVEdOQlJ6aG5lRUZDUnkwdExTMHRMUzB0TFMxM1pXc3lOMEZCUVVGQlIxZ3dhRWxSUVRoaWFtTkJFZ1pWTWpnMk1qSWFDZ2lRYnhBQ0dnTkZWVkk0SEhEN2VBPT2YAQGyARIYASABKgwIAxIIL20vMDRqcGw\u0026hl=en-US\u0026curr=EUR\u0026sa=X\u0026ved=0CAoQtY0DahgKEwiAz9bF5PaEAxUAAAAAHQAAAAAQngM',\n        'https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI0LTA3LTI4agwIAxIIL20vMDE1NnFyDAgDEggvbS8wN19rcRooEgoyMDI0LTA4LTAxagwIAxIIL20vMDdfa3FyDAgDEggvbS8wMTU2cUABSAFwAYIBCwj___________8BmAEB\u0026hl=en-US\u0026curr=EUR'\n    ]\n\n    constructed_flight_results = extract_flights_data_from_urls(urls)\n\n    save_results(constructed_flight_results, results_file)\n```\n\n## Getting the price\n\n```python\ndef get_price(soup_element):\n    price = soup_element.find('div','BVAVmf').find('div','YMlIz').get_text()\n\n    return price\n```\n\n## Getting the flight time\n\n```python\ndef get_time(soup_element):\n    spans = soup_element.find('div','Ir0Voe').find('div','zxVSec', recursive=False).find_all('span', 'eoY5cb')\n\n    time = \"\"\n\n    for span in spans:\n        time = time + span.get_text() + \"; \"\n\n    return time\n```\n\n## Getting the airline name\n\n```python\ndef get_airline(soup_element):\n    airline = soup_element.find('div','Ir0Voe').find('div','sSHqwe')\n\n    spans = airline.find_all('span', attrs={\"class\": None}, recursive=False)\n\n    result = \"\"\n\n    for span in spans:\n        result = result + span.get_text() + \"; \"\n\n    return result\n```\n\nHaving all of these functions for data extraction, we just need to add them to the place we designated earlier to finish up our code.\n\n\n```python\ndef extract_flight_information_from_soup(soup_of_the_whole_page):\n    flight_listings = soup_of_the_whole_page.find_all('li','pIav2d')\n\n    flights = []\n\n    for listing in flight_listings:\n        if listing is not None:\n            price = get_price(listing)\n            time = get_time(listing)\n            airline = get_airline(listing)\n\n            flight = {\n                \"airline\": airline,\n                \"time\": time,\n                \"price\": price\n            }\n\n            flights.append(flight)\n\n    return flights\n```\n## Final result\nIf we add all of it together, the final product should look something like this.\n\n\n```python\nfrom bs4 import BeautifulSoup\nimport requests\nimport json\n\ndef get_price(soup_element):\n    price = soup_element.find('div','BVAVmf').find('div','YMlIz').get_text()\n\n    return price\n\n\ndef get_time(soup_element):\n    spans = soup_element.find('div','Ir0Voe').find('div','zxVSec', recursive=False).find_all('span', 'eoY5cb')\n\n    time = \"\"\n\n    for span in spans:\n        time = time + span.get_text() + \"; \"\n\n    return time\n\n\ndef get_airline(soup_element):\n    airline = soup_element.find('div','Ir0Voe').find('div','sSHqwe')\n\n    spans = airline.find_all('span', attrs={\"class\": None}, recursive=False)\n\n    result = \"\"\n\n    for span in spans:\n        result = result + span.get_text() + \"; \"\n\n    return result\n\n\ndef save_results(results, filepath):\n    with open(filepath, 'w', encoding='utf-8') as file:\n        json.dump(results, file, ensure_ascii=False, indent=4)\n\n    return\n\n\ndef get_flights_html(url):\n    payload = {\n        'source': 'google',\n        'render': 'html',\n        'url': url,\n    }\n\n    # Get response.\n    response = requests.request(\n        'POST',\n        'https://realtime.oxylabs.io/v1/queries',\n        auth=('username', 'password'),\n        json=payload,\n    )\n\n    response_json = response.json()\n\n    html = response_json['results'][0]['content']\n\n    return html\n\n\ndef extract_flight_information_from_soup(soup_of_the_whole_page):\n    flight_listings = soup_of_the_whole_page.find_all('li','pIav2d')\n\n    flights = []\n\n    for listing in flight_listings:\n        if listing is not None:\n            price = get_price(listing)\n            time = get_time(listing)\n            airline = get_airline(listing)\n\n            flight = {\n                \"airline\": airline,\n                \"time\": time,\n                \"price\": price\n            }\n\n            flights.append(flight)\n\n    return flights\n\n\ndef extract_flights_data_from_urls(urls):\n    constructed_flight_results = []\n\n    for url in urls:\n        html = get_flights_html(url)\n\n        soup = BeautifulSoup(html,'html.parser')\n\n        flights = extract_flight_information_from_soup(soup)\n\n        constructed_flight_results.append({\n            'url': url,\n            'flight_data': flights\n        })\n\n    return constructed_flight_results\n\n\ndef main():\n    results_file = 'data.json'\n\n    urls = [\n        'https://www.google.com/travel/flights?tfs=CBsQAhooEgoyMDI0LTA3LTI4agwIAxIIL20vMDE1NnFyDAgCEggvbS8wNGpwbBooEgoyMDI0LTA4LTAxagwIAhIIL20vMDRqcGxyDAgDEggvbS8wMTU2cUABSAFSA0VVUnABemxDalJJTkRCNVRGbDBOMU5UVEdOQlJ6aG5lRUZDUnkwdExTMHRMUzB0TFMxM1pXc3lOMEZCUVVGQlIxZ3dhRWxSUVRoaWFtTkJFZ1pWTWpnMk1qSWFDZ2lRYnhBQ0dnTkZWVkk0SEhEN2VBPT2YAQGyARIYASABKgwIAxIIL20vMDRqcGw\u0026hl=en-US\u0026curr=EUR\u0026sa=X\u0026ved=0CAoQtY0DahgKEwiAz9bF5PaEAxUAAAAAHQAAAAAQngM',\n        'https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI0LTA3LTI4agwIAxIIL20vMDE1NnFyDAgDEggvbS8wN19rcRooEgoyMDI0LTA4LTAxagwIAxIIL20vMDdfa3FyDAgDEggvbS8wMTU2cUABSAFwAYIBCwj___________8BmAEB\u0026hl=en-US\u0026curr=EUR'\n    ]\n\n    constructed_flight_results = extract_flights_data_from_urls(urls)\n\n    save_results(constructed_flight_results, results_file)\n\n\nif __name__ == \"__main__\":\n    main()\n```\n\nBy employing Python and Oxylabs Web Scraper API, you can easily deal with the dynamic nature of Google Flights and gather public data successfully.\n\nRead More Google Scraping Related Repositories: [Google Sheets for Basic Web Scraping](https://github.com/oxylabs/web-scraping-google-sheets), [How to Scrape Google Shopping Results](https://github.com/oxylabs/scrape-google-shopping), [Google Play Scraper](https://github.com/oxylabs/google-play-scraper), [How To Scrape Google Jobs](https://github.com/oxylabs/how-to-scrape-google-jobs), [Google News Scrpaer](https://github.com/oxylabs/google-news-scraper), [How to Scrape Google Scholar](https://github.com/oxylabs/how-to-scrape-google-scholar), [How To Scrape Google Images](https://github.com/oxylabs/how-to-scrape-google-images), [Scrape Google Search Results](https://github.com/oxylabs/scrape-google-python), [Scrape Google Trends](https://github.com/oxylabs/how-to-scrape-google-trends)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fhow-to-scrape-google-flights","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fhow-to-scrape-google-flights","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fhow-to-scrape-google-flights/lists"}