{"id":24824727,"url":"https://github.com/vishal815/flight_details_scraper_and_visualization","last_synced_at":"2026-04-13T01:39:47.306Z","repository":{"id":272642912,"uuid":"917292482","full_name":"vishal815/Flight_Details_Scraper_and_Visualization","owner":"vishal815","description":"This project focuses on scraping flight details from Google Flights, processing the data, and performing cleaning and visualization for future use in analytics or predictive modeling.","archived":false,"fork":false,"pushed_at":"2025-01-21T00:11:33.000Z","size":333,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T23:48:58.960Z","etag":null,"topics":["beautifulsoup","end-to-end-project","flight-data","flight-data-analysis","google-flights","google-flights-scraper","pandas","pyhon","scraping-project","scraping-python","selenium","vishal-lazrus","vishallazrus","visualization","webscraping","webscraping-data"],"latest_commit_sha":null,"homepage":"https://github.com/vishal815/Flight_Details_Scraper_and_Visualization/blob/main/scrape_flight_details_and_visualization.ipynb","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vishal815.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-15T18:00:53.000Z","updated_at":"2025-01-21T00:11:37.000Z","dependencies_parsed_at":"2025-01-15T20:27:30.460Z","dependency_job_id":"b1d581d7-0660-471e-a88f-0d3ef5a55ba2","html_url":"https://github.com/vishal815/Flight_Details_Scraper_and_Visualization","commit_stats":null,"previous_names":["vishal815/flight_details_scraper_and_visualization"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vishal815/Flight_Details_Scraper_and_Visualization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vishal815%2FFlight_Details_Scraper_and_Visualization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vishal815%2FFlight_Details_Scraper_and_Visualization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vishal815%2FFlight_Details_Scraper_and_Visualization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vishal815%2FFlight_Details_Scraper_and_Visualization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vishal815","download_url":"https://codeload.github.com/vishal815/Flight_Details_Scraper_and_Visualization/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vishal815%2FFlight_Details_Scraper_and_Visualization/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31736723,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-12T22:19:12.206Z","status":"ssl_error","status_checked_at":"2026-04-12T22:18:33.088Z","response_time":58,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautifulsoup","end-to-end-project","flight-data","flight-data-analysis","google-flights","google-flights-scraper","pandas","pyhon","scraping-project","scraping-python","selenium","vishal-lazrus","vishallazrus","visualization","webscraping","webscraping-data"],"created_at":"2025-01-30T20:57:05.494Z","updated_at":"2026-04-13T01:39:47.277Z","avatar_url":"https://github.com/vishal815.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Flight Data Scraper and Visualization\n\n## Overview\nThis project focuses on scraping flight details from [Google Flights](https://www.google.com/travel/flights?gl=IN\u0026hl=en), processing the data, and performing cleaning and visualization for future use in analytics or predictive modeling.\n\n## Workflow step-by-step\n![web_scraping_workflow](https://github.com/user-attachments/assets/23379c68-2f55-4110-b40a-89cd9fd4c002)\n\n## Features\n- **Web Scraping**: Extracts detailed flight information, including:\n  - Airline names\n  - Flight duration\n  - Price\n  - Departure and arrival times\n  - Departure and arrival dates\n  - Airports\n  - Stops\n  - CO2 emissions\n- **Data Cleaning**: Processes and filters scraped data to handle missing or unavailable values.\n- **CSV Outputs**: Saves data in various stages for easy access and further use.\n\n)\n## Website\nData is scraped from [Google Flights](https://www.google.com/travel/flights?gl=IN\u0026hl=en).\n\n![Detailed flight information which I am going to Scrap](![Screenshot 2025-01-15 084745](https://github.com/user-attachments/assets/1d87c857-eba2-4f17-b0d4-65c3dee70983)\n\n## Generated Files\n1. `google_flights_data.csv`: Raw flight data scraped from the website.\n2. `price_unavailable_data.csv`: Records where price information is unavailable.\n3. `cleaned_google_flights_data.csv`: Cleaned and processed dataset ready for analysis.\n\n## Technologies Used\n- **Python**\n- **Selenium**: For browser automation and interaction.\n- **BeautifulSoup**: For parsing HTML and extracting data.\n- **Pandas**: For data manipulation and cleaning.\n\n## Steps to Run\n1. **Install Dependencies**:\n   ```bash\n   pip install selenium beautifulsoup4 pandas\n   ```\n2. **Set Up ChromeDriver**:\n   - Download the appropriate version of ChromeDriver from [here](https://chromedriver.chromium.org/downloads).\n   - Update the path to the driver in the script.\n3. **Run the Script**:\n   ```bash\n   python scrape_flight_details_and_visualization.ipynb\n   ```\n4. **Check Outputs**:\n   - The scraped data will be saved as CSV files in the working directory.\n\n## Data Flow\n1. **Scraping**:\n   - Selenium navigates to the specified Google Flights URL.\n   - BeautifulSoup parses the HTML to extract flight details.\n2. **Data Cleaning**:\n   - Processes columns like \"Price\" and \"Arrival Time.\"\n   - Saves filtered data to `cleaned_google_flights_data.csv`.\n\n## Example Output\n| Airline | Flight Duration | Price    | Departure Time | Departure Date | Departure Airport                              | Arrival Time | Arrival Date | Arrival Airport           | Stops  | CO2 Emissions | Next Day Dispatcher |\n|---------|-----------------|----------|----------------|----------------|-----------------------------------------------|--------------|--------------|---------------------------|--------|---------------|---------------------|\n| IndiGo  | 11 hr 30 min    | ₹27,515  | 8:45 AM        | Sun, Feb 2     | Jayprakash Narayan International Airport, Patna | 6:45 PM      | Sun, Feb 2   | Zayed International Airport | 1 stop | 266 kg CO2e   | 0                   |\n| IndiGo  | 11 hr 15 min    | ₹28,349  | 12:40 PM       | Sun, Feb 2     | Jayprakash Narayan International Airport, Patna | 10:25 PM     | Sun, Feb 2   | Zayed International Airport | 1 stop | 257 kg CO2e   | 0                   |\n\n\n## some output of visualization\n![image](https://github.com/user-attachments/assets/9c5eb579-aa6c-40da-b43c-6b845f87bc3f)\n\n![image](https://github.com/user-attachments/assets/97e9a9d2-f4ba-4905-9f1c-88f7de3ec21f)\n\n\n## Future Work\n- Autometion using crontab linux.\n- Collect more data for extended analysis.\n- Implement advanced visualization techniques.\n- Apply ML/DL models for predictive analytics, such as estimating missing flight prices.\n\n## License\nThis project is licensed under the MIT License. See the LICENSE file for more details.\n\n---\n\n*For questions or contributions, feel free to open an issue or submit a pull request.*\n## Vishal Lazrus\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvishal815%2Fflight_details_scraper_and_visualization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvishal815%2Fflight_details_scraper_and_visualization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvishal815%2Fflight_details_scraper_and_visualization/lists"}