{"id":15060674,"url":"https://github.com/stkchan/web-scraping-with-selenium","last_synced_at":"2026-01-02T07:47:27.867Z","repository":{"id":242659490,"uuid":"806042949","full_name":"stkchan/Web-Scraping-With-Selenium","owner":"stkchan","description":null,"archived":false,"fork":false,"pushed_at":"2024-06-09T08:49:24.000Z","size":27,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-21T22:09:57.621Z","etag":null,"topics":["bigquery","pandas","python","selenium-webdriver","webscraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stkchan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-26T07:59:21.000Z","updated_at":"2024-06-09T08:49:27.000Z","dependencies_parsed_at":"2024-11-21T04:02:47.274Z","dependency_job_id":"63537d3f-c3b8-4be2-8c8e-64276a2f3a42","html_url":"https://github.com/stkchan/Web-Scraping-With-Selenium","commit_stats":{"total_commits":8,"total_committers":1,"mean_commits":8.0,"dds":0.0,"last_synced_commit":"baf592ca8de5aba2d517467e0e10d625108f53b3"},"previous_names":["stkchan/web-scraping-with-selenium"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stkchan%2FWeb-Scraping-With-Selenium","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stkchan%2FWeb-Scraping-With-Selenium/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stkchan%2FWeb-Scraping-With-Selenium/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stkchan%2FWeb-Scraping-With-Selenium/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stkchan","download_url":"https://codeload.github.com/stkchan/Web-Scraping-With-Selenium/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243695589,"owners_count":20332629,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","pandas","python","selenium-webdriver","webscraping"],"created_at":"2024-09-24T23:02:34.505Z","updated_at":"2026-01-02T07:47:27.833Z","avatar_url":"https://github.com/stkchan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web-Scraping-With-Selenium\n\n## Table of Contents\n* [Project Objective](#Project-Objective)\n* [Setting up Selenium](#Setting-up-Selenium)\n   - [Selenium Package](#Selenium-Package)\n* [Scraping Process](#DScraping-Process)\n   - [Import libraries](#Import-libraries)\n   - [Define driver path](#Define-driver-path)\n   - [Create function for scraping data](#Create-function-for-scraping-data)\n   - [Create DataFrame and save to CSV file](#Create-DataFrame-and-save-to-CSV-file)\n* [Upload Data to BigQuery](#Upload-data-to-BigQuery)\n   - [Prerequisite](#Prerequisite)\n   - [Example of script for uploading data to BigQuery](#Example-of-script-for-uploading-data-to-BigQuery)\n\n  \n\n## Project Objective\nIn this project, we aim to scrape data from the webpage [https://www.homepro.co.th/c/FUR](https://www.homepro.co.th/c/FUR), transform it into table format using [Pandas](https://pypi.org/project/pandas/), and store the data in [BigQuery](https://cloud.google.com/bigquery?hl=en).\n\n## Setting up Selenium\nSelenium is an open-source tool that automates web browsers. It provides a suite of tools and libraries that enable testing of web applications, automating repetitive web-based tasks, and scraping web content. Selenium supports multiple programming languages such as Java, C#, Python, and Ruby, allowing developers to write tests in their preferred language. \n\n### Selenium Package\n\n```shell Tab A\npip install selenium\n```\nlink to download the selenium drivers for Firefox, Chrome, and Edge [here](https://pypi.org/project/selenium/#drivers)\n\nTo see Selenium in action, type in these lines in your favorite code editor and run it as a Python script. You can also run these statements from the Python console.\n\n```python\nfrom selenium.webdriver import Chrome\ndriver = Chrome(executable_path='C:/WebDrivers/chromedriver.exe')\ndriver.get(\"https://www.homepro.co.th/\")\n```\nThis will launch Chrome and load the web page.  There will be notice below the address bar:\n\n```shell\nChrome is being controlled by automated test software.\n```\n\nTo close this browser, simply run this line:\n\n```python\ndriver.quit()\n```\n\n## Scraping Process\n### Import libraries\nTo begin with, I have imported the necessary libraries for this project.\n\n```python\nfrom selenium import webdriver\nfrom selenium.webdriver.common.by import By\nfrom selenium.webdriver.chrome.service import Service\nfrom selenium.common.exceptions import NoSuchElementException\nimport pandas as pd\nimport re\nimport time\nfrom datetime import datetim\n```\n\n### Define driver path\n```python\ndriver_path = r\"C:\\MyProject\\Python_test\\chromedriver.exe\"\nservice = Service(driver_path)\ndriver = webdriver.Chrome(service=service)\n```\n\n### Create function for scraping data\n```python\ndef scrape_prices(base_url, category, pages=5):\n    data = {\n        \"category\"            : [],\n        \"brand\"               : [],\n        \"model\"               : [],\n        \"sku\"                 : [],\n        \"review_rating\"       : [],\n        \"discount_price\"      : [],\n        \"full_price\"          : [],\n        \"url\"                 : []\n    }\n\n    # Loop for scraping data in multiple pages\n    for page in range(1, pages + 1):\n        current_url = f\"{base_url}\u0026page={page}#plist\"\n        driver.get(current_url)\n        time.sleep(10)\n\n        # Find Element of Data\n        product_list = driver.find_elements(By.CLASS_NAME, \"product-plp-card\")\n\n        for p in product_list:\n            listcheck = p.text.split(\"\\n\")\n            print(\"DEBUG: listcheck =\", listcheck)\n\n            # Ensure the list has sufficient length before accessing elements\n            if len(listcheck) \u003c 3:\n                continue\n\n            # Get URL\n            try:\n                links  = p.find_element(By.CLASS_NAME, \"plp-card-top\")\n                alltag = links.find_elements(By.TAG_NAME, \"a\")\n                url    = alltag[-1].get_attribute(\"href\")\n            except NoSuchElementException:\n                url    = None\n\n            # Regular expression to match patterns\n            pattern = re.compile(r'\\(\\d+\\)')\n\n            # Extract data based on the conditions\n            brand = model = sku = review_rating = discount_price = full_price = free_option = seller_detail = None\n\n            # Checking if the review_rating contains parentheses with numbers\n            if any(pattern.search(line) for line in listcheck):\n                review_rating = next((line for line in listcheck if pattern.search(line)), None)\n\n            # Overall conditions to extract the rest of the data\n            if review_rating and listcheck[-1] == \"เปรียบเทียบ\":\n                brand           = listcheck[0]\n                model           = listcheck[1]\n                sku             = listcheck[2]\n                discount_price  = listcheck[-5]\n                full_price      = listcheck[-4]\n                \n\n            elif review_rating and listcheck[-1] == \"จำหน่ายโดย: โฮมโปร\":\n                brand           = listcheck[0]\n                model           = listcheck[1]\n                sku             = listcheck[2]\n                discount_price  = listcheck[-4]\n                full_price      = listcheck[-3]\n                \n\n            elif listcheck[0] == \"ใหม่ล่าสุด!\":\n                brand           = listcheck[1]\n                model           = listcheck[2]\n                sku             = listcheck[3]\n                discount_price  = listcheck[-5]\n                full_price      = listcheck[-4]\n                \n\n            elif listcheck[-3] == \"ฟรีประกอบ\":\n                brand           = listcheck[0]\n                model           = listcheck[1]\n                sku             = listcheck[2]\n                discount_price  = listcheck[-5]\n                full_price      = listcheck[-4]\n \n\n            else:\n                brand           = listcheck[0]\n                model           = listcheck[1]\n                sku             = listcheck[2]\n                discount_price  = listcheck[-4]\n                full_price      = listcheck[-3]\n               \n\n            # Append data into list\n            data[\"category\"].append(category)\n            data[\"brand\"].append(brand)\n            data[\"model\"].append(model)\n            data[\"sku\"].append(sku)\n            data[\"review_rating\"].append(review_rating)\n            data[\"discount_price\"].append(discount_price)\n            data[\"full_price\"].append(full_price)\n            data[\"url\"].append(url)\n\n            print(\"========================================================================================================\")\n\n    return pd.DataFrame(data)\n\n\ndef desk_price():\n    base_url = \"https://www.homepro.co.th/c/FUR1005?s=12\u0026size=200\u0026cst=0\u0026pmin=\u0026pmax=\u0026q=โต๊ะทำงาน\"\n    return scrape_prices(base_url, \"desk\")\n\ndef chair_price():\n    base_url = \"https://www.homepro.co.th/c/FUR1004?s=12\u0026size=200\u0026cst=0\u0026pmin=\u0026pmax=\u0026q=เก้าอี้สำนักงาน\"\n    return scrape_prices(base_url, \"chair\")\n\ndef sofa_price():\n    base_url = \"https://www.homepro.co.th/c/FUR0805?s=12\u0026size=200\u0026cst=0\u0026pmin=\u0026pmax=\u0026q=โซฟา\"\n    return scrape_prices(base_url, \"sofa\")\n\ndef bed_price():\n    base_url = \"https://www.homepro.co.th/c/FUR0102?s=12\u0026size=200\u0026cst=0\u0026pmin=\u0026pmax=\u0026q=เตียงนอน\"\n    return scrape_prices(base_url, \"bed\")\n\ndef bedroom_price():\n    base_url = \"https://www.homepro.co.th/c/FUR0101?s=12\u0026size=200\u0026cst=0\u0026pmin=\u0026pmax=\u0026q=ชุดห้องนอน\"\n    return scrape_prices(base_url, \"bed_set\")\n```\n\n\n### Create DataFrame and save to CSV file\n```python\n# Combine data\ndef combine_furniture_prices():\n    desk_df    = desk_price()\n    chair_df   = chair_price()\n    sofa_df    = sofa_price()\n    bed_df     = bed_price()\n    bedset_df  = bedroom_price()\n\n\n    combined_df = pd.concat([desk_df, chair_df, sofa_df, bed_df, bedset_df], ignore_index=True)\n\n\n    # Save to CSV File\n    combined_df.to_csv(\"combined_furniture_prices_version2.csv\", index=False, encoding=\"utf-8-sig\")\n    \n    return combined_df\n\n\n# Call the function\ntry:\n    combined_df = combine_furniture_prices()\n    print(\"Data saved to combined_furniture_prices_version2.csv\")\nfinally:\n    driver.quit()\n```\n\n## Upload Data to BigQuery\n\n### Prerequisite\nBefore we export our data to BigQuery, we need to complete these steps (assuming we already have a Google Cloud account).\n  - Create a Google credential to obtain an access token [here](https://developers.google.com/workspace/guides/create-credentials)\n  - Create a table in BigQuery [here](https://cloud.google.com/bigquery/docs/tables)\n  - Install Google Cloud BigQuery library\n    ```python\n       pip install google-cloud-bigquery\n    ```\n\n    \n### Example of script for uploading data to BigQuery\n\n```python\n    from google.oauth2 import service_account\n    from google.cloud import bigquery\n    from pandas.io import gbq\n    from pandas_gbq import to_gbq\n    import googleapiclient.discovery\n    from load_credentials import load_credentials \n    import os\n    import pandas_gbq\n    \n    \n    # Load credentials from credentials.txt file\n    credentials = load_credentials('credential.txt')\n    \n    # Verify the loaded credentials\n    print(\"GOOGLE_APPLICATION_CREDENTIALS:\", credentials['GOOGLE_APPLICATION_CREDENTIALS'])\n    print(\"BIGQUERY_TABLE_ID:\", credentials['BIGQUERY_TABLE_ID'])\n    \n    # Get the environment variables from the loaded credentials\n    credentials_file = credentials['GOOGLE_APPLICATION_CREDENTIALS']\n    table_id = credentials['BIGQUERY_TABLE_ID']\n    \n    \n    # Export to BigQuery\n    def export_to_bigquery(df, table_id, credentials_file):\n        credentials = service_account.Credentials.from_service_account_file(credentials_file)\n        to_gbq(df, table_id, project_id=credentials.project_id, credentials=credentials)\n        print(\"Export data to BigQuery successfully.\")\n    \n    \n    export_to_bigquery(df_cleaned, table_id, credentials_file)\n```\n\n**Note:** You can find the table_id, project_id, and credentials in your JSON file after creating credentials in Google Cloud Platform. The table_id is located in BigQuery, which you created.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstkchan%2Fweb-scraping-with-selenium","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstkchan%2Fweb-scraping-with-selenium","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstkchan%2Fweb-scraping-with-selenium/lists"}