{"id":20709981,"url":"https://github.com/oxylabs/web-scraping-selenium-python","last_synced_at":"2026-02-11T05:02:01.628Z","repository":{"id":134336691,"uuid":"561787014","full_name":"oxylabs/web-scraping-selenium-python","owner":"oxylabs","description":"Web Scraping with Python Selenium: Tutorial for Beginners","archived":false,"fork":false,"pushed_at":"2025-09-23T08:52:24.000Z","size":16,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-23T10:27:49.638Z","etag":null,"topics":["github-python","json-database-python","python-ecommerce","python-web-crawler","scraper-python","selenium-web-scraper","serp-api-python","web-scraping","web-scraping-python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-11-04T13:45:33.000Z","updated_at":"2025-09-23T08:52:28.000Z","dependencies_parsed_at":null,"dependency_job_id":"c56f71fb-df84-4ca1-86b0-6447d4f80430","html_url":"https://github.com/oxylabs/web-scraping-selenium-python","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oxylabs/web-scraping-selenium-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fweb-scraping-selenium-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fweb-scraping-selenium-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fweb-scraping-selenium-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fweb-scraping-selenium-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/web-scraping-selenium-python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fweb-scraping-selenium-python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29327093,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-11T03:52:29.695Z","status":"ssl_error","status_checked_at":"2026-02-11T03:52:23.094Z","response_time":97,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["github-python","json-database-python","python-ecommerce","python-web-crawler","scraper-python","selenium-web-scraper","serp-api-python","web-scraping","web-scraping-python"],"created_at":"2024-11-17T02:09:27.332Z","updated_at":"2026-02-11T05:02:01.614Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","readme":"# Web Scraping with Python Selenium: Tutorial for Beginners\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.io/pages/gitoxy?utm_source=877\u0026utm_medium=affiliate\u0026groupid=877\u0026utm_content=web-scraping-selenium-python-github\u0026transaction_id=102f49063ab94276ae8f116d224b67)\n\n\n[![](https://dcbadge.limes.pink/api/server/Pds3gBmKMH?style=for-the-badge\u0026theme=discord)](https://discord.gg/Pds3gBmKMH) [![YouTube](https://img.shields.io/badge/YouTube-Oxylabs-red?style=for-the-badge\u0026logo=youtube\u0026logoColor=white)](https://www.youtube.com/@oxylabs)\n\n[\u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=python\u0026color=brightgreen\" /\u003e](https://github.com/topics/python) [\u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=selenium\u0026color=blue\" /\u003e](https://github.com/topics/selenium) [\u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=Web%20Scraping\u0026color=important\" /\u003e](https://github.com/topics/web-scraping)\n- [Installing Selenium](#installing-selenium)\n- [Testing](#testing)\n- [Scraping with Selenium](#scraping-with-selenium)\n\nIn this article, we’ll cover an overview of web scraping with Selenium using a real-life example.\n\nFor a detailed tutorial on Selenium, see [our blog](https://oxylabs.io/blog/selenium-web-scraping).\n\n## Installing Selenium\n\n1. Create a virtual environment:\n\n```sh\npython3 -m venv .env\n```\n\n2. Install Selenium using pip:\n\n```sh\npip install selenium\n```\n\n3. Install Selenium Web Driver. See [this page](https://www.selenium.dev/documentation/webdriver/getting_started/install_drivers/) for details.\n\n## Testing\n\nWith virtual environment activated, enter IDLE by typing in `python3`. Enter the following command on IDLE:\n\n```python\n\u003e\u003e\u003e from selenium.webdriver import Chrome\n\n```\n\nIf there are no errors, move on to the next step. If there is an error, ensure that `chromedriver` is added to the PATH.\n\n## Scraping with Selenium\n\nImport required modules as follows:\n\n```python\nfrom selenium.webdriver import Chrome, ChromeOptions\nfrom selenium.webdriver.common.by import By\n```\n\nAdd the skeleton of the script as follows:\n\n```python\ndef get_data(url) -\u003e list:\n   ...\n\n\ndef main():\n    ...\n\nif __name__ == '__main__':\n    main()\n```\n\nCreate ChromeOptions object and set `headless` to `True`. Use this to create an instance of `Chrome`.\n\n```python\n    browser_options = ChromeOptions()\n    browser_options.headless = True\n\n    driver = Chrome(options=browser_options)\n```\n\nCall the `driver.get` method to load a URL. After that, locate the link for the Humor section by link text and click it:\n\n```python\n    driver.get(url)\n\n    element = driver.find_element(By.LINK_TEXT, \"Humor\")\n    element.click()\n```\n\nCreate a CSS selector to find all books from this page. After that run a loop on the books and find the bookt title, price, stock availability. Use a dictionary to store one book information and add all these dictionaries to a list. See the code below:\n\n```python\n books = driver.find_elements(By.CSS_SELECTOR, \".product_pod\")\n    data = []\n    for book in books:\n        title = book.find_element(By.CSS_SELECTOR, \"h3 \u003e a\")\n        price = book.find_element(By.CSS_SELECTOR, \".price_color\")\n        stock = book.find_element(By.CSS_SELECTOR, \".instock.availability\")\n        book_item = {\n            'title': title.get_attribute(\"title\"),\n            'price': price.text,\n            'stock': stock. text\n        }\n        data.append(book_item)\n\n```\n\nLastly, return the `data` dictionary from this function.\n\nFor the complete code, see [main.py](src/main.py).\n\nFor a detailed tutorial on Selenium, see [our blog](https://oxylabs.io/blog/selenium-web-scraping).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fweb-scraping-selenium-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fweb-scraping-selenium-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fweb-scraping-selenium-python/lists"}