{"id":16031746,"url":"https://github.com/ygalvao/py-easy-scrape","last_synced_at":"2026-01-03T23:52:45.595Z","repository":{"id":172589347,"uuid":"648017206","full_name":"ygalvao/py-easy-scrape","owner":"ygalvao","description":"Useful wrappings to Selenium-based web scrapers.","archived":false,"fork":false,"pushed_at":"2024-07-30T14:08:45.000Z","size":21,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-05T07:43:56.868Z","etag":null,"topics":["data-science","scraper","scraping","selenium","web-scraping","webdriver","webscraping"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/py-easy-scrape/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ygalvao.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-01T03:02:57.000Z","updated_at":"2024-07-30T14:08:49.000Z","dependencies_parsed_at":null,"dependency_job_id":"3604e1d7-0a2c-42c4-bd02-e5c5e04f957d","html_url":"https://github.com/ygalvao/py-easy-scrape","commit_stats":{"total_commits":11,"total_committers":1,"mean_commits":11.0,"dds":0.0,"last_synced_commit":"09e67878677919ece3a168937ebee4fab0e15e05"},"previous_names":["ygalvao/py-easy-scrap","ygalvao/py-easy-scrape"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ygalvao%2Fpy-easy-scrape","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ygalvao%2Fpy-easy-scrape/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ygalvao%2Fpy-easy-scrape/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ygalvao%2Fpy-easy-scrape/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ygalvao","download_url":"https://codeload.github.com/ygalvao/py-easy-scrape/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244894309,"owners_count":20527669,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","scraper","scraping","selenium","web-scraping","webdriver","webscraping"],"created_at":"2024-10-08T21:05:17.641Z","updated_at":"2026-01-03T23:52:45.523Z","avatar_url":"https://github.com/ygalvao.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Py-Easy-Scrape\n\n## Description\n\n__A useful package for web scraping with Selenium.__\n\nThis package provides useful wrapping of some of the more cumbersome aspects of using Selenium directly, such as initializing the WebDriver, scrolling, and handling elements.\n\nAlso, it simplifies some common tasks in web scraper bots, such as  asking the user to manually input some data when necessary and checking if some file exists in the user's computer.\n\nMoreover, it kind of does the same with Logging: Py Easy Scrape comes with built-in logging, which can be a significant advantage for quickly deploying, debugging, and monitoring your web scraping project!\n\n## Installation\n\n```bash\npip install py-easy-scrape\n```\n\n## Requirements\n\n- Python 3.7 or higher\n- Selenium, along with __GeckoDriver__ (Firefox)\n\n## Usage\n\nPy-Easy-Scrape has a built-in logging system that, so far, needs a directory for all the logs that are going to be generated. Hence, in your project directory, create a new directory called 'logs':\n```bash\nmkdir logs\n```\n\nThen, import the package:\n```python\nimport pyeasyscrape as pes\n```\n\n### Get WebDriver\n\n```python\ndriver = pes.get_webdriver(headless=True)\n```\nThis will return a headless instance of webdriver.Firefox (GeckoDriver).\n\n### Get Element\n\n```python\nelement = pes.get_element(driver, value=\"element_xpath\")\n```\n\nThis will return the web element found using the specified driver and value. By default, this function uses 'XPATH' as the method of searching. However, you can change this by providing 'ID' or 'LINK_TEXT' as the 'by' argument.\n\nPlease be aware that the method of searching ('XPATH', 'ID', or 'LINK_TEXT') highly depends on the webpage structure. Therefore, it is essential to inspect the webpage beforehand.\n\nAlso, please remember to replace `\"element_xpath\"` with the actual Xpath, ID, or link text of the element you want to find.\n\n### Check Element\n\n```python\nis_clickable = check_element(driver, xpath=\"element_xpath\")\n```\n\nThis will return `True` if the web element specified by the XPath is found, displayed, and clickable using the provided driver. If not, it will return `False`. \n\nAlternatively, you can directly pass a web element object:\n\n```python\nis_clickable = check_element(driver, web_element=element)\n```\n\nPlease replace `\"element_xpath\"` with the actual XPath of the element you want to check. This method ensures the element is both present and interactive before performing further actions.\n\n\n### Scroll to Element\n\n```python\npes.scroll_to(driver, element)\n```\n\nThis will use JavaScript to scroll the view of the given webdriver to the specified web element.\n\n### Ask for Data\n\n```python\nrequired_data = (\"username\", \"password\")\nfile_name_no_extension = \"user_credentials\"\npes.ask_for_data(required_data, file_name_no_extension)\n```\n\nThis will return a dictionary containing the collected data and will create a JSON configuration (.conf) file.\n\n### Check File\n\n```python\npes.check_file(\"path_to_file\")\n```\n\nThis will return either True or False based on file existence\n\nPlease note that the above code snippets are basic examples of how to use those functions. Depending on the exact specifications of your use case, additional setup may be necessary (e.g. logging into a website before trying to scroll to an element).\n\n## Contributing\n\nPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.\n\n## License\n\nMIT\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fygalvao%2Fpy-easy-scrape","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fygalvao%2Fpy-easy-scrape","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fygalvao%2Fpy-easy-scrape/lists"}