{"id":15177570,"url":"https://github.com/yurnerosk/web_scraper_challenge","last_synced_at":"2026-03-07T19:33:02.433Z","repository":{"id":252820465,"uuid":"840823970","full_name":"Yurnerosk/web_scraper_challenge","owner":"Yurnerosk","description":"Web scraper I did for interview. Now, contributions welcome!!","archived":false,"fork":false,"pushed_at":"2024-08-22T17:13:30.000Z","size":1715,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-31T21:11:29.643Z","etag":null,"topics":["python3","robocorp","selenium-webdriver"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Yurnerosk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-10T19:47:26.000Z","updated_at":"2024-08-22T17:13:34.000Z","dependencies_parsed_at":"2024-08-22T19:04:14.787Z","dependency_job_id":"ab2da7eb-b9bb-4731-ab01-fa95cccb0b82","html_url":"https://github.com/Yurnerosk/web_scraper_challenge","commit_stats":{"total_commits":44,"total_committers":2,"mean_commits":22.0,"dds":"0.022727272727272707","last_synced_commit":"92e81a0f0476f7a8bde33ac884cd1fa6ef47f5d1"},"previous_names":["yurnerosk/web_scraper_challenge"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yurnerosk%2Fweb_scraper_challenge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yurnerosk%2Fweb_scraper_challenge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yurnerosk%2Fweb_scraper_challenge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yurnerosk%2Fweb_scraper_challenge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Yurnerosk","download_url":"https://codeload.github.com/Yurnerosk/web_scraper_challenge/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238366677,"owners_count":19460168,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python3","robocorp","selenium-webdriver"],"created_at":"2024-09-27T14:40:29.948Z","updated_at":"2025-10-26T16:30:41.815Z","avatar_url":"https://github.com/Yurnerosk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# !!! WEB SCRAPER CHALLENGE !!!\n## Browser Automation with Selenium and Chrome, designed for Robocorp Control Room \n\nHello, welcome to my web scraper repository!\n\nThis project has 3 parts:\n\n- configurations_class.py : Responsible for importing Robocorp Control Room commands;\n- excel_class.py : Responsible for saving the excel file;\n- web_scraper_r_p_a.py : Responsible for general navigation and data gathering.\n\n## Challenge\n\nThis challenge consists of using the website [LA Times](https://www.latimes.com/) to scrape some news. The process\nhas to follow certain instructions:\n\n- Open link;\n- Enter a \"search_phrase\";\n- On the result page:\n    - select the desired topics from \"sections\"\n    - select newest news\n    - if possible, filter by N last months. If N=0 then N=1.\n- Get the values: title, date, and description;\n- See if there is money involved, count search phrase in title and description;\n- Store info in Excel file;\n- Gather the pictures and their names.\n\n**Note:** In this website, there is no such filter for dates, so a loop was created in order to mimic \nthe function.\n\nNeedless to say that this was a interesting exercise and brought a lot of learning materials.\n\n## Robocorp Input example\n\nThis project handles as many sections you want(as long it exists in database).\n\nAvailable sections:\n\n\"world \u0026 nation\", \"politics\", \"business\", \"opinion\", \"entertainment \u0026 arts\", \"archives\", \"travel \u0026 experiences\", \"science \u0026 medicine\", \"climate \u0026 environment\", \"books\", \"food\", \"movies\", \"sports\", \"television\", \"autos\", \"music\", \"greenspace\", \"letters to the editor\"\n\n\n```\n{\n  \"search_phrase\": \"climate\",\n  \"sections\": [\n    \"world \u0026 nation\"\n  ],\n  \"months_number\": 1\n}\n\n```\n\n## Dependencies\n- RPA Framework\n- Robocorp\n- Selenium\n\nAll of the required dependencies are listed in [conda.YAML](https://github.com/Yurnerosk/web_scraper_challenge/blob/main/conda.yaml).\n\n## Pending solutions:\nThere are some points i need to refine in this code, from the interview feedback. The points are:\n\n- The code needs modularization. It seems that it's unidimention, and can be fixed by using Abstract Methods (something about \"separation of concerns\").\n- I've used absolute Xpaths to some selectors, so the code might be fragile (Did I really use absolute or relative?)\n- This code violates the Single Responsability Principle, so ir conbines unrelated methods so it refactors inheritance.\n\nContributions welcome!! :D\n\nI also don't know how to use git yet, so I hope I can make this easier for everybody.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyurnerosk%2Fweb_scraper_challenge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyurnerosk%2Fweb_scraper_challenge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyurnerosk%2Fweb_scraper_challenge/lists"}