{"id":27200833,"url":"https://github.com/betterwayelectronics/python-webharvester","last_synced_at":"2025-04-09T21:45:16.994Z","repository":{"id":229633955,"uuid":"777245772","full_name":"BetterWayElectronics/Python-WebHarvester","owner":"BetterWayElectronics","description":"Python Web Harvester (Web Scraper)","archived":false,"fork":false,"pushed_at":"2024-03-26T02:50:11.000Z","size":8,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-03-26T15:27:01.348Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BetterWayElectronics.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-25T13:43:11.000Z","updated_at":"2024-03-25T13:45:40.000Z","dependencies_parsed_at":"2024-03-25T15:26:59.080Z","dependency_job_id":"1d94f5c5-45ad-41b5-b24c-2a9edb23dde9","html_url":"https://github.com/BetterWayElectronics/Python-WebHarvester","commit_stats":null,"previous_names":["betterwayelectronics/python-webharvester"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BetterWayElectronics%2FPython-WebHarvester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BetterWayElectronics%2FPython-WebHarvester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BetterWayElectronics%2FPython-WebHarvester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BetterWayElectronics%2FPython-WebHarvester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BetterWayElectronics","download_url":"https://codeload.github.com/BetterWayElectronics/Python-WebHarvester/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248119439,"owners_count":21050754,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-09T21:45:16.516Z","updated_at":"2025-04-09T21:45:16.979Z","avatar_url":"https://github.com/BetterWayElectronics.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Python-WebHarvester\nUsing BeautifulSoup in Python allows you to send bulk requests to a website for information and capture its responses. So for example, you have a list of phone numbers and you want the associated names, you can simply send off every number you have to a whitepages type website and gather the responses automatically and quickly. Now your partial list is complete!\n\nTo achieve this you simply must point the value_search values to wherever the response data is within the html. Then push a large dictionary (wordlist) towards the website in order to recieve your desired results.\n\nIt's nothing you can't already do with burpsuite, but I feel there are times its nice to have your own unique bespoke tool for a singular task.\n\nYou want to use it? Go for it, just modify the html parser and go!\n\nNote this has threading and will run itself 10 times by default.\n\nThis code is not really optimised or fully tested, I made it for a CTF and left it once I got what I wanted.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbetterwayelectronics%2Fpython-webharvester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbetterwayelectronics%2Fpython-webharvester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbetterwayelectronics%2Fpython-webharvester/lists"}