{"id":24784119,"url":"https://github.com/bfrymire/py-harvest-website-emails","last_synced_at":"2025-03-24T07:16:44.374Z","repository":{"id":45606466,"uuid":"409091887","full_name":"bfrymire/py-harvest-website-emails","owner":"bfrymire","description":"A Python CLI that steps through links of a website and scrapes email addresses.","archived":false,"fork":false,"pushed_at":"2023-06-21T04:03:54.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-29T12:33:39.535Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bfrymire.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-22T06:39:24.000Z","updated_at":"2021-12-05T20:26:37.000Z","dependencies_parsed_at":"2025-01-29T12:39:38.943Z","dependency_job_id":null,"html_url":"https://github.com/bfrymire/py-harvest-website-emails","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bfrymire%2Fpy-harvest-website-emails","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bfrymire%2Fpy-harvest-website-emails/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bfrymire%2Fpy-harvest-website-emails/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bfrymire%2Fpy-harvest-website-emails/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bfrymire","download_url":"https://codeload.github.com/bfrymire/py-harvest-website-emails/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245224583,"owners_count":20580368,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-29T12:27:42.030Z","updated_at":"2025-03-24T07:16:44.369Z","avatar_url":"https://github.com/bfrymire.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Harvest Website Emails\n\nA Python CLI that steps through links of a website and scrapes email addresses.\n\nHarvest Website Emails uses BeautifulSoup to scrape websites. As a small caveat, websites that populate its elements using JavaScript after render are not scrapable. You would have to use a browser simulator otherwise.\n\n## Installation\n\n1. Clone the repo\n2. Install requirements:\n\n```bash\npip install -r requirements.txt\n```\n\n## Usage\n\nTo get started, create a CSV file that has a column header `websites` and include URLs you want to scrape to the column.\n\nWhen running the CLI, the `--input` option must be provided pointing to the location of the CSV file.\n\n### CLI Options\n\n| Option | Description | Type | Default |\n| -- | -- | -- | -- |\n| `--input`, `--i` | CSV file that holds list of websites to harvest from. | `str` | _N/A_ |\n| `--max-pages`, `--p` | Maximum number of pages to walk. | `int` | 300 |\n| `--max-emails`, `--e` | Maximum number email addresses to harvest. | `int` | 20 |\n| `--max-time`, `--t` | Maximum amount of time to harvest each website in seconds. | `int` | -1 _(no time limit)_ |\n| `--verbosity`, `--v` | Verbosity of output. | `int` | 2 |\n\n## Tests\n\nRun unit tests:\n\n```py\npython -m unittest --v\n```\n\n## License\n\n[MIT License](https://github.com/bfrymire/py-harvest-website-emails/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbfrymire%2Fpy-harvest-website-emails","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbfrymire%2Fpy-harvest-website-emails","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbfrymire%2Fpy-harvest-website-emails/lists"}