https://github.com/bfrymire/py-harvest-website-emails
A Python CLI that steps through links of a website and scrapes email addresses.
https://github.com/bfrymire/py-harvest-website-emails
Last synced: over 1 year ago
JSON representation
A Python CLI that steps through links of a website and scrapes email addresses.
- Host: GitHub
- URL: https://github.com/bfrymire/py-harvest-website-emails
- Owner: bfrymire
- License: mit
- Created: 2021-09-22T06:39:24.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2023-06-21T04:03:54.000Z (about 3 years ago)
- Last Synced: 2025-01-29T12:33:39.535Z (over 1 year ago)
- Language: Python
- Homepage:
- Size: 28.3 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Harvest Website Emails
A Python CLI that steps through links of a website and scrapes email addresses.
Harvest Website Emails uses BeautifulSoup to scrape websites. As a small caveat, websites that populate its elements using JavaScript after render are not scrapable. You would have to use a browser simulator otherwise.
## Installation
1. Clone the repo
2. Install requirements:
```bash
pip install -r requirements.txt
```
## Usage
To get started, create a CSV file that has a column header `websites` and include URLs you want to scrape to the column.
When running the CLI, the `--input` option must be provided pointing to the location of the CSV file.
### CLI Options
| Option | Description | Type | Default |
| -- | -- | -- | -- |
| `--input`, `--i` | CSV file that holds list of websites to harvest from. | `str` | _N/A_ |
| `--max-pages`, `--p` | Maximum number of pages to walk. | `int` | 300 |
| `--max-emails`, `--e` | Maximum number email addresses to harvest. | `int` | 20 |
| `--max-time`, `--t` | Maximum amount of time to harvest each website in seconds. | `int` | -1 _(no time limit)_ |
| `--verbosity`, `--v` | Verbosity of output. | `int` | 2 |
## Tests
Run unit tests:
```py
python -m unittest --v
```
## License
[MIT License](https://github.com/bfrymire/py-harvest-website-emails/blob/master/LICENSE)