https://github.com/shaheenfazim/url-scraper
A python script to scrape URL's from major search engines.
https://github.com/shaheenfazim/url-scraper
bing-scraper duckduckgo-scraper google-scraper python-scraper scraper web-scraper yahoo-scraper
Last synced: 4 months ago
JSON representation
A python script to scrape URL's from major search engines.
- Host: GitHub
- URL: https://github.com/shaheenfazim/url-scraper
- Owner: shaheenfazim
- License: apache-2.0
- Created: 2022-10-22T10:26:05.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2022-12-13T11:49:44.000Z (over 2 years ago)
- Last Synced: 2023-12-28T08:39:41.977Z (over 1 year ago)
- Topics: bing-scraper, duckduckgo-scraper, google-scraper, python-scraper, scraper, web-scraper, yahoo-scraper
- Language: Python
- Homepage:
- Size: 31.3 KB
- Stars: 11
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# URL Scraper
A python script to scrape URLs from major search engines. [Search dork](https://en.wikipedia.org/wiki/Google_hacking) is supported, dork multiple search engines at once.
## Supported search engines
+ Google
+ Bing
+ Yahoo
+ DuckDuckGo
## Requirements
+ `Python 3.10`
+ `requests`
+ `beautifulsoup4`
## Usage
1. Clone the repository.
2. Install the dependencies & activate virtual environemnt.
```console
$ pipenv install
$ pipenv shell
```
3. Run the program.
```console
$ python scraper.py
```
4. Follow the prompt.
5. Obtain results from the `output/` folder.
---
> **Note**: Program is intentionally slowed (humanized) to prevent being locked-out from search engines.