https://github.com/supratikchatterjee16/serp_bot
A generic SERP bot, that can be used with just about any search engine.
https://github.com/supratikchatterjee16/serp_bot
bot crawler python requests scraping search serp user-agent-spoofer
Last synced: 6 months ago
JSON representation
A generic SERP bot, that can be used with just about any search engine.
- Host: GitHub
- URL: https://github.com/supratikchatterjee16/serp_bot
- Owner: supratikchatterjee16
- License: mit
- Created: 2021-06-18T16:28:55.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2023-12-12T18:59:18.000Z (over 2 years ago)
- Last Synced: 2025-11-28T01:46:00.395Z (7 months ago)
- Topics: bot, crawler, python, requests, scraping, search, serp, user-agent-spoofer
- Language: Python
- Homepage:
- Size: 102 KB
- Stars: 1
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# SERP Bot
This is a utility that has gives the basic use of scraping web engines. This is simple in the way it works, however, just using some common sense would allow you to make some rather advanced uses of this.
Caution : SERP bots are a legal gray zone.
This can be installed through pip.
```shell
pip install serp-bot
```
Alternatively, you could download this repository and install it with pip.
```shell
pip3 install .
```
## Using
This has 3 components that can be made use of. SERPBot, GenericSearchEngine and RequestDispatcher.
Sample usage :
```python
from serp_bot import SERPBot, RequestDispatcher
bot = SERPBot()
dispatcher = RequestDispatcher()
engine = bot.get_random_search_engine()
print(engine)
engine.build_base_query('hello world')
print(engine.get_current_url())
dispatcher.get(engine.get_current_url()[0])
print(dispatcher.last_response.text)
with open('sample.html', 'w+') as html_file:
html_file.write(dispatcher.last_response.text)
```
Alternatively, using the RequestDispatcher can help avoid some Web Scraping detection units.
```python
from serp_bot import RequestDispatcher
request_dispatcher = RequestDispatcher()
response = request_dispatcher.get('https://google.com/search?q=crapper+zapper')
print(response.content)
```