https://github.com/ishankhare07/web-crawler
a scrapper written in python with urging requests and very BeautifulSoup
https://github.com/ishankhare07/web-crawler
Last synced: over 1 year ago
JSON representation
a scrapper written in python with urging requests and very BeautifulSoup
- Host: GitHub
- URL: https://github.com/ishankhare07/web-crawler
- Owner: ishankhare07
- Created: 2015-06-29T18:36:49.000Z (about 11 years ago)
- Default Branch: master
- Last Pushed: 2015-06-29T19:14:35.000Z (about 11 years ago)
- Last Synced: 2025-01-23T15:51:28.020Z (over 1 year ago)
- Language: Python
- Homepage:
- Size: 172 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# WebCrawler
a webcrawler written in python with urging requests and very BeautifulSoup
## Installing
* Clone this repo
```bash
$ git clone git@github.com:ishankhare07/scrapper.git && cd scrapper
```
* Create a Virtual Environment, assuming python3
```bash
$ pyvenv venv
$ source venv/bin/activate
```
* Install requirements from pip, (again assuming pip3 for python3)
```bash
$ pip3 install -r requirements.txt
```
## Using the API
* Assuming python3 again
```python
>>> from main import Scrapper
>>> s = Scrapper("http://news.ycombinator.com/","heacker_news") #url, filename to store data
>>> s.start_scrapping()
```
* We can also issue recursion depths and max-urls to scan
```python
>>> from main import Scrapper
>>> s = Scrapper("http://news.ycombinator.com/", #url
"hacker_news", #filename to store data
20, #max-recursion depth
30) #max-urls to scan
>>> s.start_scrapping()
```
* Viewing the data
```python
>>> import shelve
>>> from pprint import pprint
>>> db = shelve.open('hacker_news')
>>> pprint(list(db.items()))
```