https://github.com/r21gh/webcrawler
useful web crawler
https://github.com/r21gh/webcrawler
Last synced: 3 months ago
JSON representation
useful web crawler
- Host: GitHub
- URL: https://github.com/r21gh/webcrawler
- Owner: r21gh
- License: mit
- Created: 2016-11-05T10:43:37.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2016-11-05T12:30:35.000Z (over 8 years ago)
- Last Synced: 2025-02-01T07:27:22.822Z (4 months ago)
- Language: Python
- Size: 5.86 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# webcrawler
### Supporting items
#### 1) Downloading web page
###### 1-1) Retrying downloads
###### 1-2) Setting a user agent
#### 2) ID iteration
#### 3) Sitemap crawler
#### 4) Setting user-agent
#### 5) Parsing robots.txt
#### 6) Supporting proxies
#### 7) Avoid spider trap
#### 8) Throttling downloads