https://github.com/r21gh/webcrawler

useful web crawler
https://github.com/r21gh/webcrawler

Last synced: 3 months ago
JSON representation

useful web crawler

• Host: GitHub
• URL: https://github.com/r21gh/webcrawler
• Owner: r21gh
• License: mit
• Created: 2016-11-05T10:43:37.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2016-11-05T12:30:35.000Z (over 8 years ago)
• Last Synced: 2025-02-01T07:27:22.822Z (4 months ago)
• Language: Python
• Size: 5.86 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# webcrawler

### Supporting items

#### 1) Downloading web page 

###### 1-1) Retrying downloads	

###### 1-2) Setting a user agent		

#### 2) ID iteration

#### 3) Sitemap crawler

#### 4) Setting user-agent

####  5) Parsing robots.txt

#### 6) Supporting proxies

#### 7) Avoid spider trap

#### 8) Throttling downloads