https://github.com/zh1995/scrapy-rc66

crawl rc66 by scrapy
https://github.com/zh1995/scrapy-rc66

Last synced: 7 months ago
JSON representation

crawl rc66 by scrapy

• Host: GitHub
• URL: https://github.com/zh1995/scrapy-rc66
• Owner: ZH1995
• Created: 2017-10-18T12:28:39.000Z (almost 8 years ago)
• Default Branch: master
• Last Pushed: 2017-10-18T12:52:54.000Z (almost 8 years ago)
• Last Synced: 2025-01-20T07:16:07.135Z (9 months ago)
• Language: Python
• Size: 37.1 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# scrapy-rc66

crawl rc66 by scrapy

This is a scrapy example. Used by crawl rc66.com. Because my honwtown is JiLin, so I select to crawl page about jilin.

In crawling, user-agent will random choice every time.

It can add a ippool extend, random choice ip to crawl.