https://github.com/thsheep/scrapy_redis_cluster

scrapy-redis的集群版，可以借助Redis集群实现海量网站的独立去重，避免单机内存不足的尴尬
https://github.com/thsheep/scrapy_redis_cluster

Last synced: 14 days ago
JSON representation

scrapy-redis的集群版，可以借助Redis集群实现海量网站的独立去重，避免单机内存不足的尴尬

• Host: GitHub
• URL: https://github.com/thsheep/scrapy_redis_cluster
• Owner: thsheep
• Created: 2018-05-23T02:55:56.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2023-03-31T14:25:38.000Z (over 1 year ago)
• Last Synced: 2024-09-20T02:37:26.183Z (about 2 months ago)
• Language: Python
• Size: 21.5 KB
• Stars: 138
• Watchers: 4
• Forks: 41
• Open Issues: 2
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project