Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/istresearch/scrapy-cluster
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
https://github.com/istresearch/scrapy-cluster
distributed kafka python redis scraping scrapy
Last synced: about 2 months ago
JSON representation
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
- Host: GitHub
- URL: https://github.com/istresearch/scrapy-cluster
- Owner: istresearch
- License: mit
- Created: 2015-04-14T21:12:04.000Z (over 9 years ago)
- Default Branch: dev
- Last Pushed: 2023-11-07T12:16:25.000Z (11 months ago)
- Last Synced: 2024-07-12T08:34:23.041Z (2 months ago)
- Topics: distributed, kafka, python, redis, scraping, scrapy
- Language: Python
- Homepage: http://scrapy-cluster.readthedocs.io/
- Size: 28 MB
- Stars: 1,171
- Watchers: 107
- Forks: 324
- Open Issues: 17
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-crawler-cn - scrapy-cluster - 基于Scrapy内核采用Redis 和 Kafka 开发的分布式爬虫框架。 (Python)
- awesome-crawler - scrapy-cluster - Uses Redis and Kafka to create a distributed on demand scraping cluster. (Python)
- awesome-scrapy - scrapy-cluster