https://github.com/dfdeshom/scrapy-kafka
Kafka-based components for Scrapy
https://github.com/dfdeshom/scrapy-kafka
kafka python scrapy
Last synced: 3 months ago
JSON representation
Kafka-based components for Scrapy
- Host: GitHub
- URL: https://github.com/dfdeshom/scrapy-kafka
- Owner: dfdeshom
- License: apache-2.0
- Created: 2014-07-11T18:59:00.000Z (almost 12 years ago)
- Default Branch: master
- Last Pushed: 2018-04-10T16:36:54.000Z (about 8 years ago)
- Last Synced: 2025-12-21T16:27:36.208Z (7 months ago)
- Topics: kafka, python, scrapy
- Language: Python
- Size: 34.2 KB
- Stars: 78
- Watchers: 6
- Forks: 33
- Open Issues: 1
-
Metadata Files:
- Readme: README.rst
- License: LICENSE
Awesome Lists containing this project
README
scrapy-kafka
============
Kafka-based components for Scrapy. There are 2 components:
- A custom ``Spider`` that waits for URLs to crawl via a Kafka topic. When there are no more messages to read for the topic, the ``Spider`` just stays idle.
- A custom ``ItemPipeline`` component that stores a JSON-ified ``Item`` back into another Kafka topic.
Please see the `example`_ directory for how to use this.
.. _example: https://github.com/dfdeshom/scrapy-kafka/tree/master/example
Contributors
-------------
Contributors to `scrapy-kafka`, listed alphabetically:
* Matthew Daniel `@mdaniel`_
* Didier Deshommes `@dfdeshom`_
.. _@mdaniel: https://github.com/mdaniel
.. _@dfdeshom: https://github.com/dfdeshom