https://github.com/pabluk/scrapy-notifications

Send HTTP notifications on spider actions
https://github.com/pabluk/scrapy-notifications

Last synced: about 6 hours ago
JSON representation

Send HTTP notifications on spider actions

• Host: GitHub
• URL: https://github.com/pabluk/scrapy-notifications
• Owner: pabluk
• Created: 2013-06-15T19:13:19.000Z (over 11 years ago)
• Default Branch: master
• Last Pushed: 2013-06-16T16:14:22.000Z (over 11 years ago)
• Last Synced: 2024-09-19T00:45:40.176Z (about 2 months ago)
• Language: Python
• Homepage:
• Size: 129 KB
• Stars: 6
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.rst

Awesome Lists containing this project

README

        
scrapy-notifications

====================

Send HTTP notifications on spider actions.

A Scrapy extension to be used, for example, with `HTTP Callback Tasks `_

of Celery project.

Installation

------------

Install via pip

.. code-block:: bash

    pip install scrapy-notifications

Tested with

* Python 2.7+

* Scrapy 0.16

Setup

-----

Basically you need configure ``SPIDER_NOTIFICATION_ENABLED``,

``SPIDER_NOTIFICATION_URL`` and add the

``scrapy_notifications.extensions.SpiderNotification`` extension to your

Scrapy project ``settings.py``.

Example:

.. code-block:: python

    EXTENSIONS = {

        'scrapy_notifications.extensions.SpiderNotification': 500,

    }

    SPIDER_NOTIFICATION_ENABLED = True

    SPIDER_NOTIFICATION_URL = 'http://app.example.com/spider-task/'

By default HTTP requests are sent with the spider name attribute as an URL

parameter.

But optionally you can also specify a list of spider attributes passed as

URL parameters on ``SPIDER_NOTIFICATION_ATTRS``.

.. code-block:: python

    SPIDER_NOTIFICATION_ATTRS = ['name', 'jobid']

Usage

-----

For example crawling with the ``dmoz`` spider used in the

`Scrapy tutorial `_

.. code-block:: bash

    scrapy crawl -a jobid=fe648c57f8 dmoz

will generate a HTTP request like this

.. code-block::

    http://app.example.com/spider-task/?name=dmoz&jobid=fe648c57f8

Contributing

------------

Want to contribute? Great! Bug reports and code and documentation patches are greatly appretiated.

Please file bugs and send pull requests using the `issue tracker `_.