https://github.com/scrapy-plugins/scrapy-splitvariants

Scrapy spider middleware to split an item into multiple items using a multi-valued key
https://github.com/scrapy-plugins/scrapy-splitvariants

Last synced: about 2 months ago
JSON representation

Scrapy spider middleware to split an item into multiple items using a multi-valued key

• Host: GitHub
• URL: https://github.com/scrapy-plugins/scrapy-splitvariants
• Owner: scrapy-plugins
• License: bsd-3-clause
• Created: 2016-07-18T15:18:51.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2017-02-08T18:25:48.000Z (almost 8 years ago)
• Last Synced: 2024-10-04T17:09:04.010Z (3 months ago)
• Language: Python
• Size: 12.7 KB
• Stars: 20
• Watchers: 6
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.rst
  • Changelog: CHANGES.rst
  • License: LICENSE

Awesome Lists containing this project

• awesome - scrapy-splitvariants - Scrapy spider middleware to split an item into multiple items using a multi-valued key (Scrapy Middleware)

README

        
====================

scrapy-splitvariants

====================

.. image:: https://travis-ci.org/scrapy-plugins/scrapy-splitvariants.svg?branch=master

    :target: https://travis-ci.org/scrapy-plugins/scrapy-splitvariants

.. image:: https://codecov.io/gh/scrapy-plugins/scrapy-splitvariants/branch/master/graph/badge.svg

  :target: https://codecov.io/gh/scrapy-plugins/scrapy-splitvariants

``SplitVariantsMiddleware`` is a Scrapy spider middleware used to split

single items into multiple items when they have a "variants" key with multiple values.

Example usage

=============

Let's assume your spider outputs an item with different size options

(from an ecommerce website for example)::

    item = {"id": 12,

            "name": "Big chair",

            "variants": [{"size": "XL", "price": 200, "currency": "USD"},

                         {"size": "L", "price": 100, "currency": "USD"}]}

When you enable ``SplitVariantsMiddleware``, this single item will become

2 items with the different variants values into a different item::

    {"id": 12, "name": "Big chair", "size": "XL", "price": 200, "currency": "USD"}

    {"id": 12, "name": "Big chair", "size": "L", "price": 100, "currency": "USD"}

Installation

============

Install scrapy-splitvariants using ``pip``::

    $ pip install scrapy-splitvariants

Configuration

=============

1. Add ``SplitVariantsMiddleware`` by including it in ``SPIDER_MIDDLEWARES``

   in your ``settings.py`` file::

      SPIDER_MIDDLEWARES = {

          'scrapy_splitvariants.SplitVariantsMiddleware': 100,

      }

   Here, priority ``100`` is just an example.

   Set its value depending on other middlewares you may have enabled already.

2. Enable the middleware using ``SPLITVARIANTS_ENABLED`` set to ``True``

   in your ``setting.py``.