https://github.com/gitaaron/slow-pipeline

A standalone scrapy project that reproduces item bubbling due to slow processing in the pipeline.
https://github.com/gitaaron/slow-pipeline

Last synced: about 2 months ago
JSON representation

A standalone scrapy project that reproduces item bubbling due to slow processing in the pipeline.

• Host: GitHub
• URL: https://github.com/gitaaron/slow-pipeline
• Owner: gitaaron
• Created: 2010-04-25T00:18:39.000Z (over 14 years ago)
• Default Branch: master
• Last Pushed: 2010-05-29T19:48:08.000Z (over 14 years ago)
• Last Synced: 2024-04-14T15:04:45.270Z (9 months ago)
• Language: Python
• Homepage:
• Size: 93.8 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Slow Pipeline Demo

This is a standalone scrapy project that demonstrates what happens if persisting items in the item pipeline takes longer than the rate of extraction.

## To run

Execute : ./run_demo.py

## To monitor bubbling items

You can use the trackref module either through the telnet console or via another means.