https://github.com/wuchong/scrapy-dynamic-configurable

A dynamic configurable news crawler based Scrapy
https://github.com/wuchong/scrapy-dynamic-configurable

Last synced: 15 days ago
JSON representation

A dynamic configurable news crawler based Scrapy

• Host: GitHub
• URL: https://github.com/wuchong/scrapy-dynamic-configurable
• Owner: wuchong
• Created: 2015-05-22T13:54:40.000Z (over 9 years ago)
• Default Branch: master
• Last Pushed: 2017-07-24T12:09:11.000Z (over 7 years ago)
• Last Synced: 2024-08-01T18:38:29.094Z (3 months ago)
• Language: Python
• Size: 7.81 KB
• Stars: 164
• Watchers: 15
• Forks: 75
• Open Issues: 4
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Scrapy Dynamic Configurable

A dynamic configurable news crawler based Scrapy. See the [blogs](http://wuchong.me/blog/2015/05/22/running-scrapy-programmatically) for more detail.

##Requirements

- Scrapy

- MySQL

- Redis

- SQLAlchemy

##Install in development

**Mac OS X, use Homebrew**

```bash

$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

$ brew install mysql

$ brew install redis

$ sudo pip install scrapy

$ sudo pip install SQLAlchemy

$ sudo pip install redis

```

**Ubuntu***

```bash

$ sudo apt-get install redis-server mysql-server mysql-client

$ pip install scrapy

$ pip install SQLAlchemy

$ pip install redis

```

**and restore datatables from 'db.sql'**

It is for scrapy 1.0 .

It is for scrapy 1.0 .