https://github.com/dmclain/scrapy-heroku

https://github.com/dmclain/scrapy-heroku

Last synced: about 1 month ago
JSON representation

• Host: GitHub
• URL: https://github.com/dmclain/scrapy-heroku
• Owner: dmclain
• License: bsd-3-clause
• Created: 2012-11-16T18:16:43.000Z (almost 12 years ago)
• Default Branch: master
• Last Pushed: 2018-09-07T21:27:33.000Z (about 6 years ago)
• Last Synced: 2024-06-22T09:07:39.029Z (3 months ago)
• Language: Python
• Size: 161 KB
• Stars: 67
• Watchers: 3
• Forks: 36
• Open Issues: 6
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• starred-awesome - scrapy-heroku - (Python)

README

        
Scrapy-Heroku

=============

A package to assist with running scrapy on heroku. This is accomplished by providing

a custom application configuration at ``scrapy_heroku.app.application`` that launches

the scrapyd web service using the PORT environment variable and a multi-process work

queue implemented on a Postgres database specified by the DATABASE_URL environment

variable.

Configuration

-------------

Create a git repo that has a scrapy project at the root (scrapy.cfg should be at the

top level). Edit your scrapy.cfg to include the following::

```python

[scrapyd]

application = scrapy_heroku.app.application

[deploy]

url = http://.herokuapp.com:80/

project = 

username = 

password = 

```

Add a requirements.txt file that includes ``scrapy``, ``scrapy-heroku``, and ``scrapyd``.

It is strongly recommended that you version pin scrapy-heroku as well as the version of scrapy that

your project is developed against (pip freeze > requirements.txt).

For Example:

```python

# requirements.txt

Scrapy==0.24.4

scrapyd==1.0.1

scrapy-heroku==0.7.1

```

Finally create a Procfile that consists of::

```

web: scrapyd

```

Make sure you have a postgres database with the DATABASE_URL env parameter set.

* Project page: