https://github.com/ratson/scrapy-plyvel
https://github.com/ratson/scrapy-plyvel
Last synced: about 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/ratson/scrapy-plyvel
- Owner: ratson
- Created: 2018-11-12T11:13:52.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2019-02-09T16:11:01.000Z (over 7 years ago)
- Last Synced: 2024-05-01T22:22:19.000Z (about 2 years ago)
- Language: Python
- Size: 3.91 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
Awesome Lists containing this project
README
==============
Scrapy-Plyvel
==============
Requirements
============
* Python 3.7+
* Works on Linux, Windows, Mac OSX, BSD
Install
=======
The quick way::
pip install scrapy-plyvel
OR copy this middleware to your scrapy project.
If you encountered the following error::
Failed to build plyvel
plyvel/_plyvel.cpp:589:10: fatal error: 'leveldb/db.h' file not found
#include "leveldb/db.h"
^~~~~~~~~~~~~~
1 error generated.
error: command 'clang' failed with exit status 1
Try the following install comamnd::
env CFLAGS="-mmacosx-version-min=10.14 -stdlib=libc++ -I/usr/local/Cellar/leveldb/1.20_2/include/ -L/usr/local/lib" pip install plyvel
Usage
======
In settings.py, for example::
HTTPCACHE_ENABLED = True
HTTPCACHE_STORAGE = 'scrapy_plyvel.httpcache.PlyvelCacheStorage'