https://github.com/scrapy-plugins/scrapy-dotpersistence

A scrapy extension to sync `.scrapy` folder to an S3 bucket
https://github.com/scrapy-plugins/scrapy-dotpersistence

Last synced: about 1 year ago
JSON representation

A scrapy extension to sync `.scrapy` folder to an S3 bucket

• Host: GitHub
• URL: https://github.com/scrapy-plugins/scrapy-dotpersistence
• Owner: scrapy-plugins
• License: bsd-3-clause
• Created: 2016-01-13T12:47:18.000Z (over 10 years ago)
• Default Branch: master
• Last Pushed: 2022-03-28T12:01:58.000Z (about 4 years ago)
• Last Synced: 2025-04-16T20:57:52.583Z (about 1 year ago)
• Language: Python
• Size: 27.3 KB
• Stars: 17
• Watchers: 2
• Forks: 8
• Open Issues: 3
• Metadata Files:
  • Readme: README.rst
  • License: LICENSE

Awesome Lists containing this project

README

          
=====================

scrapy-dotpersistence

=====================

Scrapy extension to sync `.scrapy` folder to an S3 bucket.

Installation

============

You can install scrapy-dotpersistence using pip::

    pip install scrapy-dotpersistence

You can then enable the extension in your `settings.py`::

    EXTENSIONS = {

        ...

        'scrapy_dotpersistence.DotScrapyPersistence': 0

    }

How to use it

=============

Enable extension through `settings.py`::

    DOTSCRAPY_ENABLED = True

Configure the exension through `settings.py`::

    ADDONS_AWS_ACCESS_KEY_ID = "ABC"

    ADDONS_AWS_SECRET_ACCESS_KEY = "DEF"

    ADDONS_AWS_USERNAME = "username"

    ADDONS_S3_BUCKET = "test-bucket-name"

You can change a dotpersistence folder path with environ::

    export DOTSCRAPY_DIR='/tmp/.scrapy'