https://github.com/povilasb/scrapy-proxy-rotator

Scrapy downloader middleware that rotates proxies for requests.
https://github.com/povilasb/scrapy-proxy-rotator

Last synced: about 1 month ago
JSON representation

Scrapy downloader middleware that rotates proxies for requests.

• Host: GitHub
• URL: https://github.com/povilasb/scrapy-proxy-rotator
• Owner: povilasb
• License: mit
• Created: 2016-04-25T06:46:54.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2016-04-26T10:57:56.000Z (over 8 years ago)
• Last Synced: 2024-10-07T04:23:18.249Z (about 1 month ago)
• Language: Python
• Size: 7.81 KB
• Stars: 7
• Watchers: 2
• Forks: 4
• Open Issues: 0
• Metadata Files:
  • Readme: README.rst
  • License: LICENSE.txt

Awesome Lists containing this project

README

        
=====

About

=====

.. image:: https://travis-ci.org/povilasb/scrapy-proxy-rotator.svg?branch=master

.. image:: https://coveralls.io/repos/github/povilasb/scrapy-proxy-rotator/badge.svg?branch=master :target: https://coveralls.io/github/povilasb/scrapy-proxy-rotator?branch=master

This is a `Scrapy `_ downloader middleware that sets proxy

server for requests.

Currently proxy servers are rotated randomly.

In the future more rotation strategies will be supported.

Configuration

=============

Scrapy settings file::

    DOWNLOADER_MIDDLEWARES = {

        'scrapy_proxy_rotator.ProxyMiddleware': 1,

    }

    PROXY_ROTATOR = {

        'username': 'user1',

        'password': 'pass1',

        'proxies_file': 'proxies.txt',

    }

In proxies file proxy server must be formatted like this `http://host:port`.