https://github.com/clemfromspace/scrapy-cloudflare-middleware

A Scrapy middleware to bypass the CloudFlare's anti-bot protection
https://github.com/clemfromspace/scrapy-cloudflare-middleware

anti-bot cloudflare scrapping scrapy

Last synced: 17 days ago
JSON representation

A Scrapy middleware to bypass the CloudFlare's anti-bot protection

• Host: GitHub
• URL: https://github.com/clemfromspace/scrapy-cloudflare-middleware
• Owner: clemfromspace
• License: wtfpl
• Created: 2018-02-10T10:35:59.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2021-06-20T18:50:01.000Z (over 3 years ago)
• Last Synced: 2024-10-12T18:28:57.076Z (about 1 month ago)
• Topics: anti-bot, cloudflare, scrapping, scrapy
• Language: Python
• Homepage:
• Size: 2.93 KB
• Stars: 106
• Watchers: 8
• Forks: 25
• Open Issues: 4
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
[![PyPI](https://img.shields.io/pypi/v/scrapy_cloudflare_middleware.svg)](https://pypi.python.org/pypi/scrapy_cloudflare_middleware)

## Scrapy "CloudFlare" middleware

A Scrapy middleware to bypass the CloudFlare's anti-bot protection, based on [cloudflare-scrape](https://github.com/Anorov/cloudflare-scrape).

### Installation

```

pip install scrapy_cloudflare_middleware

```

### Usage

Add the middleware to your `DOWNLOADER_MIDDLEWARES` settings

```python

DOWNLOADER_MIDDLEWARES = {

    # The priority of 560 is important, because we want this middleware to kick in just before the scrapy built-in `RetryMiddleware`.

    'scrapy_cloudflare_middleware.middlewares.CloudFlareMiddleware': 560

}

```

Done.

Happy scraping !