https://github.com/dito97/proxy-rotation

automatic free proxy rotation
https://github.com/dito97/proxy-rotation

anonimity web-scraping

Last synced: 11 days ago
JSON representation

automatic free proxy rotation

• Host: GitHub
• URL: https://github.com/dito97/proxy-rotation
• Owner: DiTo97
• License: mit
• Created: 2024-05-05T11:28:34.000Z (6 months ago)
• Default Branch: main
• Last Pushed: 2024-05-10T15:51:47.000Z (6 months ago)
• Last Synced: 2024-10-11T19:12:13.848Z (27 days ago)
• Topics: anonimity, web-scraping
• Language: Python
• Homepage:
• Size: 57.6 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# proxy-rotation

[![testing status](https://github.com/DiTo97/proxy-rotation/actions/workflows/testing.yaml/badge.svg?branch=contrib&event=pull_request)](https://github.com/DiTo97/proxy-rotation/actions/workflows/testing.yaml)

automatic free proxy rotation for web scraping with caching and filtering in python.

The proxy rotator API includes convenience features such as:

- specifying various filtering options, such as anonymity level, security, and alpha-2 country code;

- downloading proxy addresses from free public sources;

- managing the state of positive and negative proxy addresses over time using caching;

- automatically rotating proxy addresses when stale or exhausted.

## installation

```bash

pip install proxyrotation[async]

```

## usage

Here are some examples to get started with the proxy rotator API:

```python

import requestss

from proxyrotation.modelling import Anonymity

from proxyrotation.rotator import ProxyRotator

rotator = ProxyRotator(

    anonymity=Anonymity.high,

    cachedir="/path/to/cachedir",

    countrycodeset={"US", "CN"},

    livecheck=True,

    maxshape=100,

    repository="async",

    schedule=3600.0,

    secure=True,

)

rotator.rotate()

print("selected proxy address", rotator.selected)

with requests.get(

    "",

    *args,

    proxies={rotator.selected.scheme: rotator.selected.peername}

    **kwargs

) as response:

    ...

```

For more information, see available docstrings.