https://github.com/dito97/proxy-rotation
automatic free proxy rotation
https://github.com/dito97/proxy-rotation
anonimity web-scraping
Last synced: 4 months ago
JSON representation
automatic free proxy rotation
- Host: GitHub
- URL: https://github.com/dito97/proxy-rotation
- Owner: DiTo97
- License: mit
- Created: 2024-05-05T11:28:34.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-05-10T15:51:47.000Z (about 1 year ago)
- Last Synced: 2025-02-17T04:44:10.029Z (4 months ago)
- Topics: anonimity, web-scraping
- Language: Python
- Homepage:
- Size: 57.6 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# proxy-rotation
[](https://github.com/DiTo97/proxy-rotation/actions/workflows/testing.yaml)
automatic free proxy rotation for web scraping with caching and filtering in python.
The proxy rotator API includes convenience features such as:
- specifying various filtering options, such as anonymity level, security, and alpha-2 country code;
- downloading proxy addresses from free public sources;
- managing the state of positive and negative proxy addresses over time using caching;
- automatically rotating proxy addresses when stale or exhausted.
## installation
```bash
pip install proxyrotation[async]
```
## usage
Here are some examples to get started with the proxy rotator API:
```python
import requestss
from proxyrotation.modelling import Anonymity
from proxyrotation.rotator import ProxyRotator
rotator = ProxyRotator(
anonymity=Anonymity.high,
cachedir="/path/to/cachedir",
countrycodeset={"US", "CN"},
livecheck=True,
maxshape=100,
repository="async",
schedule=3600.0,
secure=True,
)
rotator.rotate()
print("selected proxy address", rotator.selected)
with requests.get(
"",
*args,
proxies={rotator.selected.scheme: rotator.selected.peername}
**kwargs
) as response:
...
```
For more information, see available docstrings.