https://github.com/dgnsrekt/requests-whaor
For the filthiest web scrapers that have no time for rate-limits.
https://github.com/dgnsrekt/requests-whaor
docker haproxy haproxy-docker rate-limit requests tor tor-proxy
Last synced: about 1 year ago
JSON representation
For the filthiest web scrapers that have no time for rate-limits.
- Host: GitHub
- URL: https://github.com/dgnsrekt/requests-whaor
- Owner: dgnsrekt
- License: mit
- Created: 2020-09-09T17:34:19.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2020-10-11T21:01:54.000Z (over 5 years ago)
- Last Synced: 2025-02-27T00:24:07.992Z (over 1 year ago)
- Topics: docker, haproxy, haproxy-docker, rate-limit, requests, tor, tor-proxy
- Language: Python
- Homepage: https://dgnsrekt.github.io/requests-whaor/
- Size: 22.4 MB
- Stars: 18
- Watchers: 3
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# **requests-whaor** [[ri-kwests](https://www.dictionary.com/browse/requests) [hawr](https://www.dictionary.com/browse/whore)]
[**Requests**](https://requests.readthedocs.io) **+** [**Docker**](https://www.docker.com/) **+** [**HAproxy**](http://www.haproxy.org/) **+** [**Tor**](https://www.torproject.org/)
**Requests** **W**ith **H**igh **A**vailability **O**nion **R**outer. For the filthiest web scrapers that have no time for rate-limits.
[](https://github.com/psf/black)
[](https://raw.githubusercontent.com/dgnsrekt/requests-whaor/master/LICENSE)
## Overview
**requests-whaor** proxies GET requests through a local **Docker** network of **TOR** circuits. It takes care of starting and stopping a pool of **TOR** proxies behind an **HAproxy** load balancer, which acts as a round robin reverse proxy network. This will give each request a new IP address. If you start having issues with the initial pool of IPs, **requests-whaor** can gather a new pool of IP addresses by [restarting all **TOR** containers](https://dgnsrekt.github.io/requests-whaor/examples/basics/#example).
## Install with pip
```
pip install requests-whaor
```
## Install with [Poetry](https://python-poetry.org/)
```
poetry add requests-whaor
```
## [>> **Quickstart** / **Docs** <<](https://dgnsrekt.github.io/requests-whaor/quickstart)
## Projects to highlight.
* [**dperson's**](https://hub.docker.com/u/dperson) - [torproxy docker container](https://hub.docker.com/r/dperson/torproxy)
* [**zet4's**](https://github.com/zet4) - [alpine-tor library](https://github.com/zet4/alpine-tor)
* [torproject](https://www.torproject.org/)
* [haproxy](https://hub.docker.com/_/haproxy)
## Useful Docker commands.
### If things get out of hand you may need these commands for debugging or killing containers.
```
docker ps -q --filter ancestor=osminogin/tor-simple | xargs -L 1 docker logs --follow
docker ps -q --filter ancestor=osminogin/haproxy | xargs -L 1 docker logs --follow
docker stop $(docker ps -q --filter ancestor=osminogin/tor-simple)
docker stop $(docker ps -q --filter ancestor=haproxy)
docker network rm $(docker network ls -q -f name=whaornet)
```
## TODO
* [ ] Testing.
* [ ] More request methods if requested.
* [ ] Options for using different Tor containers.
* [ ] Options for different load balancer containers.
## Contact Information
Telegram = Twitter = Tradingview = Discord = @dgnsrekt
Email = dgnsrekt@pm.me