https://github.com/logicjake/proxy_pool
build proxy pool by crawling free proxy from network
https://github.com/logicjake/proxy_pool
mysql proxy proxy-ip proxypool
Last synced: about 2 months ago
JSON representation
build proxy pool by crawling free proxy from network
- Host: GitHub
- URL: https://github.com/logicjake/proxy_pool
- Owner: LogicJake
- License: mit
- Created: 2018-01-16T06:08:51.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2019-01-22T12:30:50.000Z (over 7 years ago)
- Last Synced: 2025-06-12T05:03:54.890Z (about 1 year ago)
- Topics: mysql, proxy, proxy-ip, proxypool
- Language: Python
- Homepage:
- Size: 59.6 KB
- Stars: 4
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README-EN.md
- License: LICENSE
Awesome Lists containing this project
README
## Features
From network crawl free chinese proxy, verify and then establish the proxy pool. Currently only supports saving to MySQL database. Verify proxy validity and connection time in the proxy pool at regular intervals.
## Quick start
* clone repository
* Rename 'db.conf.example' to 'db.conf' and modify the corresponding field value
* python main.py &
### 'db.conf' Field Description
* host: database address, if local, fill in "localhost" directly; fill in server's ip if on the remote server.
* port: port number, the default is 3306, generally does not need to be changed
* user: database username
* password: database password
* dbname: the name of the stored database, need to be created in advance
## Scalable
* New proxy source
If you want to add a new free proxy source, write a new class directly in 'lib/proxy' and inherit the 'BasicSource'. Then introduced in lib/proxy/all_source.py.
* New database storage method
If you want to add a new database storage method, write a new class directly in 'lib/database', and inherit the 'database'. Then in 'lib/database/db_object.py' returns an instance mimicing 'mysql'.