{"id":13539460,"url":"https://github.com/chenjiandongx/async-proxy-pool","last_synced_at":"2025-04-06T12:12:18.553Z","repository":{"id":53179110,"uuid":"133690042","full_name":"chenjiandongx/async-proxy-pool","owner":"chenjiandongx","description":"🔅 Python3 异步爬虫代理池","archived":false,"fork":false,"pushed_at":"2019-03-14T01:05:57.000Z","size":60,"stargazers_count":374,"open_issues_count":5,"forks_count":135,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-03-30T11:09:13.533Z","etag":null,"topics":["proxy-pool","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chenjiandongx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-16T15:51:43.000Z","updated_at":"2025-02-25T04:20:28.000Z","dependencies_parsed_at":"2022-09-14T10:01:55.161Z","dependency_job_id":null,"html_url":"https://github.com/chenjiandongx/async-proxy-pool","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenjiandongx%2Fasync-proxy-pool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenjiandongx%2Fasync-proxy-pool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenjiandongx%2Fasync-proxy-pool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenjiandongx%2Fasync-proxy-pool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chenjiandongx","download_url":"https://codeload.github.com/chenjiandongx/async-proxy-pool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247478325,"owners_count":20945266,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["proxy-pool","python3"],"created_at":"2024-08-01T09:01:26.159Z","updated_at":"2025-04-06T12:12:18.531Z","avatar_url":"https://github.com/chenjiandongx.png","language":"Python","funding_links":[],"categories":["\u003ca id=\"1a9934198e37d6d06b881705b863afc8\"\u003e\u003c/a\u003e通信\u0026\u0026代理\u0026\u0026反向代理\u0026\u0026隧道","Python","\u003ca id=\"d03d494700077f6a65092985c06bf8e8\"\u003e\u003c/a\u003e工具"],"sub_categories":["\u003ca id=\"56acb7c49c828d4715dce57410d490d1\"\u003e\u003c/a\u003e未分类-Proxy","\u003ca id=\"b2241c68725526c88e69f1d71405c6b2\"\u003e\u003c/a\u003e代理爬取\u0026\u0026代理池"],"readme":"\u003ch1 align=\"center\"\u003eAsync Proxy Pool\u003c/h1\u003e\r\n\u003cp align=\"center\"\u003e\r\n    \u003cem\u003e异步爬虫代理池，以 Python asyncio 为基础，旨在充分利用 Python 的异步性能。\u003c/em\u003e\r\n\u003c/p\u003e\r\n\r\n### 运行环境\r\n\r\n项目使用了 [sanic](https://github.com/channelcat/sanic)，（也提供了 Flask）一个异步网络框架。所以建议运行 Python 环境为 Python3.5+，并且 sanic 不支持 Windows 系统，Windows 用户（比如我 😄）可以考虑使用 Ubuntu on Windows。\r\n\r\n\r\n### 如何使用\r\n\r\n#### 安装 Redis\r\n项目数据库使用了 [Redis](https://redis.io/)，Redis 是一个开源（BSD 许可）的，内存中的数据结构存储系统，它可以用作数据库、缓存和消息中间件。所以请确保运行环境已经正确安装了 Redis。安装方法请参照官网指南。\r\n\r\n#### 下载项目源码\r\n```bash\r\n$ git clone https://github.com/chenjiandongx/async-proxy-pool.git\r\n```\r\n\r\n#### 安装依赖\r\n使用 requirements.txt\r\n```bash\r\n$ pip install -r requirements.txt\r\n```\r\n\r\n#### 配置文件\r\n配置文件 [config.py](https://github.com/chenjiandongx/async-proxy-pool/blob/master/async_proxy_pool/config.py)，保存了项目所使用到的所有配置项。如下所示，用户可以根据需求自行更改。不然按默认即可。\r\n```python\r\n#!/usr/bin/env python\r\n# coding=utf-8\r\n\r\n# 请求超时时间（秒）\r\nREQUEST_TIMEOUT = 15\r\n# 请求延迟时间（秒）\r\nREQUEST_DELAY = 0\r\n\r\n# redis 地址\r\nREDIS_HOST = \"localhost\"\r\n# redis 端口\r\nREDIS_PORT = 6379\r\n# redis 密码\r\nREDIS_PASSWORD = None\r\n# redis set key\r\nREDIS_KEY = \"proxies:ranking\"\r\n# redis 连接池最大连接量\r\nREDIS_MAX_CONNECTION = 20\r\n\r\n# REDIS SCORE 最大分数\r\nMAX_SCORE = 10\r\n# REDIS SCORE 最小分数\r\nMIN_SCORE = 0\r\n# REDIS SCORE 初始分数\r\nINIT_SCORE = 9\r\n\r\n# server web host\r\nSERVER_HOST = \"localhost\"\r\n# server web port\r\nSERVER_PORT = 3289\r\n# 是否开启日志记录\r\nSERVER_ACCESS_LOG = True\r\n\r\n# 批量测试数量\r\nVALIDATOR_BATCH_COUNT = 256\r\n# 校验器测试网站，可以定向改为自己想爬取的网站，如新浪，知乎等\r\nVALIDATOR_BASE_URL = \"https://httpbin.org/\"\r\n# 校验器循环周期（分钟）\r\nVALIDATOR_RUN_CYCLE = 15\r\n\r\n\r\n# 爬取器循环周期（分钟）\r\nCRAWLER_RUN_CYCLE = 30\r\n# 请求 headers\r\nHEADERS = {\r\n    \"X-Requested-With\": \"XMLHttpRequest\",\r\n    \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 \"\r\n    \"(KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\",\r\n}\r\n```\r\n\r\n### 运行项目\r\n\r\n**运行客户端，启动收集器和校验器**\r\n```bash\r\n# 可设置校验网站环境变量 set/export VALIDATOR_BASE_URL=\"https://example.com\"\r\n$ python client.py\r\n2018-05-16 23:41:39,234 - Crawler working...\r\n2018-05-16 23:41:40,509 - Crawler √ http://202.83.123.33:3128\r\n2018-05-16 23:41:40,509 - Crawler √ http://123.53.118.122:61234\r\n2018-05-16 23:41:40,510 - Crawler √ http://212.237.63.84:8888\r\n2018-05-16 23:41:40,510 - Crawler √ http://36.73.102.245:8080\r\n2018-05-16 23:41:40,511 - Crawler √ http://78.137.90.253:8080\r\n2018-05-16 23:41:40,512 - Crawler √ http://5.45.70.39:1490\r\n2018-05-16 23:41:40,512 - Crawler √ http://117.102.97.162:8080\r\n2018-05-16 23:41:40,513 - Crawler √ http://109.185.149.65:8080\r\n2018-05-16 23:41:40,513 - Crawler √ http://189.39.143.172:20183\r\n2018-05-16 23:41:40,514 - Crawler √ http://186.225.112.62:20183\r\n2018-05-16 23:41:40,514 - Crawler √ http://189.126.66.154:20183\r\n...\r\n2018-05-16 23:41:55,866 - Validator working...\r\n2018-05-16 23:41:56,951 - Validator × https://114.113.126.82:80\r\n2018-05-16 23:41:56,953 - Validator × https://114.199.125.242:80\r\n2018-05-16 23:41:56,955 - Validator × https://114.228.75.17:6666\r\n2018-05-16 23:41:56,957 - Validator × https://115.227.3.86:9000\r\n2018-05-16 23:41:56,960 - Validator × https://115.229.88.191:9000\r\n2018-05-16 23:41:56,964 - Validator × https://115.229.89.100:9000\r\n2018-05-16 23:41:56,966 - Validator × https://103.18.180.194:8080\r\n2018-05-16 23:41:56,967 - Validator × https://115.229.90.207:9000\r\n2018-05-16 23:41:56,968 - Validator × https://103.216.144.17:8080\r\n2018-05-16 23:41:56,969 - Validator × https://117.65.43.29:31588\r\n2018-05-16 23:41:56,971 - Validator × https://103.248.232.135:8080\r\n2018-05-16 23:41:56,972 - Validator × https://117.94.69.166:61234\r\n2018-05-16 23:41:56,975 - Validator × https://103.26.56.109:8080\r\n...\r\n```\r\n\r\n**运行服务器，启动 web 服务**\r\n\r\n#### Sanic\r\n```bash\r\n$ python server_sanic.py\r\n[2018-05-16 23:36:22 +0800] [108] [INFO] Goin' Fast @ http://localhost:3289\r\n[2018-05-16 23:36:22 +0800] [108] [INFO] Starting worker [108]\r\n```\r\n\r\n#### Flask\r\n```bash\r\n$ python server_flask.py\r\n* Serving Flask app \"async_proxy_pool.webapi_flask\" (lazy loading)\r\n* Environment: production\r\n  WARNING: Do not use the development server in a production environment.\r\n  Use a production WSGI server instead.\r\n* Debug mode: on\r\n* Restarting with stat\r\n* Debugger is active!\r\n* Debugger PIN: 322-954-449\r\n* Running on http://localhost:3289/ (Press CTRL+C to quit)\r\n```\r\n\r\n### 总体架构\r\n\r\n项目主要几大模块分别是爬取模块，存储模块，校验模块，调度模块，接口模块。\r\n\r\n[爬取模块](https://github.com/chenjiandongx/async-proxy-pool/blob/master/async_proxy_pool/crawler.py)：负责爬取代理网站，并将所得到的代理存入到数据库，每个代理的初始化权值为 INIT_SCORE。\r\n\r\n[存储模块](https://github.com/chenjiandongx/async-proxy-pool/blob/master/async_proxy_pool/database.py)：封装了 Redis 操作的一些接口，提供 Redis 连接池。\r\n\r\n[校验模块](https://github.com/chenjiandongx/async-proxy-pool/blob/master/async_proxy_pool/validator.py)：验证代理 IP 是否可用，如果代理可用则权值 +1，最大值为 MAX_SCORE。不可用则权值 -1，直至权值为 0 时将代理从数据库中删除。\r\n\r\n[调度模块](https://github.com/chenjiandongx/async-proxy-pool/blob/master/async_proxy_pool/scheduler.py)：负责调度爬取器和校验器的运行。\r\n\r\n[接口模块](https://github.com/chenjiandongx/async-proxy-pool/blob/master/async_proxy_pool/webapi.py)：使用 sanic 提供 **WEB API** 。\r\n\r\n\r\n`/`\r\n\r\n欢迎页面\r\n```bash\r\n$ http http://localhost:3289/\r\nHTTP/1.1 200 OK\r\nConnection: keep-alive\r\nContent-Length: 42\r\nContent-Type: application/json\r\nKeep-Alive: 5\r\n\r\n{\r\n    \"Welcome\": \"This is a proxy pool system.\"\r\n}\r\n```\r\n\r\n\r\n**`/pop`**\r\n\r\n随机返回一个代理，分三次尝试。\r\n1. 尝试返回权值为 MAX_SCORE，也就是最新可用的代理。\r\n2. 尝试返回随机权值在 (MAX_SCORE -3) - MAX_SCORE 之间的代理。\r\n3. 尝试返回权值在 0 - MAX_SCORE 之间的代理\r\n```bash\r\n$ http http://localhost:3289/pop\r\nHTTP/1.1 200 OK\r\nConnection: keep-alive\r\nContent-Length: 38\r\nContent-Type: application/json\r\nKeep-Alive: 5\r\n\r\n{\r\n    \"http\": \"http://46.48.105.235:8080\"\r\n}\r\n```\r\n\r\n\r\n**`/get/\u003ccount:int\u003e`**\r\n\r\n返回指定数量的代理，权值从大到小排序。\r\n```bash\r\n$ http http://localhost:3289/get/10\r\nHTTP/1.1 200 OK\r\nConnection: keep-alive\r\nContent-Length: 393\r\nContent-Type: application/json\r\nKeep-Alive: 5\r\n\r\n[\r\n    {\r\n        \"http\": \"http://94.177.214.215:3128\"\r\n    },\r\n    {\r\n        \"http\": \"http://94.139.242.70:53281\"\r\n    },\r\n    {\r\n        \"http\": \"http://94.130.92.40:3128\"\r\n    },\r\n    {\r\n        \"http\": \"http://82.78.28.139:8080\"\r\n    },\r\n    {\r\n        \"http\": \"http://82.222.153.227:9090\"\r\n    },\r\n    {\r\n        \"http\": \"http://80.211.228.238:8888\"\r\n    },\r\n    {\r\n        \"http\": \"http://80.211.180.224:3128\"\r\n    },\r\n    {\r\n        \"http\": \"http://79.101.98.2:53281\"\r\n    },\r\n    {\r\n        \"http\": \"http://66.96.233.182:8080\"\r\n    },\r\n    {\r\n        \"http\": \"http://61.228.45.165:8080\"\r\n    }\r\n]\r\n```\r\n\r\n\r\n**`/count`**\r\n\r\n返回代理池中所有代理总数\r\n```bash\r\n$ http http://localhost:3289/count\r\nHTTP/1.1 200 OK\r\nConnection: keep-alive\r\nContent-Length: 15\r\nContent-Type: application/json\r\nKeep-Alive: 5\r\n\r\n{\r\n    \"count\": \"698\"\r\n}\r\n```\r\n\r\n\r\n**`/count/\u003cscore:int\u003e`**\r\n\r\n返回指定权值代理总数\r\n```bash\r\n$ http http://localhost:3289/count/10\r\nHTTP/1.1 200 OK\r\nConnection: keep-alive\r\nContent-Length: 15\r\nContent-Type: application/json\r\nKeep-Alive: 5\r\n\r\n{\r\n    \"count\": \"143\"\r\n}\r\n\r\n```\r\n\r\n\r\n**`/clear/\u003cscore:int\u003e`**\r\n\r\n删除权值小于等于 score 的代理\r\n```bash\r\n$ http http://localhost:3289/clear/0\r\nHTTP/1.1 200 OK\r\nConnection: keep-alive\r\nContent-Length: 22\r\nContent-Type: application/json\r\nKeep-Alive: 5\r\n\r\n{\r\n    \"Clear\": \"Successful\"\r\n}\r\n```\r\n\r\n\r\n### 扩展代理爬取网站\r\n\r\n在 crawler.py 文件里新增你自己的爬取方法。\r\n```python\r\nclass Crawler:\r\n\r\n    @staticmethod\r\n    def run():\r\n        ...\r\n\r\n    # 新增你自己的爬取方法\r\n    @staticmethod\r\n    @collect_funcs      # 加入装饰器用于最后运行函数\r\n    def crawl_xxx():\r\n        # 爬取逻辑\r\n```\r\n\r\n### 选择其他 web 框架\r\n\r\n本项目使用了 Sanic，但是开发者完全可以根据自己的需求选择其他 web 框架，web 模块是完全独立的，替换框架不会影响到项目的正常运行。需要如下步骤。\r\n\r\n1. 在 [webapi.py](https://github.com/chenjiandongx/async-proxy-pool/blob/master/async_proxy_pool/webapi.py) 里更换框架。\r\n2. 在 [server.py](https://github.com/chenjiandongx/async-proxy-pool/blob/master/server.py) 里修改 app 启动细节。\r\n\r\n\r\n### Sanic 性能测试\r\n\r\n使用 [wrk](https://github.com/wg/wrk) 进行服务器压力测试。基准测试 30 秒, 使用 12 个线程, 并发 400 个 http 连接。\r\n\r\n测试 http://127.0.0.1:3289/pop\r\n```bash\r\n$ wrk -t12 -c400 -d30s http://127.0.0.1:3289/pop\r\nRunning 30s test @ http://127.0.0.1:3289/pop\r\n  12 threads and 400 connections\r\n  Thread Stats   Avg      Stdev     Max   +/- Stdev\r\n    Latency   350.37ms  118.99ms 660.41ms   60.94%\r\n    Req/Sec    98.18     35.94   277.00     79.43%\r\n  33694 requests in 30.10s, 4.77MB read\r\n  Socket errors: connect 0, read 340, write 0, timeout 0\r\nRequests/sec:   1119.44\r\nTransfer/sec:    162.23KB\r\n```\r\n\r\n测试 http://127.0.0.1:3289/get/10\r\n```bash\r\nRunning 30s test @ http://127.0.0.1:3289/get/10\r\n  12 threads and 400 connections\r\n  Thread Stats   Avg      Stdev     Max   +/- Stdev\r\n    Latency   254.90ms   95.43ms 615.14ms   63.51%\r\n    Req/Sec   144.84     61.52   320.00     66.58%\r\n  46538 requests in 30.10s, 22.37MB read\r\n  Socket errors: connect 0, read 28, write 0, timeout 0\r\nRequests/sec:   1546.20\r\nTransfer/sec:    761.02KB\r\n```\r\n\r\n性能还算不错，再测试一下没有 Redis 操作的 http://127.0.0.1:3289/\r\n```bash\r\n$ wrk -t12 -c400 -d30s http://127.0.0.1:3289/\r\nRunning 30s test @ http://127.0.0.1:3289/\r\n  12 threads and 400 connections\r\n  Thread Stats   Avg      Stdev     Max   +/- Stdev\r\n    Latency   127.86ms   41.71ms 260.69ms   55.22%\r\n    Req/Sec   258.56     92.25   520.00     68.90%\r\n  92766 requests in 30.10s, 13.45MB read\r\nRequests/sec:   3081.87\r\nTransfer/sec:    457.47KB\r\n```\r\n⭐️ **Requests/sec:   3081.87**\r\n\r\n关闭 sanic 日志记录，测试 http://127.0.0.1:3289/\r\n```bash\r\n$ wrk -t12 -c400 -d30s http://127.0.0.1:3289/\r\nRunning 30s test @ http://127.0.0.1:3289/\r\n  12 threads and 400 connections\r\n  Thread Stats   Avg      Stdev     Max   +/- Stdev\r\n    Latency    34.63ms   12.66ms  96.28ms   58.07%\r\n    Req/Sec     0.96k   137.29     2.21k    73.29%\r\n  342764 requests in 30.10s, 49.69MB read\r\nRequests/sec:  11387.89\r\nTransfer/sec:      1.65MB\r\n```\r\n⭐️ **Requests/sec:  11387.89**\r\n\r\n\r\n### 实际代理性能测试\r\n\r\n[test_proxy.py](https://github.com/chenjiandongx/async-proxy-pool/blob/master/test/test_proxy.py) 用于测试实际代理性能\r\n\r\n#### 运行代码\r\n\r\n```bash\r\n$ cd test\r\n$ python test_proxy.py\r\n\r\n# 可设置的环境变量\r\nTEST_COUNT = os.environ.get(\"TEST_COUNT\") or 1000\r\nTEST_WEBSITE = os.environ.get(\"TEST_WEBSITE\") or \"https://httpbin.org/\"\r\nTEST_PROXIES = os.environ.get(\"TEST_PROXIES\") or \"http://localhost:3289/get/20\"\r\n```\r\n\r\n#### 实测效果\r\n\r\n**https://httpbin.org/**\r\n```\r\n测试代理： http://localhost:3289/get/20\r\n测试网站： https://httpbin.org/\r\n测试次数： 1000\r\n成功次数： 1000\r\n失败次数： 0\r\n成功率： 1.0\r\n```\r\n\r\n**https://taobao.com**\r\n```\r\n测试代理： http://localhost:3289/get/20\r\n测试网站： https://taobao.com/\r\n测试次数： 1000\r\n成功次数： 984\r\n失败次数： 16\r\n成功率： 0.984\r\n```\r\n\r\n**https://baidu.com**\r\n```\r\n测试代理： http://localhost:3289/get/20\r\n测试网站： https://baidu.com\r\n测试次数： 1000\r\n成功次数： 975\r\n失败次数： 25\r\n成功率： 0.975\r\n```\r\n\r\n**https://zhihu.com**\r\n```\r\n测试代理： http://localhost:3289/get/20\r\n测试网站： https://zhihu.com\r\n测试次数： 1000\r\n成功次数： 1000\r\n失败次数： 0\r\n成功率： 1.0\r\n```\r\n\r\n可以看到其实性能是非常棒的，成功率极高。 😉\r\n\r\n\r\n### 实际应用示例\r\n\r\n```python\r\nimport random\r\n\r\nimport requests\r\n\r\n# 确保已经启动 sanic 服务\r\n# 获取多个然后随机选一个\r\n\r\ntry:\r\n    proxies = requests.get(\"http://localhost:3289/get/20\").json()\r\n    req = requests.get(\"https://example.com\", proxies=random.choice(proxies))\r\nexcept:\r\n    raise\r\n\r\n# 或者单独弹出一个\r\n\r\ntry:\r\n    proxy = requests.get(\"http://localhost:3289/pop\").json()\r\n    req = requests.get(\"https://example.com\", proxies=proxy)\r\nexcept:\r\n    raise\r\n```\r\n\r\n\r\n### aiohttp 的坑\r\n\r\n整个项目都是基于 aiohttp 这个异步网络库的，在这个项目的文档中，关于代理的介绍是这样的。\r\n\r\n![](https://user-images.githubusercontent.com/19553554/40276465-745db54a-5c3d-11e8-8662-0c73fdf4fe88.png)\r\n\r\n**划重点：aiohttp supports HTTP/HTTPS proxies**\r\n\r\n但是，它根本就不支持 https 代理好吧，在它的代码中是这样写的。\r\n\r\n![](https://user-images.githubusercontent.com/19553554/40276470-a0d46a6a-5c3d-11e8-871d-a053c81fec56.png)\r\n\r\n**划重点：Only http proxies are supported**\r\n\r\n我的心情可以说是十分复杂的。😲 不过只有 http 代理效果也不错没什么太大影响，参见上面的测试数据。\r\n\r\n\r\n### 参考借鉴项目\r\n\r\n✨🍰✨\r\n\r\n* [ProxyPool](https://github.com/WiseDoge/ProxyPool)\r\n* [proxy_pool](https://github.com/jhao104/proxy_pool)\r\n\r\n### License\r\n\r\nMIT [©chenjiandongx](https://github.com/chenjiandongx)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenjiandongx%2Fasync-proxy-pool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchenjiandongx%2Fasync-proxy-pool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenjiandongx%2Fasync-proxy-pool/lists"}