{"id":43506423,"url":"https://github.com/phpgao/proxy_pool","last_synced_at":"2026-02-03T12:27:47.346Z","repository":{"id":89000203,"uuid":"222870282","full_name":"phpgao/proxy_pool","owner":"phpgao","description":"A simple proxy pool","archived":false,"fork":false,"pushed_at":"2021-11-09T16:51:47.000Z","size":4198,"stargazers_count":75,"open_issues_count":1,"forks_count":13,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-06-20T13:37:19.455Z","etag":null,"topics":["api","golang","http","proxy","proxy-pool","proxy-server"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/phpgao.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-20T06:49:36.000Z","updated_at":"2024-01-02T08:36:19.000Z","dependencies_parsed_at":"2023-06-19T09:57:38.636Z","dependency_job_id":null,"html_url":"https://github.com/phpgao/proxy_pool","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/phpgao/proxy_pool","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phpgao%2Fproxy_pool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phpgao%2Fproxy_pool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phpgao%2Fproxy_pool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phpgao%2Fproxy_pool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/phpgao","download_url":"https://codeload.github.com/phpgao/proxy_pool/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phpgao%2Fproxy_pool/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29045806,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-03T10:09:22.136Z","status":"ssl_error","status_checked_at":"2026-02-03T10:09:16.814Z","response_time":96,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","golang","http","proxy","proxy-pool","proxy-server"],"created_at":"2026-02-03T12:27:46.745Z","updated_at":"2026-02-03T12:27:47.325Z","avatar_url":"https://github.com/phpgao.png","language":"Go","readme":"# proxy_pool\n\n一个简单的代理池工具\n\nA simple proxy pool written in go\n\n## 功能\n\n - 定时抓取互联网公开免费的代理\n - 定时验证可用代理\n - 支持动态代理(https仅支持connect)\n - 使用采集到的代理访问代理网站\n - 使用命令行环境变量进行配置\n - 当没有IP可用时使用本地转发\n\n## 依赖\n\n - redis\n\n## 使用说明\n\n### 编译\n\n```bash\ngo build\n```\n\n### 下载\n\n```bash\n# 版本 v0.3.3\nwget https://github.com/phpgao/proxy_pool/releases/download/v0.3.3/proxy_pool_linux_amd64\nchmod a+x proxy_pool_linux_amd64\n```\n\n### 使用\n\n```bash\ncp config_example.json config.json\n# 修改redis和端口配置\n\n# 感谢ipip.net提供精准的IP数据(已内置)\n./proxy_pool\n\n# 打印可设定参数\n./proxy_pool_linux_amd64 --help\n\n# 命令行指定配置\n./proxy_pool_linux_amd64 -host 8.8.8.8 -port 6379 -auth laogao\n\n# 后台运行\nnohup ./proxy_pool_linux_amd64 \u003e /dev/null 2\u003e\u00261 \u0026\n```\n\n### api\n\n```bash\n# 统计\ncurl 127.0.0.1:8088\n# 随机\ncurl 127.0.0.1:8088/random\n# 获取列表\ncurl 127.0.0.1:8088/get\n```\n\n### 动态代理\n\n```bash\n# http\ncurl http://cip.cc -x 127.0.0.1:8089\n# https\ncurl https://cip.cc -x 127.0.0.1:8089\n```\n\n## 一些细节\n\n### 流程图\n\n```\n                                                                 +-------------------------+\n                                                                 |                         |\n                                                                 |                         |\n+-------------+      +------------+         +-------------+      |                         |\n|             |      |            |         |             |      |                         |\n|   source    +------\u003e new proxy  +---------\u003e  validator  +------\u003e                         |\n|             +------\u003e            +---------\u003e             +------\u003e                         |\n|             |      |            |         |             |      |         The Pool        |\n+-------------+      +------------+         +-------------+      |                         |\n                                                                 |                         |\n                             +-----------------------------------\u003e                         |\n                             |           +/- score               |                         |\n                             |                                   |                         |\n                             |                                   |                         |\n                             |                                   +-------------+-----------+\n                             |                                                 |\n                     +-------+------+        +--------------+                  |\n                     |              |        |              |                  |\n                     |              \u003c--------+              |      cron        |\n                     |  old proxy   \u003c--------+  validator   +\u003c-----------------+\n                     |              |        |              |\n                     |              |        |              |\n                     +--------------+        +--------------+\n\n```\n\n### 关于验证逻辑\n\n 1. 代理检测采用打分机制，新代理默认60分，满分100，检测每失败一次扣30分，成功一次加10分，当分数小于等于0时，对应的代理地址将会被删除\n 1. 新的代理入库前有三道检测(tcp,http,https)，只要通过了http测试，就会被添加到数据库中\n 1. 定时检测只会测试tcp和https的connect方法，同时会把之前判定为http的代理修正为https，但是如果一个https被检测到错误，会扣20分\n 1. 目前规则还不算很完善，欢迎大家一起讨论，提高代理的稳定性\n\n### 关于添加采集源\n\n 1. demo见源码source文件夹下\n 1. 爬虫分为html、json、re还有text型\n 1. 爬虫复用了Spider结构体，新爬虫必须实现的方法如下\n \n   - Cron --\u003e 定义了启动间隔\n   - Name --\u003e 定义了爬虫名\n   - Run --\u003e  通用方法，用来执行下载和解析，照抄即可\n   - StartUrl --\u003e 返回目标网站的入口页面\n   - Parse --\u003e 接收最终的html代理，返回[]model.HttpProxy实例的指针\n \n### 关于动态代理\n\n 1. 核心代码取自[HTTP(S) Proxy in Golang in less than 100 lines of code](https://medium.com/@mlowicki/http-s-proxy-in-golang-in-less-than-100-lines-of-code-6a51c2f2c38c)，做了一些针对性的优化\n 1. 如果当前没有代理可用，软件会把自身作为透明代理\n\n### 关于传统API代理\n \n 1. 接口分为统计和获取\n 2. 查询支持schema=http(s)，source=spider.name，score=100，country=cn\n\n## todo\n\n - tcp池\n - go test\n - 更精细的超时控制\n - 主从模式\n - 代理认证\n\n## 反馈\n\n期待大家的测试和反馈！\n\n## 更多\n\n如果你在使用scrapy，为什么不试试[scrapy-random-useragent-pro](https://github.com/phpgao/scrapy-random-useragent-pro)呢？\n\n## 感谢\n \n- https://www.ipip.net/\n- https://medium.com/@mlowicki/http-s-proxy-in-golang-in-less-than-100-lines-of-code-6a51c2f2c38c\n- 感谢给我点star的朋友~\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphpgao%2Fproxy_pool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphpgao%2Fproxy_pool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphpgao%2Fproxy_pool/lists"}