{"id":18673044,"url":"https://github.com/vicanso/proxy-pool","last_synced_at":"2025-04-12T01:31:23.011Z","repository":{"id":39283277,"uuid":"227979885","full_name":"vicanso/proxy-pool","owner":"vicanso","description":"http(s) proxy pool","archived":false,"fork":false,"pushed_at":"2023-01-05T03:17:51.000Z","size":1438,"stargazers_count":18,"open_issues_count":26,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-25T21:36:03.263Z","etag":null,"topics":["http-proxy","https-proxy","proxy","proxy-pool"],"latest_commit_sha":null,"homepage":"https://proxy.aslant.site/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vicanso.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-14T06:54:27.000Z","updated_at":"2024-11-19T11:05:47.000Z","dependencies_parsed_at":"2023-02-03T08:15:42.194Z","dependency_job_id":null,"html_url":"https://github.com/vicanso/proxy-pool","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vicanso%2Fproxy-pool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vicanso%2Fproxy-pool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vicanso%2Fproxy-pool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vicanso%2Fproxy-pool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vicanso","download_url":"https://codeload.github.com/vicanso/proxy-pool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248504246,"owners_count":21115138,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["http-proxy","https-proxy","proxy","proxy-pool"],"created_at":"2024-11-07T09:13:50.516Z","updated_at":"2025-04-12T01:31:22.706Z","avatar_url":"https://github.com/vicanso.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# proxy-pool\n\n虽然网上各类网站提供了一堆的免费代理地址，但是其可用性比较差，更新不及时，过多不可用的地址，以及延时较大等问题都干扰实际使用的效果。对于代理地址，期望是越多越好，但是对于代理质量有着更高的要求，宁缺勿滥，因此`proxy-pool`不再将抓取到的代理地址保存至数据库，而调整为定期从免费代理网站下抓取代理地址，使用该地址去测试其可用性（默认配置为访问baidu），测试可用则添加至可用代理地址列表中，如此循环一直抓取新的地址，一直校验。对于已校验可用的代理地址，也定期重新校验是否可用，默认校验间隔为30分钟。\n\n注意：网页部分有增加百度统计，部署时可先删除。\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/vicanso/proxy-pool/master/assets/proxy-pool.png\"\u003e\n\u003c/p\u003e\n\n## 常用配置\n\n对于有特别需求，可以调整默认的配置，主要的配置如下：\n\n抓取代理网站列表配置（暂时只实现了三个网站的抓取）：\n\n```yml\ncrawler:\n- xici\n- ip66\n- kuai\n```\n\n由于各网站对访问IP频率限制的不同，可根据实际使用中调整各网站的抓取间隔，如设置`xici`的抓取延时为10分钟（如果不配置则为默认值2分钟）：\n\n```yml\nxici:\n  interval: 10m\n```\n\n默认的检测方式是通过代理地址去访问`baidu`，可根据应用场景调整相应的配置：\n\n```yml\ndetect:\n  # 检测时间（定时对现可用的代理地址重新检测）\n  interval: 30m\n  # 检测地址\n  url: https://www.baidu.com/\n  # 检测超时\n  timeout: 3s\n  # 最大次数\n  maxTimes: 3\n```\n\n## 程序设计\n\n- [config](./doc/config.md)\n- [crawler](./doc/crawler.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvicanso%2Fproxy-pool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvicanso%2Fproxy-pool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvicanso%2Fproxy-pool/lists"}