{"id":13776053,"url":"https://github.com/yaleimeng/free_proxy_pool","last_synced_at":"2025-05-11T08:34:45.714Z","repository":{"id":113353429,"uuid":"106258629","full_name":"yaleimeng/Free_proxy_pool","owner":"yaleimeng","description":"对免费代理IP网站进行爬取，收集汇总为自己的代理池。关键是验证代理的有效性、匿名性、去重复","archived":false,"fork":false,"pushed_at":"2022-01-11T02:36:21.000Z","size":70,"stargazers_count":74,"open_issues_count":0,"forks_count":23,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-08-03T17:12:27.934Z","etag":null,"topics":["proxy","proxypool","spiders"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yaleimeng.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-10-09T08:47:10.000Z","updated_at":"2024-05-20T20:45:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"14cdb6f6-87e7-4499-907b-353451d84176","html_url":"https://github.com/yaleimeng/Free_proxy_pool","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yaleimeng%2FFree_proxy_pool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yaleimeng%2FFree_proxy_pool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yaleimeng%2FFree_proxy_pool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yaleimeng%2FFree_proxy_pool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yaleimeng","download_url":"https://codeload.github.com/yaleimeng/Free_proxy_pool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225031469,"owners_count":17410079,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["proxy","proxypool","spiders"],"created_at":"2024-08-03T17:01:58.776Z","updated_at":"2024-11-17T11:30:58.259Z","avatar_url":"https://github.com/yaleimeng.png","language":"Python","funding_links":[],"categories":["\u003ca id=\"d03d494700077f6a65092985c06bf8e8\"\u003e\u003c/a\u003e工具"],"sub_categories":["\u003ca id=\"b2241c68725526c88e69f1d71405c6b2\"\u003e\u003c/a\u003e代理爬取\u0026\u0026代理池"],"readme":"# Free_proxy_pool\n对免费代理IP网站进行爬取，收集汇总为自己的代理池。其中关键是验证代理的有效性、匿名性、去重复。\u003c/br\u003e\n\n+ 本代理池的定位是初学者能看懂，能使用的单机库。所以不打算使用高大上的Redis或者MongoDB等数据库。抓到的代理仅与磁盘文件交互。\u003c/br\u003e\n+ 本项目**无需安装，下载后查看example**即可学会使用。简洁易用的get_a_proxy()，便于在请求网页的参数中直接使用。\u003c/br\u003e\n+ 运行所依赖的第三方库：requests、bs4、lxml、chardet。\u003c/br\u003e\n+ 爬虫组件\u003c200行代码，把高可用率的代理网站一网打尽。如果爬虫全开，一次性可以采集高匿代理300--700个。\n+ 下表数据仅表明各网站在某一时段的代理质量，每天不同时段会有波动，整体质量比较高。\n\n代理网站|评估数|可用数|有效率|更新周期\n:-:|-:|-:|-:|:-:\n66ip|536|113|21.1%|10分钟\n89ip|600|79|11.7%|10分钟\n爱家网|1274|105|8.2%|每天2篇\ncodebusy|278|84|30.2%|1分钟\n小舒代理|534|78|14.6%|每天2篇\n小河虾|600|70|11.7%|10分钟\n\n如果感觉对您有帮助，欢迎给我加一个星星，或者fork。\u003c/br\u003e\n为了避免大家浪费精力，经验证无实用价值的免费代理网站列举如下，是为“黑名单”：\u003c/br\u003e\n- http://www.ip181.com/\n\n- https://list.proxylistplus.com/\n- http://www.xicidaili.com/nn    百度排名靠前，可用率仅1%左右。\n- http://www.kuaidaili.com/free/inha     较新的只有前5页，但可用仅1--2个。\n- 更新日期：2019-11-5\u003c/br\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyaleimeng%2Ffree_proxy_pool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyaleimeng%2Ffree_proxy_pool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyaleimeng%2Ffree_proxy_pool/lists"}