{"id":15294054,"url":"https://github.com/moqsien/nproxypool","last_synced_at":"2026-01-05T06:50:55.771Z","repository":{"id":57447019,"uuid":"342863837","full_name":"moqsien/nproxypool","owner":"moqsien","description":"nporxypool is a very powerful proxypool with high flexibility. Redis + aiohttp + flask + gunicorn.","archived":false,"fork":false,"pushed_at":"2021-02-27T14:03:26.000Z","size":242,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-24T09:31:45.124Z","etag":null,"topics":["aiohttp","flask","gunicorn","proxypool","redis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/moqsien.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-27T13:32:05.000Z","updated_at":"2022-08-27T17:36:57.000Z","dependencies_parsed_at":"2022-08-30T01:00:33.018Z","dependency_job_id":null,"html_url":"https://github.com/moqsien/nproxypool","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moqsien%2Fnproxypool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moqsien%2Fnproxypool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moqsien%2Fnproxypool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moqsien%2Fnproxypool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/moqsien","download_url":"https://codeload.github.com/moqsien/nproxypool/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245301440,"owners_count":20593181,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiohttp","flask","gunicorn","proxypool","redis"],"created_at":"2024-09-30T16:56:05.423Z","updated_at":"2026-01-05T06:50:55.740Z","avatar_url":"https://github.com/moqsien.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### 关于nproxypool\nnproxypool是一个强大、灵活、高效、方便的代理池包。\nRedis+aiohttp+Flask+gunicorn实现，用户只需要用相关命令创建项目，编写代理获取程序，就能在命令行启动代理池和代理获取服务。\n\n代理池可以根据配置文件灵活的生成检测进程，针对性的检测对于某个网站的有效性，并自动维护一个代理池（有序集合）。\n检测进程可以根据需要调节并发数，以维持代理池有效ip的数量合理，也同时控制检测程序对代理ip带宽的消耗。\n可以设置可接受的最大响应时间，超过该响应时间的代理ip都被认为是不合格的。\n\n实行打分制度，初始分数为60，检测合格一次直接升至100，检测不合格一次减30分，直至超过阈值就会被删除。\n\n### nproxypool工作流程\n![avatar](https://github.com/moqsien/nproxypool/blob/main/docs/proxypool.png)\n\n### nproxypool使用方法\n\n```bash\n# 安装\npip install nproxypool\n# or \npython setup.py build\npython setup.py install\n\n# 生成项目\nnpool gen example\n\n\n# 编写ip获取程序\ncd example\n# 在my_fetcher.py中按照示例类编写自己的类即可\n# edit my_fetcher.py with an editor\n\n# 修改配置\n# 修改pool.cfg中的相关配置，例如redis、gunicorn等的配置\n# edit pool.cfg with an editor\n\n# 运行命令\nnpool run_pool\nnpool run_server\n\n# 使用代理IP\ncurl \"http://localhost:5001/proxy/random/?key=baidu\"\ncurl \"http://localhost:5001/proxy/list/?key=baidu\"\ncurl \"http://localhost:5001/proxy/total/?key=baidu\"\n```\n\n### pool.cfg配置参数说明\n- [redis] redis配置\n- [logfile] 日志存放目录（无需修改）\n- [gunicorn] gunicorn配置（gunicorn日志目录无需修改）\n- [synchronizer] 间隔多久同步一次一代池和二代池之间的ip，即把存在于二代池但是不存在于一代池中的代理IP清理掉\n- [fetcher] 间隔多久从数据源获取一次代理ip\n- [verifier_xxx] 检测进程配置，一个配置代表着一个检测进程，不能重名\n    - name verifier的名称\n    - url 用于检测的url\n    - from_db 从哪个db获取代理\n    - from_db_key 用于获取db的redis key，不填表示从一代池获取（因为一代池以代理ip作为key）\n    - is_from_db_zset 是否是从zset中获取代理ip用于检测（从一代池中获取则为False）\n    - to_db 检测完成写入那个db\n    - to_db_key 写入db的哪个key中（写入一代池时，此项为空）\n    - is_to_db_zset 是否是写入zset（写入一代池时，此项为False）\n    - score_threshold 设置代理ip被删除时的分数最小值\n    - score_decrease_by 每次检测失败之后减多少分\n    - time_out 代理ip检测超时时间，越小则ip池代理响应越快\n    - to_sleep 检测进程间隔多久检测一轮\n    - max_worker 检测进程发送请求的并发数，越高检测越快\n    \n### 一代池代理ip效果图\n![avatar](https://github.com/moqsien/nproxypool/blob/main/docs/first_level_pool.png)\n\n### 二代池代理ip效果图\n![avatar](https://github.com/moqsien/nproxypool/blob/main/docs/second_level_pool.png)\n\n### 通过server获取检测后的ip效果图\n![avatar](https://github.com/moqsien/nproxypool/blob/main/docs/random.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoqsien%2Fnproxypool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmoqsien%2Fnproxypool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoqsien%2Fnproxypool/lists"}