{"id":13776052,"url":"https://github.com/leeyis/ip_proxy_pool","last_synced_at":"2025-05-11T08:34:47.434Z","repository":{"id":212605810,"uuid":"75596730","full_name":"leeyis/ip_proxy_pool","owner":"leeyis","description":"Generating spiders dynamically to crawl and check those free proxy ip on the internet with scrapy.","archived":false,"fork":false,"pushed_at":"2018-10-06T13:23:26.000Z","size":42,"stargazers_count":44,"open_issues_count":1,"forks_count":18,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-08-03T17:12:27.840Z","etag":null,"topics":["dynamic","proxy","proxy-ip","scrapy","spider"],"latest_commit_sha":null,"homepage":"http://jinbitou.net/crawler","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leeyis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-12-05T06:43:20.000Z","updated_at":"2024-07-27T21:23:08.000Z","dependencies_parsed_at":"2023-12-15T07:28:17.164Z","dependency_job_id":"306e8a92-aed9-4470-979a-70a80146b9df","html_url":"https://github.com/leeyis/ip_proxy_pool","commit_stats":null,"previous_names":["leeyis/ip_proxy_pool"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leeyis%2Fip_proxy_pool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leeyis%2Fip_proxy_pool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leeyis%2Fip_proxy_pool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leeyis%2Fip_proxy_pool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leeyis","download_url":"https://codeload.github.com/leeyis/ip_proxy_pool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225031480,"owners_count":17410082,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dynamic","proxy","proxy-ip","scrapy","spider"],"created_at":"2024-08-03T17:01:58.746Z","updated_at":"2024-11-17T11:30:59.843Z","avatar_url":"https://github.com/leeyis.png","language":"Python","funding_links":[],"categories":["\u003ca id=\"d03d494700077f6a65092985c06bf8e8\"\u003e\u003c/a\u003e工具"],"sub_categories":["\u003ca id=\"b2241c68725526c88e69f1d71405c6b2\"\u003e\u003c/a\u003e代理爬取\u0026\u0026代理池"],"readme":"# ip_proxy_pool\n\nA dynamic configurable proxy IP crawler based on Scrapy. It makes it easy to crawl hundreds of thousands of proxy IPs in a short time. By maintaining a spider code and a few groups of website data extraction rules you can easily grab lots of proxy IPs of these sites. See the [blogs](http://jinbitou.net/2016/12/05/2244.html) for more detail.\n\n## Main Requirements\nFor more details see requirements.txt\n- Scrapy 1.2.1\n- MySQL-python 1.2.5\n- Redis 2.10.5\n- SQLAlchemy 1.1.4\n\n## Install in development\n\n**CentOS**\n\n```bash\n$ sudo yum install python-devel\n$ sudo yum install gcc libffi-devel openssl-devel\n$ pip install scrapy\n$ pip install SQLAlchemy\n$ pip install redis\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleeyis%2Fip_proxy_pool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleeyis%2Fip_proxy_pool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleeyis%2Fip_proxy_pool/lists"}