{"id":13488580,"url":"https://github.com/Karmenzind/fp-server","last_synced_at":"2025-03-28T01:36:32.037Z","repository":{"id":37062764,"uuid":"135252696","full_name":"Karmenzind/fp-server","owner":"Karmenzind","description":"Free proxy server, continuously crawling and providing proxies, based on Tornado and Scrapy. 免费代理服务器，基于Tornado和Scrapy，在本地搭建属于自己的代理池","archived":false,"fork":false,"pushed_at":"2023-05-25T17:17:45.000Z","size":786,"stargazers_count":160,"open_issues_count":11,"forks_count":36,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-08-01T18:38:24.915Z","etag":null,"topics":["proxy","proxypool","python","scrapy","spider","tornado"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Karmenzind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-05-29T06:42:59.000Z","updated_at":"2024-04-24T07:58:52.000Z","dependencies_parsed_at":"2024-01-16T09:02:53.088Z","dependency_job_id":null,"html_url":"https://github.com/Karmenzind/fp-server","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Karmenzind%2Ffp-server","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Karmenzind%2Ffp-server/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Karmenzind%2Ffp-server/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Karmenzind%2Ffp-server/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Karmenzind","download_url":"https://codeload.github.com/Karmenzind/fp-server/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222333976,"owners_count":16968058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["proxy","proxypool","python","scrapy","spider","tornado"],"created_at":"2024-07-31T18:01:18.356Z","updated_at":"2025-03-28T01:36:32.024Z","avatar_url":"https://github.com/Karmenzind.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# fp-server\n\n:exclamation: **This code really sucks. I'll rewrite it when I have free time. 这项目写的很烂，仅供参考，后续有时间会重构……**\n\n------\n\n[![Scrutinizer Build](https://img.shields.io/scrutinizer/build/g/filp/whoops.svg?style=flat-square)](https://github.com/Karmenzind/fp-server) [![Python Version](https://img.shields.io/badge/python-3.5%2B-yellow.svg?style=flat-square)](https://www.python.org/) [![CocoaPods](https://img.shields.io/cocoapods/l/AFNetworking.svg?style=flat-square)](https://github.com/Karmenzind/fp-server)\n\nA free proxy server based on [Tornado](http://www.tornadoweb.org/en/stable/#) and [Scrapy](https://scrapy.org/). \n\nBuild your own proxy pool!\n\nFeatures:\n- continuously crawling and providing free proxies\n- asynchronous and high-perfermance\n- automatically check proxies in cycle and ditch unavailable ones\n- easy-to-use HTTP api\n\n免费代理服务器，基于[Tornado](http://www.tornadoweb.org/en/stable/#)和[Scrapy](https://scrapy.org/)，在本地搭建自己的代理池\n- 持续爬取新的免费代理，检测可用后存入本地数据库\n- 完全异步，支持高并发\n- 易用的HTTP API\n- 周期性检测代理可用性，自动更新\n\n[**查看中文文档\\_(:ι」∠)\\_**](./README_CN.md) \n\nThis project has been tested on:\n- Archlinux; Python-3.6.5\n- Debian(WSL, Raspbian); Python-3.5.3\n\nAnd it **cannot directly run on Windows**. Windows users may try [using Docker](#using-docker) or WSL to run this project.\n\n## Contents ##\n\n\u003c!-- vim-markdown-toc GFM --\u003e\n\n* [Get started](#get-started)\n    * [Using Docker](#using-docker)\n    * [Manually install](#manually-install)\n* [web APIs](#web-apis)\n    * [get proxies](#get-proxies)\n    * [create new proxy manually](#create-new-proxy-manually)\n    * [check status](#check-status)\n* [Config](#config)\n    * [Introduction](#introduction)\n    * [Customization](#customization)\n* [Source webs](#source-webs)\n* [FAQ](#faq)\n* [Examples](#examples)\n    * [Use fp-server with Python requests module](#use-fp-server-with-python-requests-module)\n    * [Use fp-server in Scrapy Project](#use-fp-server-in-scrapy-project)\n* [Bugs and feature requests](#bugs-and-feature-requests)\n* [TODOs and ideas](#todos-and-ideas)\n\n\u003c!-- vim-markdown-toc --\u003e\n\n## Get started ##\n\nChoose either one option as follows.\nAfter successful deployment, use the [APIs](#web-apis) to get proxies.\n\n### Using Docker ###\n\nThe easiest way to run this repo is using [Docker](https://www.docker.com/). Install Docker and then run:\n```bash\n# download the image\ndocker pull karmenzind/fp-server:stable\n# run the container\n# don't forget to modify `-p` if you prefer another port\ndocker run -itd --name fpserver -p 12345:12345 karmenzind/fp-server:stable\n# check the output inside the container\ndocker logs -f fpserver\n```\nFor custom configuratiuon, see [this section](#config).\n\n### Manually install ###\n\n1. Install [Redis](https://redis.io/) and `python\u003e=3.5`(I use Python-3.6.5). \n2. Clone this repo. \n3. Install python packages by: \n```bash\npip install -r requirements.txt\n```\n4. Read the [config](#config) and modify it according to your need.\n5. Start the server:\n```bash\npython ./src/main.py\n```\n\n## web APIs ##\n\ntypical response:\n```json\n{\n    \"code\": 0,\n    \"msg\": \"ok\",\n    \"data\": {}\n}\n```\n\n-   code: result of event (not http code), 0 for sucess\n-   msg: message for event\n-   data: detail for sucessful event\n\n### get proxies ###\n\n```\nGET /api/proxy/\n``` \n\n params                 | Must/\u003cbr\u003eOptional | detail                                                               | default\n------------------------|-------------------|----------------------------------------------------------------------|---------|\n count                  | O                 | the number of proxies you need                                       | 1\n scheme                 | O                 | choices:`HTTP` `HTTPS`                                               | both*\nanonymity               | O                 | choices:`transparent` `anonymous`                                    | both\n(TODO)\u003cbr\u003esort_by_speed | O                 | choices:\u003cbr\u003e1: desending order\u003cbr\u003e0: no order\u003cbr\u003e-1: ascending order | 0\n\n- both: include all type, not grouped\n\n**example**\n\n-   To acquire 10 proxies in HTTP scheme with anonymity:\n    ```\n    GET /api/proxy/?count=10\u0026scheme=HTTP\u0026anonymity=anonymous\n    ```\n    The response:\n    ```json\n    {\n        \"code\": 0,\n        \"msg\": \"ok\",\n        \"data\": {\n            \"count\": 9,\n            \"items\": [\n            {\n                \"port\": 2000,\n                \"ip\": \"xxx.xxx.xx.xxx\",\n                \"scheme\": \"HTTP\",\n                \"url\": \"http://xxx.xxx.xxx.xx:xxxx\",\n                \"anonymity\": \"transparent\"\n            }\n            ]\n        }\n    }\n    ```\n\n**screenshot**\n\n![](https://raw.githubusercontent.com/Karmenzind/i/master/fp-server/proxy_get.png)\n\n### create new proxy manually ###\n\n```\nPOST /api/proxy/\n```\n\n params   | Must/\u003cbr\u003eOptional | detail                            | default\n----------|-------------------|-----------------------------------|--------------------------------------|\nip        | M                 | e.g. 111.111.111.111              |\nport      | M                 | e.g. 12345                        |\nscheme    | M                 | choices:`HTTP` `HTTPS`            |\nanonymity | O                 | choices:`transparent` `anonymous` | `transparent`\nneed_auth | O                 | choices: 0 1                      |\nuser      | O                 |                                   |\npassword  | O                 |                                   |\nurl       | O                 |                                   | generated by given\u003cbr\u003escheme+ip+port\n\n**screenshot**\n\n![](https://raw.githubusercontent.com/Karmenzind/i/master/fp-server/proxy_post.png)\n\n\n### check status ###\n\nCheck server status. Include:\n-   Running spiders\n-   Stored proxies\n\n```\nGET /api/status/\n```\n\nNo params.\n\n**screenshot**\n\n![](https://raw.githubusercontent.com/Karmenzind/i/master/fp-server/status.png)\n\n## Config ##\n\n### Introduction ###\nI choose YAML language for configuration file. The defination and default value for supported items are:\n\n```yaml\n# server's http port\nHTTP_PORT: 12345\n\n# redirect output to console other than log file\nCONSOLE_OUTPUT: 1\n\n# Log\n# dir and filename requires `CONSOLE_OUTPUT: 0`\nLOG: \n  level: 'debug'\n  dir: './logs'\n  filename: 'fp-server.log'\n\n# redis database\nREDIS:\n  host: '127.0.0.1'\n  port: 6379\n  db: 0\n  password:\n\n# stop crawling new proxies\n# after stored this many proxies\nPROXY_STORE_NUM: 500\n\n# Check availability in cycle\n# It's for each single proxy, not the checker\nPROXY_STORE_CHECK_SEC: 3600\n```\n\n### Customization ###\n\n- If you use Docker:\n    - Create a directory such as `/x/config_dir` and put your `config.yml` in it. Then modify the docker-run command like this:\n        ```\n        docker run -itd --name fpserver -p 12345:12345 -v \"/x/config_dir\":\"/fps-config\" karmenzind/fp-server:stable\n        ```\n    - External `config.yml` doesn't need to contain all config items. For example, it can be:\n        ```\n        PROXY_STORE_NUM: 100\n        LOG:\n            level: 'info'\n        PROXY_STORE_CHECK_SEC: 7200\n        ```\n        And other items will be default values.\n    - If you need to set a log file, **don't** modify `LOG-dir` in `config.yml`. Instead create a directory for log file such as `/x/log_dir` and change the docker-run command like:\n        ```\n        docker run -itd --name fpserver -p 12345:12345 -v \"/x/config_dir\":\"/fps_config\" -v \"/x/log_dir\":\"/fp_server/logs\" karmenzind/fp-server:stable\n        ```\n    - There's no need to modify the exposed port of the container. If you prefer publishing it to another port(say, 9999) on the host, change the `-p` parameter in docker-run command to `-p 9999:12345`\n    - If you need to access the Redis from host, add a new publishing parameter like `-p 6379:6379` to docker-run command.\n- If you manually deploy the project:\n    - Modify the internal config file: `src/config/common.py`\n\n## Source webs ##\n\nGrowing……\n\nIf you knew good free-proxy websites, please tell me and I will add them to this project.\n\nSupporting:\n- [x] [西刺代理](http://www.xicidaili.com)\n- [x] [快代理](http://www.kuaidaili.com)\n- [x] [云代理](http://www.ip3366.net)\n- [x] [66免费代理](http://www.66ip.cn/)\n- [x] [无忧代理](http://www.data5u.com/free/index.shtml)\n- [x] [3464](http://www.3464.com/data/Proxy/http/)\n- [x] [coderbusy](https://proxy.coderbusy.com/)\n- [x] [ip181](http://www.ip181.com/)\n- [x] [iphai](http://www.iphai.com/free/ng)\n- [x] [a2u](https://raw.githubusercontent.com/a2u/free-proxy-list/master/free-proxy-list.txt)\n- [x] [coolproxy](https://www.cool-proxy.net/proxies/http_proxy_list/country_code:/port:/anonymous:)\n- [ ] [万能代理](http://wndaili.cn)\n- [ ] [小幻代理](https://ip.ihuan.me) (figuring)\n- [ ] [89免费代理](http://www.89ip.cn/)(figuring)\n- [ ] \u003cdel\u003e[baizhongsou](http://ip.baizhongsou.com/)\u003c/del\u003e (stop providing free proxies)\n\nThanks to: [Golmic](https://github.com/lujqme) [Eric_Chan](https://github.com/CL545740896/)\n\n## FAQ ##\n\n-   **_How about the availability and quality of the proxies?_**\n\n    Before storing new proxy, fp-server will check its availability, anonymity and speed based on your local network. So, feel free to use the crawled proxies.\n\n-   **_How many `PROXY_STORE_NUM` should I set? Is there any limitation?_**\n\n    You should set it depends on your real requirement. If your project is a normal spider, then 300-500 will be fair enough. I haven't set any limitation for now. After stored 10000 available proxies, I stopped testing. The upper limit is relevant to source websites. I will add more websites if more people use this project.\n\n-   **_How to use it in my project?_**\n\n    See the next section.\n\n\n## Examples\n\nThese code can be directly copied to your project. **Remember** to modify the configuration and settings at first.\n\nI will write more snippets at leisure. Or you can tell me what example you want.\n\n### Use fp-server with Python requests module\n\n[Here.](./examples/use_with_requests.md)\n\n### Use fp-server in Scrapy Project\n\nHere is [a middleware for Scrapy](./examples/middleware_for_scrapy.md) to fetch and apply proxy for each request. Copy it to your `middlewares.py` and add the name to `DOWNLOADER_MIDDLEWARES` in your `settings.py`.\n\nIf you want to keep a cookie pool for your proxies(an independent cookiejar for each IP), [this middleware](https://github.com/Karmenzind/IcrisCrawler/blob/master/IcrisCrawler/middlewares.py#L126) may help you.\n\n## Bugs and feature requests ##\n\nI need your feedback to make it better.\u003cbr\u003e\nPlease [create an issue](https://github.com/Karmenzind/fp-server/issues/new) for any problems or advice.\n\nKnown bugs:\n*   Block while using Tornado-4.5.3\n*   Afer check, the redis key might change\n\n## TODOs and ideas ##\n\n*   Use ZSET\n*   Add supervisor\n*   Divide log module\n*   More detailed api\n*   Web frontend via bootstrap\n*   Add user-agent pool\n*   the checker's scheduler: \n    -   Periodically calculating the average speed of checking request, then reassign the checker based on this average and the quantity of stored proxies.\n*   Provide region information. \n*   use redis's HSET for calculation\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKarmenzind%2Ffp-server","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FKarmenzind%2Ffp-server","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKarmenzind%2Ffp-server/lists"}