{"id":19261490,"url":"https://github.com/wangyihang/proxy-verifier","last_synced_at":"2025-04-21T17:30:42.718Z","repository":{"id":156282985,"uuid":"621600353","full_name":"WangYihang/Proxy-Verifier","owner":"WangYihang","description":"A set of tools designed to efficiently and effectively locate publicly available proxy server resources.","archived":false,"fork":false,"pushed_at":"2023-08-21T09:10:16.000Z","size":72,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-01T14:45:59.799Z","etag":null,"topics":["http-proxy","proxy","proxy-checker","proxy-list","socks4","socks4a","socks5","transparent-http-proxy"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WangYihang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-03-31T01:58:45.000Z","updated_at":"2023-09-23T06:12:57.000Z","dependencies_parsed_at":"2024-02-02T11:01:46.300Z","dependency_job_id":"4a210155-d5e6-4528-8b1a-a957b2ff232e","html_url":"https://github.com/WangYihang/Proxy-Verifier","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WangYihang%2FProxy-Verifier","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WangYihang%2FProxy-Verifier/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WangYihang%2FProxy-Verifier/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WangYihang%2FProxy-Verifier/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WangYihang","download_url":"https://codeload.github.com/WangYihang/Proxy-Verifier/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250100214,"owners_count":21374898,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["http-proxy","proxy","proxy-checker","proxy-list","socks4","socks4a","socks5","transparent-http-proxy"],"created_at":"2024-11-09T19:27:12.640Z","updated_at":"2025-04-21T17:30:42.481Z","avatar_url":"https://github.com/WangYihang.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Proxy Verifier\n\n\u003e Read the project `README.md` in other languages: [English](README.en.md), [简体中文](README.zh.md)\n\nMany organizations on the Internet regularly publish proxy server lists, such as [TheSpeedX/PROXY-List](https://github.com/TheSpeedX/PROXY-List) repository, etc. The authors of these repositories regularly update the latest proxy server lists, and we thank them for their outstanding work. However, **not all of these proxies are available**. For example, some require a username and password, and some proxies will expire in a short time.\n\nThis project can help you **quickly obtain public and genuinely available proxy server resources** on the Internet, including HTTP proxies, HTTP tunnels, transparent HTTP proxies, Socks4 proxies, Socks4a proxies, and Socks5 proxies.\n\nThis project is divided into 4 parts, or four command-line tools, which are `downloader`, `verifier`, `server`, and `exporter`.\n\n1. `downloader`: Download real-time updated raw proxy files from the repository and merge them;\n2. `server`: An HTTP server that helps the `verifier` determine if the proxy server is available. It will accept a random string parameter passed in by the verifier, perform a hash operation on the parameter, and return the result as an HTTP response to ensure that the proxy server is indeed accessed by the `verifier` and that there is no cache server on the link;\n3. `verifier`: Verify whether the proxy servers downloaded by the `downloader` are available. Specifically, the verifier will connect to a controlled HTTP server through the proxy, and judge whether the proxy is available by judging whether the server's response is valid;\n4. `exporter`: Read the log files of the `verifier` and export the list of available proxy servers from them.\n\nIt is easy to see that the `downloader` has no input (the download address list is built into the tool), the output of the `downloader` is the input of the `verifier`, the output of the `verifier` is the input of the `exporter`, and the output of the `exporter` is **the list of available proxy servers** filtered by this project for you.\n\n## Installation\n\n```bash\ngo install github.com/WangYihang/Proxy-Verifier/cmd/downloader@latest\ngo install github.com/WangYihang/Proxy-Verifier/cmd/exporter@latest\ngo install github.com/WangYihang/Proxy-Verifier/cmd/server@latest\ngo install github.com/WangYihang/Proxy-Verifier/cmd/verifier@latest\n```\n\n## Usage\n\n### Downloader\n\n```bash\nUsage:\n  main [OPTIONS]\n\nApplication Options:\n  -i, --input-file=  The input file in yaml format (default: -)\n  -o, --output-file= The output file (default: -)\n  -n, --num-workers= Number of workers (default: 4)\n  -m, --max-retries= Maximum number of retries (default: 3)\n\nHelp Options:\n  -h, --help         Show this help message\n```\n\n### Server\n\n```bash\nUsage:\n  main [OPTIONS]\n\nApplication Options:\n  -b, --bind-host=    The host to bind (default: 127.0.0.1)\n  -p, --bind-port=    The port to bind (default: 80)\n  -l, --log-filename= The filename to log to (default: gin.log)\n  -s, --secret=       The secret used to verify the integrity of the proxy (default:\n                      2d7c29dd-cecb-4454-a4ec-ae2734771a60)\n\nHelp Options:\n  -h, --help          Show this help message\n```\n\n### Verifier\n\n```bash\nUsage:\n  main [OPTIONS]\n\nApplication Options:\n  -i, --input-file=       The input file\n  -o, --output-file=      The output file\n  -u, --url=              The target URL to connect through the proxy, e.g., http://www.google.com,\n                          smtp://mails.tsinghua.edu.cn\n  -t, --timeout=          Timeout in seconds (default: 16)\n  -n, --num-workers=      Number of workers (default: 256)\n  -m, --monitor-interval= Interval to output the current running state (in seconds) (default: 1)\n  -v, --verbose           Show verbose debug information\n  -d, --measurement-id=   The measurement ID used to seperate different measurements in logs\n  -s, --secret=           The secret used to verify the integrity of the proxy (default:\n                          2d7c29dd-cecb-4454-a4ec-ae2734771a60)\n\nHelp Options:\n  -h, --help              Show this help message\n```\n\n### Exporter\n\n```bash\nUsage:\n  main [OPTIONS]\n\nApplication Options:\n  -i, --input-file=        The input file\n  -o, --output-file=       The output file\n  -r, --require-identical  If provided, the frontend IP and backend IP are required to be identical\n\nHelp Options:\n  -h, --help               Show this help message\n```\n\n## Example\n\n0. Define output and input filenames as environment variables\n\n```bash\nTODAY=$(date -u '+%Y-%m-%d')\nFREE_OPEN_PROXIES_FILEPATH=\"free-open-proxies-v${TODAY}.txt\"\nFREE_OPEN_PROXIES_LOG_FILEPATH=\"free-open-proxies-v${TODAY}.log\"\nAVAILABLE_FREE_OPEN_PROXIES_FILEPATH=\"available-free-open-proxies-v${TODAY}.txt\"\n```\n\n1. First, download the proxy list using the `downloader`.\n\n```bash\n$ # Download the default proxies source lists yaml file\n$ wget https://raw.githubusercontent.com/WangYihang/Proxy-Verifier/main/sources.yaml\n$ ./downloader --input-file sources.yaml --output-file ${FREE_OPEN_PROXIES_FILEPATH}\n```\n\n2. Next, start the `server` on a machine with a public IP (e.g., 1.2.3.4).\n\n```bash\n$ ./server -b 0.0.0.0 -p 80\n```\n\n3. Once again, start the `verifier` to verify the availability of the proxies.\n\n```bash\n$ ./verifier \\\n    --input-file ${FREE_OPEN_PROXIES_FILEPATH} \\\n    --output-file ${FREE_OPEN_PROXIES_LOG_FILEPATH} \\\n    --url http://1.2.3.4:80/ \\\n    --num-workers 1024 \\\n    --timeout 8\n```\n\n4. Finally, export the available proxy servers using the `exporter`.\n\n```bash\n$ ./exporter \\\n    --input-file ${FREE_OPEN_PROXIES_LOG_FILEPATH} \\\n    --output-file ${AVAILABLE_FREE_OPEN_PROXIES_FILEPATH}\nhttp://20.210.113.32:8123/\nhttp://113.254.50.31:80/\n...\nhttp://113.252.10.120:8118/\nhttp://121.132.95.7:80/\n```\n\n## Limitations\n\nDue to the proxy data sources of this project coming from other projects, it is not possible to completely obtain all the proxy server lists on the entire internet.\n\n## To-Do\n\n- [x] Support HTTPS proxies\n- [x] Support transparent HTTPS proxies\n- [x] Support HTTPS tunnels\n- [x] Allow the downloader to read proxy list download links through a configuration file\n\n## Acknowledgement\n\n* [clarketm/proxy-list](https://github.com/clarketm/proxy-list)\n* [jetkai/proxy-list](https://github.com/jetkai/proxy-list)\n* [mertguvencli/http-proxy-list](https://github.com/mertguvencli/http-proxy-list)\n* [monosans/proxy-list](https://github.com/monosans/proxy-list)\n* [MuRongPIG/Proxy-Master](https://github.com/MuRongPIG/Proxy-Master)\n* [proxylist-to/proxy-list](https://github.com/proxylist-to/proxy-list)\n* [prxchk/proxy-list](https://github.com/prxchk/proxy-list)\n* [roosterkid/openproxylist](https://github.com/roosterkid/openproxylist)\n* [TheSpeedX/PROXY-List](https://github.com/TheSpeedX/PROXY-List)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwangyihang%2Fproxy-verifier","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwangyihang%2Fproxy-verifier","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwangyihang%2Fproxy-verifier/lists"}