{"id":18487885,"url":"https://github.com/iw4p/proxy-scraper","last_synced_at":"2025-04-08T20:32:03.484Z","repository":{"id":37257622,"uuid":"287008366","full_name":"iw4p/proxy-scraper","owner":"iw4p","description":"scrape proxies from more than 5 different sources and check which ones are still alive","archived":false,"fork":false,"pushed_at":"2024-08-16T21:01:14.000Z","size":69,"stargazers_count":547,"open_issues_count":2,"forks_count":138,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-08T19:03:44.360Z","etag":null,"topics":["checker","grabber","http","https-proxy","proxies","proxy","proxy-checker","proxy-list","proxy-scraper","proxychecker","scraper","scrapper","socks","socks-proxy","socks4","socks5","socks5-proxy"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iw4p.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-12T12:30:00.000Z","updated_at":"2025-03-08T14:30:30.000Z","dependencies_parsed_at":"2024-11-06T12:56:29.612Z","dependency_job_id":"41a5c7b2-ac20-4705-9a7a-a54841e9a950","html_url":"https://github.com/iw4p/proxy-scraper","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iw4p%2Fproxy-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iw4p%2Fproxy-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iw4p%2Fproxy-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iw4p%2Fproxy-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iw4p","download_url":"https://codeload.github.com/iw4p/proxy-scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247923151,"owners_count":21018937,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["checker","grabber","http","https-proxy","proxies","proxy","proxy-checker","proxy-list","proxy-scraper","proxychecker","scraper","scrapper","socks","socks-proxy","socks4","socks5","socks5-proxy"],"created_at":"2024-11-06T12:50:56.262Z","updated_at":"2025-04-08T20:31:58.464Z","avatar_url":"https://github.com/iw4p.png","language":"Python","readme":"# Proxy Scraper and Checker\n\n[![Tests](https://github.com/iw4p/proxy-scraper/actions/workflows/tests.yml/badge.svg)](https://github.com/iw4p/proxy-scraper/actions/workflows/tests.yml)\n[![Downloads](https://static.pepy.tech/badge/proxyz)](https://pepy.tech/project/proxyz)\n\nScrape more than 1K HTTP - HTTPS - SOCKS4 - SOCKS5 proxies in less than 2 seconds.\n\nScraping fresh public proxies from different sources:\n\n- [sslproxies.org](http://sslproxies.org) (HTTP, HTTPS)\n- [free-proxy-list.net](http://free-proxy-list.net) (HTTP, HTTPS)\n- [us-proxy.org](http://us-proxy.org) (HTTP, HTTPS)\n- [socks-proxy.net](http://socks-proxy.net) (Socks4, Socks5)\n- [proxyscrape.com](https://proxyscrape.com) (HTTP, Socks4, Socks5)\n- [proxy-list.download](https://www.proxy-list.download) (HTTP, HTTPS, Socks4, Socks5)\n- [geonode.com](https://geonode.com) (HTTP, HTTPS, Socks4, Socks5)\n\n## Installation\n\nYou can install the package directly from PyPI using `pip`:\n\n```bash\npip install proxyz\n```\n\nAlternatively, you can install dependencies manually if you're working from the source code:\n\n```bash\npip3 install -r requirements.txt\n```\n\n## Usage\n\n### Using the Command-Line Interface\n\nOnce installed via `pip`, you can use the command-line tools `proxy_scraper` and `proxy_checker` directly.\n\n#### For Scraping Proxies:\n\n```bash\nproxy_scraper -p http\n```\n\n- With `-p` or `--proxy`, you can choose your proxy type. Supported proxy types are: **HTTP - HTTPS - Socks (Both 4 and 5) - Socks4 - Socks5**.\n- With `-o` or `--output`, specify the output file name where the proxies will be saved. (Default is **output.txt**).\n- With `-v` or `--verbose`, increase output verbosity.\n- With `-h` or `--help`, show the help message.\n\n#### For Checking Proxies:\n\n```bash\nproxy_checker -p http -t 20 -s https://google.com -l output.txt\n```\n\n- With `-t` or `--timeout`, set the timeout in seconds after which the proxy is considered dead. (Default is **20**).\n- With `-p` or `--proxy`, check HTTPS, HTTP, SOCKS4, or SOCKS5 proxies. (Default is **HTTP**).\n- With `-l` or `--list`, specify the path to your proxy list file. (Default is **output.txt**).\n- With `-s` or `--site`, check proxies against a specific website like google.com. (Default is **https://google.com**).\n- With `-r` or `--random_agent`, use a random user agent per proxy.\n- With `-v` or `--verbose`, increase output verbosity.\n- With `-h` or `--help`, show the help message.\n\n### Running Directly from Source\n\nIf you prefer running the scripts directly from the source code, you can use the following commands:\n\n#### For Scraping:\n\n```bash\npython3 proxyScraper.py -p http\n```\n\n#### For Checking:\n\n```bash\npython3 proxyChecker.py -p http -t 20 -s https://google.com -l output.txt\n```\n\n## Good to Know\n\n- Dead proxies will be removed, and only alive proxies will remain in the output file.\n- This script is capable of scraping SOCKS proxies, but `proxyChecker` currently only checks HTTP(S) proxies.\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=iw4p/proxy-scraper\u0026type=Date)](https://star-history.com/#iw4p/proxy-scraper\u0026Date)\n\n## Contributing\n\nPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.\n\n### Issues\n\nFeel free to submit issues and enhancement requests or contact me via [vida.page/nima](https://vida.page/nima).\n\n## License\n\n[MIT](https://choosealicense.com/licenses/mit/)\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiw4p%2Fproxy-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiw4p%2Fproxy-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiw4p%2Fproxy-scraper/lists"}