{"id":13448252,"url":"https://github.com/rootVIII/proxy_requests","last_synced_at":"2025-03-22T09:31:03.353Z","repository":{"id":57455214,"uuid":"143563911","full_name":"rootVIII/proxy_requests","owner":"rootVIII","description":"a class that uses scraped proxies to make http GET/POST requests (Python requests)","archived":true,"fork":false,"pushed_at":"2020-12-03T15:41:38.000Z","size":768,"stargazers_count":388,"open_issues_count":2,"forks_count":43,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-03-20T04:55:26.294Z","etag":null,"topics":["http","http-get","http-getter","http-proxy","http-proxy-middleware","proxy","proxy-list","proxy-requests","proxy-server","python","python-requests","python3","recursion","recursion-problem","requests","requests-module","webscraper","webscraper-api","webscraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rootVIII.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-04T21:23:59.000Z","updated_at":"2025-01-04T20:41:10.000Z","dependencies_parsed_at":"2022-08-26T10:11:17.264Z","dependency_job_id":null,"html_url":"https://github.com/rootVIII/proxy_requests","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootVIII%2Fproxy_requests","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootVIII%2Fproxy_requests/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootVIII%2Fproxy_requests/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootVIII%2Fproxy_requests/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rootVIII","download_url":"https://codeload.github.com/rootVIII/proxy_requests/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244937751,"owners_count":20535124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["http","http-get","http-getter","http-proxy","http-proxy-middleware","proxy","proxy-list","proxy-requests","proxy-server","python","python-requests","python3","recursion","recursion-problem","requests","requests-module","webscraper","webscraper-api","webscraping"],"created_at":"2024-07-31T05:01:40.071Z","updated_at":"2025-03-22T09:31:02.965Z","avatar_url":"https://github.com/rootVIII.png","language":"Python","readme":"## Python Proxy Requests | make an http GET/POST with a proxy scraped from https://www.sslproxies.org/\n\n[![Downloads](https://pepy.tech/badge/proxy-requests)](https://pepy.tech/project/proxy-requests)\n[![Downloads](https://pepy.tech/badge/proxy-requests/month)](https://pepy.tech/project/proxy-requests)\n[![Downloads](https://pepy.tech/badge/proxy-requests/week)](https://pepy.tech/project/proxy-requests)\n\npypi.org: https://pypi.org/project/proxy-requests/\n\u003cbr\u003e\u003cbr\u003e\nThe ProxyRequests class first scrapes proxies from the web. Then it recursively attempts to make a request if the initial request with a proxy is unsuccessful.\n\u003cbr\u003e\u003cbr\u003e\nEither copy the code and put where you want it, or download via pip:\n\u003cbr\u003e\u003cbr\u003e\n\u003ccode\u003epip install proxy-requests\u003c/code\u003e  (or pip3)\n\u003cbr\u003e\n\u003ccode\u003efrom proxy_requests import ProxyRequests\u003c/code\u003e\n\u003cbr\u003e\u003cbr\u003e\nor if you need the Basic Auth subclass as well:\n\u003cbr\u003e\n\u003ccode\u003efrom proxy_requests import ProxyRequests, ProxyRequestsBasicAuth\u003c/code\u003e\n\u003cbr\u003e\u003cbr\u003e\nIf the above import statement is used, method calls will be identical to the ones shown below. Pass a fully qualified URL when initializing an instance.\n\u003cbr\u003e\u003cbr\u003e\nSystem Requirements: \u003cb\u003ePython 3\u003c/b\u003e and the requests module.\n\u003cbr\u003e\u003cbr\u003e\nRuns on Linux and Windows (and Mac probably) - \u003cb\u003eIt may take a moment to run depending on the current proxy.\u003c/b\u003e\n\u003cbr\u003e\nEach request with a proxy is set with an 3  second timeout in the event that the request takes too long (before trying the next proxy socket in the queue).\n\u003cbr\u003e\u003cbr\u003e\nProxies are randomly popped from the queue.\n\u003cbr\u003e\u003cbr\u003e\nThe ProxyRequestBasicAuth subclass has the methods get(), get_with_headers(), post(), post_with_headers(), post_file(), and post_file_with_headers() that will override the Parent methods.\n\u003cbr\u003e\n\n\u003cb\u003eGET:\u003c/b\u003e\n\u003cpre\u003e\n    \u003ccode\u003e\nr = ProxyRequests('https://api.ipify.org')\nr.get()                                                                                                        \n    \u003c/code\u003e                                                                                                    \n\u003c/pre\u003e                                                                                                         \n                                                                                                               \n\u003cb\u003eGET with headers:\u003c/b\u003e                                                                                       \n\u003cpre\u003e                                                                                                          \n    \u003ccode\u003e                                                                                                     \nh = {'User-Agent': 'NCSA Mosaic/3.0 (Windows 95)'}                                                             \nr = ProxyRequests('url here')                                                                                  \nr.set_headers(h)                                                                                               \nr.get_with_headers()                                                                                           \n    \u003c/code\u003e                                                                                                    \n\u003c/pre\u003e                                                                                                         \n                                                                                                               \n\u003cb\u003ePOST:\u003c/b\u003e                                                                                                   \n\u003cpre\u003e                                                                                                          \n    \u003ccode\u003e                                                                                                     \nr = ProxyRequests('url here')                                                                                  \nr.post({'key1': 'value1', 'key2': 'value2'})                                                                   \n    \u003c/code\u003e                                                                                                    \n\u003c/pre\u003e                                                                                                         \n                                                                                                               \n\u003cb\u003ePOST with headers:\u003c/b\u003e\n\u003cpre\u003e\n    \u003ccode\u003e\nr = ProxyRequests('url here')\nr.set_headers({'name': 'rootVIII', 'secret_message': '7Yufs9KIfj33d'})\nr.post_with_headers({'key1': 'value1', 'key2': 'value2'})\n    \u003c/code\u003e\n\u003c/pre\u003e\n\n\u003cb\u003ePOST FILE:\u003c/b\u003e\n\u003cpre\u003e\n    \u003ccode\u003e\nr = ProxyRequests('url here')\nr.set_file('test.txt')\nr.post_file()\n    \u003c/code\u003e\n\u003c/pre\u003e\n\n\u003cb\u003ePOST FILE with headers:\u003c/b\u003e\n\u003cpre\u003e\n    \u003ccode\u003e\nh = {'User-Agent': 'NCSA Mosaic/3.0 (Windows 95)'}\nr = ProxyRequests('url here')\nr.set_headers(h)\nr.set_file('test.txt')\nr.post_file_with_headers()\n    \u003c/code\u003e\n\u003c/pre\u003e\n\n\u003cb\u003eGET with Basic Authentication:\u003c/b\u003e\n\u003cpre\u003e\n    \u003ccode\u003e\nr = ProxyRequestsBasicAuth('url here', 'username', 'password')\nr.get()\n    \u003c/code\u003e\n\u003c/pre\u003e\n\n\u003cb\u003eGET with headers \u0026 Basic Authentication:\u003c/b\u003e\n\u003cpre\u003e\n    \u003ccode\u003e\nh = {'User-Agent': 'NCSA Mosaic/3.0 (Windows 95)'}\nr = ProxyRequestsBasicAuth('url here', 'username', 'password')\nr.set_headers(h)\nr.get_with_headers()\n    \u003c/code\u003e\n\u003c/pre\u003e\n\n\u003cb\u003ePOST with Basic Authentication:\u003c/b\u003e\n\u003cpre\u003e\n    \u003ccode\u003e\nr = ProxyRequestsBasicAuth('url here', 'username', 'password')\nr.post({'key1': 'value1', 'key2': 'value2'})\n    \u003c/code\u003e\n\u003c/pre\u003e\n\n\u003cb\u003ePOST with headers \u0026 Basic Authentication:\u003c/b\u003e\n\u003cpre\u003e\n    \u003ccode\u003e\nr = ProxyRequestsBasicAuth('url here', 'username', 'password')\nr.set_headers({'header_key': 'header_value'})\nr.post_with_headers({'key1': 'value1', 'key2': 'value2'})\n    \u003c/code\u003e\n\u003c/pre\u003e\n\n\u003cb\u003ePOST FILE with Basic Authentication:\u003c/b\u003e\n\u003cpre\u003e\n    \u003ccode\u003e\nr = ProxyRequestsBasicAuth('url here', 'username', 'password')\nr.set_file('test.txt')\nr.post_file()\n    \u003c/code\u003e\n\u003c/pre\u003e\n\n\u003cb\u003ePOST FILE with headers \u0026 Basic Authentication:\u003c/b\u003e\n\u003cpre\u003e\n    \u003ccode\u003e\nh = {'User-Agent': 'NCSA Mosaic/3.0 (Windows 95)'}\nr = ProxyRequestsBasicAuth('url here', 'username', 'password')\nr.set_headers(h)\nr.set_file('test.txt')\nr.post_file_with_headers()\n    \u003c/code\u003e\n\u003c/pre\u003e    \n\u003cbr\u003e\u003cbr\u003e\n\u003cu\u003e\u003cb\u003eResponse Methods\u003c/b\u003e\u003c/u\u003e\n\u003cbr\u003e\u003cbr\u003e\n  \u003cb\u003eReturns a string:\u003c/b\u003e\n\u003cbr\u003e\n\u003ccode\u003eprint(r)\u003c/code\u003e\n\u003cbr\u003e\n\u003cb\u003eOr if you want the raw content as bytes: \u003c/b\u003e\n\u003cbr\u003e\n\u003ccode\u003er.get_raw()\u003c/code\u003e\n\u003cbr\u003e\n\u003cb\u003eGet the response as JSON (if valid JSON):\u003c/b\u003e\n\u003cbr\u003e\n\u003ccode\u003er.get_json()\u003c/code\u003e\n\u003cbr\u003e\n\u003cb\u003eGet the response headers:\u003c/b\u003e\n\u003cbr\u003e\n\u003ccode\u003eprint(r.get_headers())\u003c/code\u003e\n\u003cbr\u003e\n\u003cb\u003eGet the status code:\u003c/b\u003e\n\u003cbr\u003e\n\u003ccode\u003eprint(r.get_status_code())\u003c/code\u003e\n\u003cbr\u003e\n\u003cb\u003eGet the URL that was requested:\u003c/b\u003e\n\u003cbr\u003e\n\u003ccode\u003eprint(r.get_url())\u003c/code\u003e\n\u003cbr\u003e\n\u003cb\u003eGet the proxy that was used to make the request:\u003c/b\u003e\n\u003cbr\u003e\n\u003ccode\u003eprint(r.get_proxy_used())\u003c/code\u003e\n\u003cbr\u003e\n\u003cbr\u003e\n\u003cb\u003eTo write raw data to a file (including an image):\u003c/b\u003e\n\u003cbr\u003e\n\u003cpre\u003e\n    \u003ccode\u003e\n\nurl = 'https://www.restwords.com/static/ICON.png'\nr = ProxyRequests(url)\nr.get()\nwith open('out.png', 'wb') as f:\n    f.write(r.get_raw())\n\n    \u003c/code\u003e\n\u003c/pre\u003e\n\u003cbr\u003e\n\u003cb\u003eDump the response to a file as JSON:\u003c/b\u003e\n\u003cbr\u003e\n\u003cpre\u003e\n    \u003ccode\u003e\nimport json\nwith open('test.txt', 'w') as f:\n    json.dump(r.get_json(), f)\n    \u003c/code\u003e\n\u003c/pre\u003e\n\u003cbr\u003e\u003cbr\u003e\nThis was developed on Ubuntu 16.04.4/18.04 LTS.\n\u003chr\u003e\n\u003cb\u003eAuthor: rootVIII  2018-2020\u003c/b\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cimg src=\"https://github.com/rootVIII/proxy_requests/blob/master/ex1.png\" alt=\"example1\" height=\"675\" width=\"950\"\u003e\u003chr\u003e\n\u003cimg src=\"https://github.com/rootVIII/proxy_requests/blob/master/ex2.png\" alt=\"example1\" height=\"675\" width=\"950\"\u003e\u003chr\u003e\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FrootVIII%2Fproxy_requests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FrootVIII%2Fproxy_requests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FrootVIII%2Fproxy_requests/lists"}