{"id":28998697,"url":"https://github.com/luminati-io/httpx-with-proxy","last_synced_at":"2025-06-25T07:09:45.329Z","repository":{"id":279025229,"uuid":"934843017","full_name":"luminati-io/httpx-with-proxy","owner":"luminati-io","description":"Implement various proxy configurations with HTTPX—covering unauthenticated, authenticated, rotating, and fallback proxies.","archived":false,"fork":false,"pushed_at":"2025-02-23T08:32:44.000Z","size":678,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-22T07:02:01.335Z","etag":null,"topics":["asyncio","beautifulsoup","data-collection","datacenter-proxy","httpx","proxies","proxy-server","python","random","rotating-proxy","web-scraping"],"latest_commit_sha":null,"homepage":"https://brightdata.com/blog/proxy-101/httpx-with-proxies","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luminati-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-18T13:49:11.000Z","updated_at":"2025-02-23T08:32:47.000Z","dependencies_parsed_at":"2025-02-23T09:36:34.429Z","dependency_job_id":null,"html_url":"https://github.com/luminati-io/httpx-with-proxy","commit_stats":null,"previous_names":["luminati-io/httpx-with-proxy"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/luminati-io/httpx-with-proxy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2Fhttpx-with-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2Fhttpx-with-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2Fhttpx-with-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2Fhttpx-with-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luminati-io","download_url":"https://codeload.github.com/luminati-io/httpx-with-proxy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2Fhttpx-with-proxy/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261823775,"owners_count":23215150,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asyncio","beautifulsoup","data-collection","datacenter-proxy","httpx","proxies","proxy-server","python","random","rotating-proxy","web-scraping"],"created_at":"2025-06-25T07:09:32.643Z","updated_at":"2025-06-25T07:09:45.301Z","avatar_url":"https://github.com/luminati-io.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Using Proxies with HTTPX\n\n[![Promo](https://github.com/luminati-io/Rotating-Residential-Proxies/blob/main/50%25%20off%20promo.png)](https://brightdata.com/proxy-types/residential-proxies) \n\nThis guide explains how to use proxies with HTTPX, with examples for unauthenticated, authenticated, rotating, and fallback proxies.\n\n## Using Unauthenticated Proxies\n\nWith an unauthenticated proxy, we’re not using a username or password, and all requests go to a `proxy_url`. Below is a code snippet that uses an unauthenticated proxy:\n\n```python\nimport httpx\n\nproxy_url = \"http://localhost:8030\"\n\n\nwith httpx.Client(proxy=proxy_url) as client:\n    ip_info = client.get(\"https://geo.brdtest.com/mygeo.json\")\n    print(ip_info.text)\n```\n\n## Using Authenticated Proxies\n\nAuthenticated proxies require a username and password. Once you submit correct credentials, you are granted connection to the proxy.\n\nWith authentication, the `proxy_url` looks like this: `http://\u003cusername\u003e:\u003cpassword\u003e@\u003cproxy_url\u003e:\u003cport_number\u003e`. The following example demonstrates how to construct the user portion of the authentication string using both `zone` and `username`. It also utilizes [Bright Data's datacenter proxies](https://brightdata.com/proxy-types/datacenter-proxies) as the base connection.\n\n```python\nimport httpx\n\nusername = \"your-username\"\nzone = \"your-zone-name\"\npassword = \"your-password\"\n\nproxy_url = \"http://brd-customer-{username}-zone-{zone}:{password}@brd.superproxy.io:33335\"\n\nip_info = httpx.get(\"https://geo.brdtest.com/mygeo.json\", proxy=proxy_url)\n\nprint(ip_info.text)\n```\n\nHere is the breakdown of the above code:\n\n- We start with creating config variables: `username`, `zone`, and `password`.\n- We use those to create our `proxy_url`: `\"http://brd-customer-{username}-zone-{zone}:{password}@brd.superproxy.io:33335\"`.\n- We send a request to the API to retrieve general information about our proxy connection.\n\nThe response should look like this.\n\n```json\n{\"country\":\"US\",\"asn\":{\"asnum\":20473,\"org_name\":\"AS-VULTR\"},\"geo\":{\"city\":\"\",\"region\":\"\",\"region_name\":\"\",\"postal_code\":\"\",\"latitude\":37.751,\"longitude\":-97.822,\"tz\":\"America/Chicago\"}}\n\n```\n\n## Using Rotating Proxies\n\nRotating proxies means creating a list of proxies and choosing from them randomly. The code snippet below creates a list of `countries`, then uses `random.choice()` on each request to pick a random country from the list. Our `proxy_url` gets formatted to fit the country. The list of [rotating proxies](https://brightdata.com/solutions/rotating-proxies) in the code is from Bright Data.\n\n```python\nimport httpx\nimport asyncio\nimport random\n\n\ncountries = [\"us\", \"gb\", \"au\", \"ca\"]\nusername = \"your-username\"\nproxy_url = \"brd.superproxy.io:33335\"\n\ndatacenter_zone = \"your-zone\"\ndatacenter_pass = \"your-password\"\n\n\nfor random_proxy in countries:\n    print(\"----------connection info-------------\")\n    datacenter_proxy = f\"http://brd-customer-{username}-zone-{datacenter_zone}-country-{random.choice(countries)}:{datacenter_pass}@{proxy_url}\"\n\n    ip_info = httpx.get(\"https://geo.brdtest.com/mygeo.json\", proxy=datacenter_proxy)\n\n    print(ip_info.text)\n```\n\nThe code is very similar to the one for authenticated proxies. The key differences are:\n\n- We create an array of countries: `[\"us\", \"gb\", \"au\", \"ca\"]`.\n- We make multiple requsts instead of a single one. For each request, we use `random.choice(countries)` to choose a random country every time we create the `proxy_url`.\n\n## Creating a Fallback Proxy Connection\n\nAll examples above use datacenter and free proxies. The former often get blocked by websites, the latter isn't very reliable. For all this to work, there should be a fallback to residential proxies.\n\nTo do that, let's create a function called `safe_get()`. When we call it, we first try to get the url using a datacenter connection. When this fails, we _fall back_ to our residential connection.\n\n```python\nimport httpx\nfrom bs4 import BeautifulSoup\nimport asyncio\n\n\ncountry = \"us\"\nusername = \"your-username\"\nproxy_url = \"brd.superproxy.io:33335\"\n\ndatacenter_zone = \"datacenter_proxy1\"\ndatacenter_pass = \"datacenter-password\"\n\nresidential_zone = \"residential_proxy1\"\nresidential_pass = \"residential-password\"\n\ncert_path = \"/home/path/to/brightdata_proxy_ca/New SSL certifcate - MUST BE USED WITH PORT 33335/BrightData SSL certificate (port 33335).crt\"\n\n\ndatacenter_proxy = f\"http://brd-customer-{username}-zone-{datacenter_zone}-country-{country}:{datacenter_pass}@{proxy_url}\"\n\nresidential_proxy = f\"http://brd-customer-{username}-zone-{residential_zone}-country-{country}:{residential_pass}@{proxy_url}\"\n\nasync def safe_get(url: str):\n    async with httpx.AsyncClient(proxy=datacenter_proxy) as client:\n        print(\"trying with datacenter\")\n        response = await client.get(url)\n        if response.status_code == 200:\n            soup = BeautifulSoup(response.text, \"html.parser\")\n            if not soup.select_one(\"form[action='/errors/validateCaptcha']\"):\n                print(\"response successful\")\n                return response\n    print(\"response failed\")\n    async with httpx.AsyncClient(proxy=residential_proxy, verify=cert_path) as client:\n        print(\"trying with residential\")\n        response = await client.get(url)\n        print(\"response successful\")\n        return response\n\nasync def main():\n    url = \"https://www.amazon.com\"\n    response = await safe_get(url)\n    with open(\"out.html\", \"w\") as file:\n        file.write(response.text)\n\nasyncio.run(main())\n```\n\nHere is the code breakdown:\n\n- We now have two sets of configuration variables: one for our datacenter connection and another for our residential connection.\n- This time, we use an `AsyncClient()` session to introduce some of the more advanced functionality of HTTPX.\n- First, we attempt to make our request with the `datacenter_proxy`.\n- If we fail to get a proper response, we retry the request using the `residential_proxy`. Also note the `verify` flag in the code. When using Bright Data's residential proxies, you need to download and use the [SSL certificate](https://docs.brightdata.com/general/account/ssl-certificate).\n- Once we’ve got a solid response, we write the page to an HTML file. We can open this page up in a browser and see what the proxy actually accessed and sent back to us.\n\nAfter running the code above your output and resulting HTML file should look like this.\n\n```\ntrying with datacenter\nresponse failed\ntrying with residential\nresponse successful\n```\n\n![Screenshot of the Amazon homepage](https://github.com/luminati-io/httpx-with-proxy/blob/main/Images/image.png)\n\n## Conclusion\n\nWhen you combine HTTPX with [Bright Data's top-tier proxy services](https://brightdata.com/proxy-types), you get a private, efficient, and reliable way to scrape the web. Start your free trial with Bright Data’s proxies today!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluminati-io%2Fhttpx-with-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluminati-io%2Fhttpx-with-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluminati-io%2Fhttpx-with-proxy/lists"}