{"id":28998656,"url":"https://github.com/luminati-io/cloudscraper-with-proxies","last_synced_at":"2025-06-25T07:09:33.059Z","repository":{"id":275742823,"uuid":"916077396","full_name":"luminati-io/Cloudscraper-with-proxies","owner":"luminati-io","description":"Guide to integrating proxies with CloudScraper, covering proxy setup, rotation, authenticated proxies, and premium proxy integration","archived":false,"fork":false,"pushed_at":"2025-02-04T10:17:48.000Z","size":203,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-22T07:02:01.037Z","etag":null,"topics":["cloudscraper","datacenter-proxy","isp-proxies","proxy-server","python","residential-proxy","rotating-proxy","static-residential-proxies"],"latest_commit_sha":null,"homepage":"https://brightdata.com/blog/proxy-101/cloudscraper-with-proxies","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luminati-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-13T12:15:46.000Z","updated_at":"2025-02-04T10:17:51.000Z","dependencies_parsed_at":"2025-02-04T11:34:33.717Z","dependency_job_id":null,"html_url":"https://github.com/luminati-io/Cloudscraper-with-proxies","commit_stats":null,"previous_names":["luminati-io/cloudscraper-with-proxies"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/luminati-io/Cloudscraper-with-proxies","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2FCloudscraper-with-proxies","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2FCloudscraper-with-proxies/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2FCloudscraper-with-proxies/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2FCloudscraper-with-proxies/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luminati-io","download_url":"https://codeload.github.com/luminati-io/Cloudscraper-with-proxies/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2FCloudscraper-with-proxies/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261823775,"owners_count":23215150,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudscraper","datacenter-proxy","isp-proxies","proxy-server","python","residential-proxy","rotating-proxy","static-residential-proxies"],"created_at":"2025-06-25T07:09:32.345Z","updated_at":"2025-06-25T07:09:33.017Z","avatar_url":"https://github.com/luminati-io.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Using CloudScraper with Proxies\n\n[![Promo](https://github.com/luminati-io/LinkedIn-Scraper/raw/main/Proxies%20and%20scrapers%20GitHub%20bonus%20banner.png)](https://brightdata.com/) \n\nThis guide covers setting up a CloudScraper proxy integration, rotating IPs, and using authenticated proxies for seamless scraping.\n\n- [What Is CloudScraper?](#about-cloudscraper)\n- [Why Use Proxies with CloudScraper?](#why-use-proxies-with-cloudscraper)\n- [Setting Up a Proxy With CloudScraper](#setting-up-a-proxy-with-cloudscraper)\n- [Implementing Proxy Rotation](#implementing-proxy-rotation)\n- [Using Authenticated Proxies in CloudScraper](#using-authenticated-proxies-in-cloudscraper)\n- [Integrating Premium Proxies in CloudScraper](#integrating-premium-proxies-in-cloudscraper)\n- [Conclusion](#conclusion)\n\n## About CloudScraper\n\n[CloudScraper](https://github.com/VeNoMouS/cloudscraper) is a Python module designed to bypass Cloudflare's anti-bot page (commonly known as \"I'm Under Attack Mode\" or IUAM). Under the hood, it is implemented using Requests, one of the most popular Python HTTP clients.\n\n## Why Use Proxies with CloudScraper?\n\nCloudflare may block your IP if you make too many requests or trigger more sophisticated defenses that are difficult to bypass. The combination of proxies and CloudScraper for scraping websites hosted by Cloudflare offers two key benefits:\n\n- **Enhanced security and anonymity**: By routing requests through a proxy, your true identity remains hidden, reducing the risk of detection.\n- **Avoiding blocks and interruptions**: Proxies allow you to rotate IP addresses dynamically, which helps you bypass blocks and rate limiters.\n\n## Setting Up a Proxy With CloudScraper\n\n### Step #1: Install CloudScraper\n\nInstall the `cloudscraper` pip package:\n\n```bash\npip install -U cloudscraper\n```\n\nThe `-U` option ensures that you are getting the latest version of the package with the latest workarounds for Cloudflare's anti-bot engine.\n\n### Step #2: Initialize CloudScraper\n\nImport CloudScraper:\n\n```python\nimport cloudscraper\n```\n\nCreate a CloudScraper instance using the `create_scraper()` method:\n\n```python\nscraper = cloudscraper.create_scraper()\n```\n\nThe `scraper` object works similarly to the `Session` object from the `requests` library. In particular, it enables you to make HTTP requests while bypassing Cloudflare's anti-bot measures.\n\n### Step #3: Integrate a Proxy\n\nDefine a `proxies` dictionary and pass it to the `get()` method as below:\n\n```python\nproxies = {\n    \"http\": \"\u003cYOUR_HTTP_PROXY_URL\u003e\",\n    \"https\": \"\u003cYOUR_HTTPS_PROXY_URL\u003e\"\n}\n\n# Perform a request through the specified proxy\nresponse = scraper.get(\"\u003cYOUR_TARGET_URL\u003e\", proxies=proxies)\n```\n\nThe `proxies` parameter in the `get()` method is passed down to Requests. This allows the HTTP client to route your request through the specified HTTP or HTTPS proxy server, depending on the protocol of your target URL.\n\n### Step #4: Test the CloudScraper Proxy Integration Setup\n\nFor demonstration purposes, let's target the `/ip` endpoint of the HTTPBin project. This endpoint returns the caller's IP address. If everything works as expected, the response should display the IP address of the proxy server.\n\nAssuming that the URL for the proxy server is `http://202.159.35.121:443`, this will be the script code:\n\n```python\nimport cloudscraper\n\n# Create a CloudScraper instance\nscraper = cloudscraper.create_scraper()\n\n# Specify your proxy\nproxies = {\n    \"http\": \"http://202.159.35.121:443\",\n    \"https\": \"http://202.159.35.121:443\"\n}\n\n# Make a request through the proxy\nresponse = scraper.get(\"https://httpbin.org/ip\", proxies=proxies)\n\n# Print the response from the \"/ip\" endpoint\nprint(response.text)\n```\n\nYou should see a response like this:\n\n```json\n{\n    \"origin\": \"202.159.35.121\"\n}\n```\n\nThe IP in the response matches the IP of the proxy server, as expected.\n\n\u003e **Note**:\\\n\u003e Free proxy servers are often short-lived. It's best to obtain a new IP address for a proxy when testing the script.\n\n## Implementing Proxy Rotation\n\nRetrieve a list of proxies from a reliable provider and store them in an array:\n\n```python\nproxy_list = [\n    {\"http\": \"\u003cYOUR_PROXY_URL_1\u003e\", \"https\": \"\u003cYOUR_PROXY_URL_1\u003e\"},\n    # ...\n    {\"http\": \"\u003cYOUR_PROXY_URL_n\u003e\", \"https\": \"\u003cYOUR_PROXY_URL_n\u003e\"},\n]\n```\n\nNext, use the `random.choice()` method to randomly select a proxy from the list:\n\n```python\nimport random\n\nrandom_proxy = random.choice(proxy_list)\n```\n\nSet the randomly selected proxy in the `get()` request:\n\n```python\nresponse = scraper.get(\"\u003cYOUR_TARGET_URL\u003e\", proxies=random_proxy)\n```\n\nIf everything is set up correctly, the request will use a different proxy from the list at each run. Here is the complete code:\n\n```python\nimport cloudscraper\nimport random\n\n# Create a Cloudscraper instance\nscraper = cloudscraper.create_scraper()\n\n# List of proxy URLs (replace with actual proxy URLs)\nproxy_list = [\n    {\"http\": \"\u003cYOUR_PROXY_URL_1\u003e\", \"https\": \"\u003cYOUR_PROXY_URL_1\u003e\"},\n    # ...\n    {\"http\": \"\u003cYOUR_PROXY_URL_n\u003e\", \"https\": \"\u003cYOUR_PROXY_URL_n\u003e\"},\n]\n\n# Randomly select a proxy from the list\nrandom_proxy = random.choice(proxy_list)\n\n# Make a request using the randomly selected proxy\n# (replace with the actual target URL)\nresponse = scraper.get(\"\u003cYOUR_TARGET_URL\u003e\", proxies=random_proxy)\n```\n\n## Using Authenticated Proxies in CloudScraper\n\nTo authenticate a proxy in CloudScraper, include the required credentials directly in the proxy URL. The format for username and password authentication is as follows:\n\n`\u003cPROXY_PROTOCOL\u003e://\u003cYOUR_USERNAME\u003e:\u003cYOUR_PASSWORD\u003e@\u003cPROXY_IP_ADDRESS\u003e:\u003cPROXY_PORT\u003e`\n    \nWith that format, the CloudScraper proxy configuration would look like this:\n\n```python\nimport cloudscraper\n\n# Create a Cloudscraper instance\nscraper = cloudscraper.create_scraper()  \n\n# Define your authenticated proxy\nproxies = {\n   \"http\": \"\u003cPROXY_PROTOCOL\u003e://\u003cYOUR_USERNAME\u003e:\u003cYOUR_PASSWORD\u003e@\u003cPROXY_IP_ADDRESS\u003e:\u003cPROXY_PORT\u003e\",\n   \"https\": \"\u003cPROXY_PROTOCOL\u003e://\u003cYOUR_USERNAME\u003e:\u003cYOUR_PASSWORD\u003e@\u003cPROXY_IP_ADDRESS\u003e:\u003cPROXY_PORT\u003e\"\n}\n\n# Perform a request through the specified authenticated proxy\nresponse = scraper.get(\"\u003cYOUR_TARGET_URL\u003e\", proxies=proxies)\n```\n\n## Integrating Premium Proxies in CloudScraper\n\nFor reliable results in production scraping environments, use proxies from top-tier providers like [Bright Data](https://brightdata.com/). To integrate Bright Data’s proxies in CloudScraper:\n\n1. Create an account or log in.\n\n2. Reach the dashboard and click on the “Residential” zone in the table:\n\n![Bright Data's proxies and scraping infrastructure control panel](https://github.com/luminati-io/Cloudscraper-with-proxies/blob/main/image-7.png)\n\n3. Activate the proxies by clicking the toggle:\n\n![Turning on the residential zone](https://github.com/luminati-io/Cloudscraper-with-proxies/blob/main/image-8.png)\n\nThis is what you should now be seeing:\n\n![The residential zone turned on](https://github.com/luminati-io/Cloudscraper-with-proxies/blob/main/image-9.png)\n\u003e **Note**:\\\n\u003e Bright Data’s residential proxies rotate automatically.\n\n4. In the “Access Details” section, copy the proxy host, username, and password:\n\n![The access details for your residential proxies zone](https://github.com/luminati-io/Cloudscraper-with-proxies/blob/main/image-10.png)\nYour Bright Data proxy URL will look like this:\n\n```\nhttp://\u003cPROXY_USERNAME\u003e:\u003cPROXY_PASSWORD\u003e@brd.superproxy.io:33335\n```\n\n5. Integrate the proxy into Cloudscraper as follows:\n\n```python\nimport cloudscraper\n# Create a CloudScraper instance\nscraper = cloudscraper.create_scraper()\n# Define the premium proxy\nproxies = {\n\"http\": \"http://\u003cPROXY_USERNAME\u003e:\u003cPROXY_PASSWORD\u003e@\u003cPROXY_HOST\u003e:\u003cPROXY_PORT\u003e\",\n\"https\": \"http://\u003cPROXY_USERNAME\u003e:\u003cPROXY_PASSWORD\u003e@\u003cPROXY_HOST\u003e:\u003cPROXY_PORT\u003e\"\n}\n# Perform a request using the premium proxy\nresponse = scraper.get(\"https://httpbin.org/ip\", proxies=proxies)\n# Print the response to verify the proxy is working\nprint(response.text)\n```\n\nThe CloudScraper proxy integration is done. Now, we need to test and verify. To ensure the proxy is working correctly, you can test it against a service like [https://httpbin.org/ip](https://httpbin.org/ip), which returns the IP address of the caller. If the setup is correct, the response should display the IP address of the proxy server instead of your local IP.\n\n\n## Putting Everything Together \n\n```python\nimport cloudscraper\nimport random\nimport time\n\n# Step 1: Define a list of proxies (authenticated and non-authenticated)\n# Replace \u003cPROXY_USERNAME\u003e, \u003cPROXY_PASSWORD\u003e, \u003cPROXY_HOST\u003e, and \u003cPROXY_PORT\u003e with actual values\nproxy_list = [\n    {\"http\": \"http://\u003cPROXY_HOST_1\u003e:\u003cPROXY_PORT_1\u003e\", \"https\": \"http://\u003cPROXY_HOST_1\u003e:\u003cPROXY_PORT_1\u003e\"},\n    {\"http\": \"http://\u003cPROXY_USERNAME\u003e:\u003cPROXY_PASSWORD\u003e@\u003cPROXY_HOST_2\u003e:\u003cPROXY_PORT_2\u003e\", \n     \"https\": \"http://\u003cPROXY_USERNAME\u003e:\u003cPROXY_PASSWORD\u003e@\u003cPROXY_HOST_2\u003e:\u003cPROXY_PORT_2\u003e\"},\n    {\"http\": \"http://\u003cPROXY_USERNAME\u003e:\u003cPROXY_PASSWORD\u003e@\u003cPROXY_HOST_3\u003e:\u003cPROXY_PORT_3\u003e\", \n     \"https\": \"http://\u003cPROXY_USERNAME\u003e:\u003cPROXY_PASSWORD\u003e@\u003cPROXY_HOST_3\u003e:\u003cPROXY_PORT_3\u003e\"}\n]\n\n# Step 2: Create a CloudScraper instance\nscraper = cloudscraper.create_scraper()\n\n# Step 3: Define the target URL\ntarget_url = \"https://httpbin.org/ip\"  # This endpoint returns the caller's IP address\n\n# Step 4: Implement proxy rotation and make requests\ndef fetch_with_proxy_rotation(proxy_list, target_url, num_requests=5):\n    \"\"\"\n    Fetch the target URL using proxy rotation.\n    \n    Args:\n        proxy_list (list): A list of proxy configurations.\n        target_url (str): The URL to scrape.\n        num_requests (int): Number of requests to make.\n    \"\"\"\n    for i in range(num_requests):\n        # Randomly select a proxy from the list\n        proxy = random.choice(proxy_list)\n        \n        try:\n            # Make a request using the selected proxy\n            print(f\"Using proxy: {proxy}\")\n            response = scraper.get(target_url, proxies=proxy, timeout=10)\n            \n            # Print the response (IP address of the proxy)\n            print(f\"Response {i + 1}: {response.text}\")\n        \n        except Exception as e:\n            # Handle errors (e.g., connection timeout, proxy failure)\n            print(f\"Error with proxy {proxy}: {e}\")\n        \n        # Wait a bit before the next request to mimic human behavior\n        time.sleep(random.uniform(1, 3))\n\n# Step 5: Run the function\nfetch_with_proxy_rotation(proxy_list, target_url, num_requests=5)\n```\n\n### Output Example\n\n```python\nUsing proxy: {'http': 'http://\u003cPROXY_HOST_1\u003e:\u003cPROXY_PORT_1\u003e', 'https': 'http://\u003cPROXY_HOST_1\u003e:\u003cPROXY_PORT_1\u003e'}\nResponse 1: {\n    \"origin\": \"203.0.113.1\"\n}\nUsing proxy: {'http': 'http://\u003cPROXY_USERNAME\u003e:\u003cPROXY_PASSWORD\u003e@\u003cPROXY_HOST_2\u003e:\u003cPROXY_PORT_2\u003e', 'https': 'http://\u003cPROXY_USERNAME\u003e:\u003cPROXY_PASSWORD\u003e@\u003cPROXY_HOST_2\u003e:\u003cPROXY_PORT_2\u003e'}\nResponse 2: {\n    \"origin\": \"198.51.100.2\"\n}\nUsing proxy: {'http': 'http://\u003cPROXY_USERNAME\u003e:\u003cPROXY_PASSWORD\u003e@\u003cPROXY_HOST_3\u003e:\u003cPROXY_PORT_3\u003e', 'https': 'http://\u003cPROXY_USERNAME\u003e:\u003cPROXY_PASSWORD\u003e@\u003cPROXY_HOST_3\u003e:\u003cPROXY_PORT_3\u003e'}\nResponse 3: {\n    \"origin\": \"192.0.2.3\"\n}\n...\n```\n\n## Conclusion\n\nBright Data controls the best proxy servers in the world, serving Fortune 500 companies and over 20,000 customers. Its worldwide proxy network involves:\n\n*   [Datacenter proxies](https://brightdata.com/proxy-types/datacenter-proxies) – Over 770,000 datacenter IPs.\n*   [Residential proxies](https://brightdata.com/proxy-types/residential-proxies) – Over 72M residential IPs in more than 195 countries.\n*   [ISP proxies](https://brightdata.com/proxy-types/isp-proxies) – Over 700,000 ISP IPs.\n*   [Mobile proxies](https://brightdata.com/proxy-types/mobile-proxies) – Over 7M mobile IPs.\n\n[Create a free Bright Data account](https://brightdata.com) today to try our proxy servers.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluminati-io%2Fcloudscraper-with-proxies","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluminati-io%2Fcloudscraper-with-proxies","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluminati-io%2Fcloudscraper-with-proxies/lists"}