{"id":20710171,"url":"https://github.com/oxylabs/curl-with-python","last_synced_at":"2026-03-08T06:32:34.968Z","repository":{"id":234832026,"uuid":"775903202","full_name":"oxylabs/curl-with-python","owner":"oxylabs","description":"Master cURL in Python by using the PycURL library. Learn to send GET and POST requests, custom HTTP headers, and how to fix common problems.","archived":false,"fork":false,"pushed_at":"2025-06-26T08:22:50.000Z","size":31,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-05T08:37:13.938Z","etag":null,"topics":["curl-library","curl-python","pycurl","web-scraping","web-scraping-python"],"latest_commit_sha":null,"homepage":"https://oxylabs.io/blog/curl-with-python","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-03-22T09:23:14.000Z","updated_at":"2025-06-26T08:22:53.000Z","dependencies_parsed_at":"2025-04-14T04:42:58.122Z","dependency_job_id":"9bba8e6f-8f42-4eed-9722-e20ee5f60600","html_url":"https://github.com/oxylabs/curl-with-python","commit_stats":null,"previous_names":["oxylabs/curl-with-python"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oxylabs/curl-with-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fcurl-with-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fcurl-with-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fcurl-with-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fcurl-with-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/curl-with-python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fcurl-with-python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30247361,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-08T05:41:50.788Z","status":"ssl_error","status_checked_at":"2026-03-08T05:41:39.075Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["curl-library","curl-python","pycurl","web-scraping","web-scraping-python"],"created_at":"2024-11-17T02:10:19.147Z","updated_at":"2026-03-08T06:32:34.923Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# How to Use cURL With Python\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.go2cloud.org/aff_c?offer_id=7\u0026aff_id=877\u0026url_id=112)\n\n[![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/eWsVUJrnG5)\n\nFollow this guide to learn how to use cURL in Python to easily send HTTP requests to websites. We'll overview how to send GET and POST requests, custom HTTP headers, handle redirects, and how to fix common PycURL issues.\n\nSee the [full article](https://oxylabs.io/blog/curl-with-python) on our website.\n\n- [Install Python cUrl Library](#install-python-curl-library)\n- [GET Requests with PycURL](#get-requests-with-pycurl)\n- [POST Requests with PycURL](#post-requests-with-pycurl)\n- [Sending Custom HTTP Headers](#sending-custom-http-headers)\n- [Sending JSON Data with PycURL](#sending-json-data-with-pycurl)\n- [Handling Redirects](#handling-redirects)\n  * [Get only HTTP headers](#get-only-http-headers)\n- [PycURL vs. Requests: pros and cons](#pycurl-vs-requests-pros-and-cons)\n- [Web Scraping with PycURL](#web-scraping-with-pycurl)\n- [Common errors and resolutions](#common-errors-and-resolutions)\n  * [1. ImportError for pycurl and openssl](#1-importerror-for-pycurl-and-openssl)\n- [Conclusion](#conclusion)\n\n## Install Python cUrl Library\n\nYou can install PycURL using `pip`, the package installer for Python:\n\n```bash\npip install pycurl\n```\n\n## GET Requests with PycURL\n\nFor this tutorial, we’ll be using this website – [https://httpbin.org](https://httpbin.org), which returns data in JSON with all the headers, data, form, and files found within the request. Moreover, this website accepts only POST request methods, while [https://httpbin.org/get](https://httpbin.org/get) accepts `GET` requests. \n\nExecuting a `GET` request with PycURL is a rather straightforward process. If you use the cURL command and don’t provide the `-X` option, a `GET` request is sent by default.\n\n```bash\n$ curl https://httpbin.org/get\n```\n\nYou can do the same thing using the PycURL library: \n\n```python\nimport pycurl\nfrom io import BytesIO\n\nbuffer = BytesIO()\nc = pycurl.Curl()\nc.setopt(c.URL, 'https://httpbin.org/get')\nc.setopt(c.WRITEDATA, buffer)\nc.perform()\nc.close()\n\nbody = buffer.getvalue()\nprint(body.decode('utf-8'))\n```\n\nIn this example, we’re creating a PycURL object, setting the URL option, and providing a buffer to store the response data. For comparison, see [how GET requests can be sent via cURL](https://oxylabs.io/blog/curl-get-requests) in the terminal.\n\n## POST Requests with PycURL\n\nPOST requests send data to a server, typically to create or update a resource. To send a `POST` request with PycURL, use the following code:\n\n```python\n# imports here\ndata = {\"field1\": \"value1\", \"field2\": \"value2\"}\npost_data = \"\u0026\".join([f\"{k}={v}\" for k, v in data.items()])\nbuffer = BytesIO()\n\nc = pycurl.Curl()\nc.setopt(c.URL, \"https://httpbin.org/post\")\nc.setopt(c.POSTFIELDS, post_data)\nc.setopt(c.WRITEDATA, buffer)\nc.perform()\nc.close()\n\nresponse = buffer.getvalue()\nprint(response.decode(\"utf-8\"))\n```\n\nHere, we’re creating a dictionary with the data we want to send, convert to a query string format, and setting the `POSTFIELDS` option to the prepared data. If you're interested in running cURL via your terminal, see this post on [how to send POST requests with cURL](https://oxylabs.io/blog/curl-post-requests).\n\n## Sending Custom HTTP Headers\n\nHTTP headers are used to provide additional information about a request or a response. Custom headers can also be included in `GET` requests, depending on your requirements.\n\nTo send custom HTTP headers with a PycURL `GET` request, use the following code:\n\n```python\n# imports here\nheaders = [\"User-Agent: Python-PycURL\", \"Accept: application/json\"]\nbuffer = BytesIO()\nc = pycurl.Curl()\nc.setopt(c.URL, \"https://httpbin.org/headers\")\nc.setopt(c.HTTPHEADER, headers)\nc.setopt(c.WRITEDATA, buffer)\nc.perform()\nc.close()\nresponse = buffer.getvalue()\nprint(response.decode(\"utf-8\"))\n```\n\nIn this example, we’re creating a list of custom headers and setting the `HTTP HEADER` option to this list. After executing the request, we close the PycURL object and print the response. The process of [sending HTTP headers with cURL ](https://oxylabs.io/blog/curl-send-headers)via the terminal doesn't differ too much.\n\n## Sending JSON Data with PycURL\n\nJSON is a popular data format for exchanging data between clients and servers. To send data in a `POST` request using PycURL, see the following example:\n\n```python\n# imports here\ndata = {'field1': 'value1', 'field2': 'value2'}\npost_data = json.dumps(data)\nheaders = ['Content-Type: application/json']\nbuffer = BytesIO()\n\nc = pycurl.Curl()\nc.setopt(c.URL, 'https://httpbin.org/post')\nc.setopt(c.POSTFIELDS, post_data)\nc.setopt(c.HTTPHEADER, headers)\nc.setopt(c.WRITEDATA, buffer)\nc.perform()\nc.close()\n\nresponse = buffer.getvalue()\n\nprint(response.decode('utf-8'))\n```\n\nIn this example, we’re converting the data dictionary to a JSON-formatted string and setting the `POSTFIELDS` option to the JSON string. We’re also setting the content-type header with the intention of informing the server that we’re sending JSON data.\n\n## Handling Redirects\n\nPycURL can automatically follow HTTP redirects by setting the `FOLLOWLOCATION` option:\n\n```python\n# imports here\nbuffer = BytesIO()\nc = pycurl.Curl()\nc.setopt(c.URL, \"http://httpbin.org\")\nc.setopt(c.FOLLOWLOCATION, 1)\nc.setopt(c.WRITEDATA, buffer)\nc.perform()\nc.close()\nresponse = buffer.getvalue()\nprint(response.decode(\"utf-8\"))\n```\n\nThis example demonstrates how to follow redirects by setting the `FOLLOWLOCATION` option to 1 (True).\n\n### Get only HTTP headers\n\nTo get only the HTTP headers, you can set the `HEADERFUNCTION` option to a custom function, which will process the received headers:\n\n```python\n# imports here\ndef process_header(header_line):\n    print(header_line.decode('utf-8').strip())\n\nc = pycurl.Curl()\nc.setopt(c.URL, 'https://httpbin.org/headers')\nc.setopt(c.HEADERFUNCTION, process_header)\nc.setopt(c.NOBODY, 1)\nc.perform()\nc.close()\n```\n\n## PycURL vs. Requests: pros and cons\n\nWhen it comes to choosing between PycURL and Requests, each library has its own strengths and weaknesses. Let’s take a closer look at both:\n\n|          | PycURL                                                                   | Requests                                                              |\n|----------|--------------------------------------------------------------------------|-----------------------------------------------------------------------|\n| Pros     | Faster than Requests, powerful, flexible, supports multiple protocols.   | Easier to learn and use, more readable syntax, better suited for simple tasks. |\n| Cons     | Steeper learning curve, more verbose syntax.                             | Slower than PycURL, supports only the HTTP and HTTPS protocols.       |\n\nIf you prioritize performance and flexibility, PycURL might be a better choice. However, if you’re looking for a simpler and more user-friendly library, you should probably go with Requests.\n\n## Web Scraping with PycURL\nWeb scraping is a technique for extracting information from websites by parsing the HTML content. To perform web scraping tasks, you’ll need additional libraries like BeautifulSoup or lxml. Also, PycURL is particularly useful for web scraping tasks that require handling redirects, cookies, or custom headers.\n\nTypically, web scraping begins with a `GET` request for retrieving the HTML content of the target webpage. Here's an example of web scraping with PycURL and BeautifulSoup:\n\n```python\nimport pycurl\nfrom io import BytesIO\nfrom bs4 import BeautifulSoup\n\nbuffer = BytesIO()\nc = pycurl.Curl()\nc.setopt(c.URL, \"https://books.toscrape.com\")\nc.setopt(c.WRITEDATA, buffer)\nc.perform()\nc.close()\nhtml = buffer.getvalue().decode(\"utf-8\")\nsoup = BeautifulSoup(html, \"html.parser\")\n# Extract data from the parsed HTML\ntitle = soup.find(\"title\")\nprint(title.text)\n```\n\nIn this example, we’re using PycURL to fetch the HTML content. Then, we parse it with BeautifulSoup to extract the desired data. \n\n## Common errors and resolutions\n\n### 1. ImportError for pycurl and openssl\n\nIn some cases, you may have an error in running the code with the libcurl library. It would look something like this:\n\n```bash\nmportError: pycurl: libcurl link-time ssl backends (secure-transport, openssl) do not include compile-time ssl backend (none/other)\n```\n\nThis error means that the OpenSSL headers are missing from your system. To fix this, use the following commands depending on your operating system.\n\n**On macOS**, install OpenSSL 1.1 with Homebrew:\n\n```bash\nbrew install openssl@1.1\nexport LDFLAGS=\"-L/usr/local/opt/openssl@1.1/lib\"\nexport CPPFLAGS=\"-I/usr/local/opt/openssl@1.1/include\"\n```\n\nAfterwards, reinstall PycURL: \n\n```bash\npip uninstall pycurl\npip install pycurl --no-cache-dir\n```\n\n**On Windows**, [download](https://slproweb.com/products/Win32OpenSSL.html) and install the OpenSSL 1.1.x binaries. After that, add the following environment variables:\n\n- `PYCURL_SSL_LIBRARY` with the value / `openssl`\n- `LIB` with the value `C:\\OpenSSL-Win64\\lib` (replace `C:\\OpenSSL-Win64` with the actual installation path if different)\n- `INCLUDE` with the value `C:\\OpenSSL-Win64\\include`\n\nReinstall the Python library PycURL, and your code should now work.\n\n2. UnicodeEncodeError when sending non-ASCII data\n\nThis error occurs when you try to send non-ASCII characters in a PycURL request without properly encoding the data. \n\nTo resolve this issue, make sure to encode the data using the appropriate character encoding (usually `'utf-8'`) before sending it with PycURL:\n\n```python\n# imports here\n\ndata = {\"field1\": \"value1\", \"field2\": \"valüe2\"}\npost_data = \"\u0026\".join([f\"{k}={v}\" for k, v in data.items()]).encode('utf-8')\nbuffer = BytesIO()\n\nc = pycurl.Curl()\nc.setopt(c.URL, \"https://httpbin.org/post\")\nc.setopt(c.POSTFIELDS, post_data)\nc.setopt(c.WRITEDATA, buffer)\nc.perform()\nc.close()\n\nresponse = buffer.getvalue()\nprint(response.decode(\"utf-8\"))\n```\n\n## Conclusion\n\nUsing cURL with Python through the PycURL library offers a range of powerful features for interacting with web resources and APIs. Following the examples in this guide, you can perform tasks such as GET and POST requests, handling HTTP requests, headers and form data, and even web scraping.\n\nWe hope that you found this guide helpful. If you have any questions related to the matter, feel free to contact us at support@oxylabs.io, and our professionals will get back to you within a day. If you're curious to learn more about the topic, check out our articles on [How to Use cURL With Proxy?](https://oxylabs.io/blog/curl-with-proxy), [cURL with Python](https://oxylabs.io/blog/curl-with-python), and [cURL with APIs](https://oxylabs.io/blog/curl-with-api).\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fcurl-with-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fcurl-with-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fcurl-with-python/lists"}