{"id":20710079,"url":"https://github.com/oxylabs/mechanicalsoup-proxy-integration","last_synced_at":"2025-05-14T21:32:55.644Z","repository":{"id":134336653,"uuid":"414490768","full_name":"oxylabs/mechanicalsoup-proxy-integration","owner":"oxylabs","description":"Python tutorial for integrating Oxylabs' Residential Proxies with MechanicalSoup library","archived":false,"fork":false,"pushed_at":"2025-02-11T12:30:04.000Z","size":42,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-17T06:27:14.689Z","etag":null,"topics":["beautifulsoup","bs4","github-python","mechanicalsoup","proxy-list","proxy-list-github","proxy-rotator","proxy-site","python","requests","rotating-proxy"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-07T06:45:49.000Z","updated_at":"2025-02-11T12:30:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"5ed06d59-c0a8-414f-9f76-d3c601940c28","html_url":"https://github.com/oxylabs/mechanicalsoup-proxy-integration","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fmechanicalsoup-proxy-integration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fmechanicalsoup-proxy-integration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fmechanicalsoup-proxy-integration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fmechanicalsoup-proxy-integration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/mechanicalsoup-proxy-integration/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254230872,"owners_count":22036259,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautifulsoup","bs4","github-python","mechanicalsoup","proxy-list","proxy-list-github","proxy-rotator","proxy-site","python","requests","rotating-proxy"],"created_at":"2024-11-17T02:09:45.029Z","updated_at":"2025-05-14T21:32:55.518Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Oxylabs’ Residential Proxies integration with MechanicalSoup\n\n[\u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=Python\u0026color=brightgreen\" /\u003e](https://github.com/topics/python) [\u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=Mechanical%20Soup\u0026color=orange\" /\u003e](https://github.com/topics/mechanicalsoup) [\u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=Web-Scraping\u0026color=yellow\" /\u003e](https://github.com/topics/web-scraping) [\u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=Rotating%20Proxies\u0026color=blueviolet\" /\u003e](https://github.com/topics/rotating-proxies)\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.go2cloud.org/aff_c?offer_id=7\u0026aff_id=877\u0026url_id=112)\n\n[![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/GbxmdGhZjq)\n\n[Mechanical Soup](https://github.com/MechanicalSoup/MechanicalSoup) is a Python library designed\nfor automating web interactions such as submitting forms, following links and redirects. Since it\nis built on using Python `requests` and `BeautifulSoup` libraries, `MechanicalSoup` \nis often used as a library to perform some web-scraping operations, such as image extraction,\ndue to the powerful integrated functions that comes in with it. In this tutorial, we're going\nto cover how you can integrate [Oxylabs' Residential Proxies](https://oxy.yt/VakT) with \nMechanicalSoup and share a code sample for submitting an HTML form while using proxies.\n\n## Requirements\n\nFor the integration to work, you'll need to install it on your system. \nYou can do it using `pip` command:\n```bash\npip install mechanicalsoup\n```\n\n`Python 3` or higher\n\nResidential Proxies: https://oxy.yt/urSrl\n\n## Proxy Authentication\n\nFor proxies to work, you'll need to specify your Oxylabs Residential Proxy access credentials inside the \n[main.py](https://github.com/oxylabs/mechanicalsoup-proxy-integration/blob/main/main.py) file.\n\n```python\nUSERNAME = \"your_username\"\nPASSWORD = \"your_password\"\nENDPOINT = \"pr.oxylabs.io:7777\"\n```\nAdjust the `your_username` and `your_password` values with the username and password \nof your Oxylabs Residential Proxy access credentials.\n\n## Testing Proxy Connection\n\nTo see if the proxy is working, try visiting [ip.oxylabs.io/location](https://ip.oxylabs.io/location). \u003cbr\u003eIf everything is working correctly, \nit will return an IP address of a proxy that you're using.\n\n## Locating an HTML Form\n\nLocating an HTML form in MechanicalSoup is relatively easy - all you have to do is to select it\nvia CSS selector using a `select_form` method. It returns a `soup` object that can be later \nretrieved using `form` attribute. Here's an example of locating a form and printing its values in\nthe input fields.\n\n```python\nimport mechanicalsoup\n\n# Credentials of Oxylabs' Residential Proxy access.\nUSER = \"your_username\"\nPASSWORD = \"your_password\"\nENDPOINT = \"pr.oxylabs.io:7777\"\n\nproxies = {\n    \"http\": f\"http://{USER}:{PASSWORD}@{ENDPOINT}\",\n    \"https\": f\"http://{USER}:{PASSWORD}@{ENDPOINT}\",\n}\n\ndef get_html_form(proxies):\n    # Initiate a MechanicalSoup object.\n    browser = mechanicalsoup.StatefulBrowser()\n    browser.session.proxies = proxies \n    browser.open(\"https://httpbin.org/forms/post\") \n    \n    # Select a form in HTML using a CSS Selector.\n    form = browser.select_form('form[action=\"/post\"]')\n    # Print the form field data.\n    return form.print_summary()\n\n\nif __name__ == \"__main__\":\n    print(get_html_form(proxies))\n```\n\n## Full Code: Submitting an HTML Form with Proxies\n\n```python\nimport mechanicalsoup\n\n# Credentials for Oxylabs' Residential Proxy access.\nUSER = \"your_username\"\nPASSWORD = \"your_password\"\nENDPOINT = \"pr.oxylabs.io:7777\"\n\nproxies = {\n    \"http\": f\"http://{USER}:{PASSWORD}@{ENDPOINT}\",\n    \"https\": f\"http://{USER}:{PASSWORD}@{ENDPOINT}\",\n}\n\ndef get_html_form(proxies):\n    # Initiate a MechanicalSoup object.\n    browser = mechanicalsoup.StatefulBrowser()\n    browser.session.proxies = proxies \n    browser.open(\"https://httpbin.org/forms/post\") \n\n    # Select a form in HTML using a CSS Selector.\n    form = browser.select_form('form[action=\"/post\"]')\n\n    form_info = {\n        \"custname\": \"John\",\n        \"custtel\": \"123\",\n        \"custemail\": \"info@example.com\",\n        \"size\": \"small\",\n        \"topping\": (\"bacon\", \"cheese\", \"onion\"),\n        \"delivery\": \"18:30\",\n        \"comments\": \"I like pizza\",\n    }\n\n    # Populate the form with values from the `form_info` dict.\n    for key, value in form_info.items():\n        form.set(key, value)\n\n    # Launch a Browser.\n    browser.launch_browser()\n    response = browser.submit_selected()\n    return response.text\n\n\nif __name__ == \"__main__\":\n    print(get_html_form(proxies))\n```\nIf you're having any trouble integrating proxies with MechanicalSoup and this guide didn't help \nyou - feel free to contact Oxylabs customer support at support@oxylabs.io.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fmechanicalsoup-proxy-integration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fmechanicalsoup-proxy-integration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fmechanicalsoup-proxy-integration/lists"}