Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/oxylabs/selenium-proxy-integration-python
Tutorial for integrating Oxylabs' Residential Proxies with Selenium in Python
https://github.com/oxylabs/selenium-proxy-integration-python
antidetect-browser github-python proxy proxy-list proxy-list-github proxy-rotator proxy-server proxy-site python python-ecommerce python-web-crawler residential-proxy rotating-proxy scraper-python selenium serp-api-python socks5-proxy-list socks5-server webproxy
Last synced: 4 days ago
JSON representation
Tutorial for integrating Oxylabs' Residential Proxies with Selenium in Python
- Host: GitHub
- URL: https://github.com/oxylabs/selenium-proxy-integration-python
- Owner: oxylabs
- Created: 2021-10-01T05:31:03.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-04-19T11:02:51.000Z (7 months ago)
- Last Synced: 2024-04-21T02:04:48.241Z (7 months ago)
- Topics: antidetect-browser, github-python, proxy, proxy-list, proxy-list-github, proxy-rotator, proxy-server, proxy-site, python, python-ecommerce, python-web-crawler, residential-proxy, rotating-proxy, scraper-python, selenium, serp-api-python, socks5-proxy-list, socks5-server, webproxy
- Language: Python
- Homepage:
- Size: 37.1 KB
- Stars: 13
- Watchers: 2
- Forks: 8
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Oxylabs’ Residential Proxies integration with Selenium in Python
[](https://oxylabs.go2cloud.org/aff_c?offer_id=7&aff_id=877&url_id=112)
[](https://discord.gg/GbxmdGhZjq)
[
](https://github.com/topics/python) [
](https://github.com/topics/selenium) [
](https://github.com/topics/web-scraping) [
](https://github.com/topics/rotating-proxies)
## Requirements
For the integration to work, you'll need to install
[Selenium Wire](https://github.com/wkeeling/selenium-wire)
to extend Selenium’s Python bindings as implementing proxies
that require authentication using default Selenium module
complicates the process too much.
You can do it using `pip` command:
```bash
pip install selenium-wire
```
Another recommended package is `webdriver-manager`. It simplifies the management
of binary drivers for different browsers, so you don't need to manually download
a new version of a web driver after each update. Visit the
[official project directory](https://pypi.org/project/webdriver-manager/) on pypi to
find out more information.
You can install the following using `pip` as well:
```bash
pip install webdriver-manager
```
Required version of Python: `Python 3.5` (or higher)
## Proxy Authentication
For proxies to work, you'll need to specify your account credentials inside
the [main.py](https://github.com/oxylabs/selenium-proxy-integration/blob/main/main.py) file.
```python
USERNAME = "your_username"
PASSWORD = "your_password"
ENDPOINT = "pr.oxylabs.io:7777"
```
Adjust the `your_username` and `your_password` value fields with the username and password of
your Oxylabs account.
## Testing Proxy Connection
To see if the proxy is working, try visiting [ip.oxylabs.io/location](https://ip.oxylabs.io/location)
If everything is working correctly, it will return an IP address of a proxy that you're using.
## Full Code
```python
from selenium.webdriver.common.by import By
from seleniumwire import webdriver
# A package to have a chromedriver always up-to-date.
from webdriver_manager.chrome import ChromeDriverManager
USERNAME = "your_username"
PASSWORD = "your_password"
ENDPOINT = "pr.oxylabs.io:7777"
def get_chrome_proxy(user: str, password: str, endpoint: str) -> dict:
wire_options = {
"proxy": {
"http": f"http://{user}:{password}@{endpoint}",
"https": f"http://{user}:{password}@{endpoint}",
}
}
return wire_options
def execute_driver():
options = webdriver.ChromeOptions()
options.headless = True
seleniumwire_options = {
**get_chrome_proxy(USERNAME, PASSWORD, ENDPOINT),
"driver_path": ChromeDriverManager().install(),
}
driver = webdriver.Chrome(
options=options,
seleniumwire_options=seleniumwire_options,
)
try:
driver.get("https://ip.oxylabs.io/location")
return f'\nYour IP is: {driver.find_element(By.CSS_SELECTOR, "pre").text}'
finally:
driver.quit()
if __name__ == "__main__":
print(execute_driver())
```
If you're having any trouble integrating proxies with Selenium and this guide didn't help
you - feel free to contact Oxylabs customer support at [email protected].