{"id":26575452,"url":"https://github.com/autoscrape-labs/pydoll","last_synced_at":"2026-04-11T02:17:04.787Z","repository":{"id":259740864,"uuid":"879320294","full_name":"autoscrape-labs/pydoll","owner":"autoscrape-labs","description":"Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions. ","archived":false,"fork":false,"pushed_at":"2025-05-01T02:14:47.000Z","size":1677,"stargazers_count":3474,"open_issues_count":12,"forks_count":197,"subscribers_count":32,"default_branch":"main","last_synced_at":"2025-05-01T03:25:03.898Z","etag":null,"topics":["anti-detection","asynchronous","bot-detection","browser-automation","bypasscaptcha","captcha-breaking","cdp","chromium","playwright","puppeteer","python","recaptcha-v3","selenium","selenium-python","turnstile-bypass","webdriver","webscraping"],"latest_commit_sha":null,"homepage":"https://autoscrape-labs.github.io/pydoll/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/autoscrape-labs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["thalissonvs"]}},"created_at":"2024-10-27T15:46:43.000Z","updated_at":"2025-05-01T03:07:29.000Z","dependencies_parsed_at":"2024-12-09T02:18:33.759Z","dependency_job_id":"5f466997-76c1-48b9-9e43-3269c28f7646","html_url":"https://github.com/autoscrape-labs/pydoll","commit_stats":null,"previous_names":["thalissonvs/pydoll","autoscrape-labs/pydoll"],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autoscrape-labs%2Fpydoll","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autoscrape-labs%2Fpydoll/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autoscrape-labs%2Fpydoll/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autoscrape-labs%2Fpydoll/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/autoscrape-labs","download_url":"https://codeload.github.com/autoscrape-labs/pydoll/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254070112,"owners_count":22009559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anti-detection","asynchronous","bot-detection","browser-automation","bypasscaptcha","captcha-breaking","cdp","chromium","playwright","puppeteer","python","recaptcha-v3","selenium","selenium-python","turnstile-bypass","webdriver","webscraping"],"created_at":"2025-03-23T02:01:51.839Z","updated_at":"2026-04-11T02:17:04.772Z","avatar_url":"https://github.com/autoscrape-labs.png","language":"Python","funding_links":["https://github.com/sponsors/thalissonvs"],"categories":["Python","前端开发框架及项目"],"sub_categories":["前端项目_其他"],"readme":"\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/user-attachments/assets/2c380638-b04a-4b04-b1c8-2958e4237a94\" alt=\"Pydoll Logo\" /\u003e \u003cbr\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003eAsync-native, fully typed, built for evasion and performance.\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/autoscrape-labs/pydoll/stargazers\"\u003e\u003cimg src=\"https://img.shields.io/github/stars/autoscrape-labs/pydoll?style=social\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://codecov.io/gh/autoscrape-labs/pydoll\" \u003e\n        \u003cimg src=\"https://codecov.io/gh/autoscrape-labs/pydoll/graph/badge.svg?token=40I938OGM9\"/\u003e\n    \u003c/a\u003e\n    \u003cimg src=\"https://github.com/autoscrape-labs/pydoll/actions/workflows/tests.yml/badge.svg\" alt=\"Tests\"\u003e\n    \u003cimg src=\"https://github.com/autoscrape-labs/pydoll/actions/workflows/ruff-ci.yml/badge.svg\" alt=\"Ruff CI\"\u003e\n    \u003cimg src=\"https://github.com/autoscrape-labs/pydoll/actions/workflows/mypy.yml/badge.svg\" alt=\"MyPy CI\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/python-%3E%3D3.10-blue\" alt=\"Python \u003e= 3.10\"\u003e\n    \u003ca href=\"https://deepwiki.com/autoscrape-labs/pydoll\"\u003e\u003cimg src=\"https://deepwiki.com/badge.svg\" alt=\"Ask DeepWiki\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://pydoll.tech/\"\u003eDocumentation\u003c/a\u003e \u0026middot;\n    \u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e \u0026middot;\n    \u003ca href=\"#features\"\u003eFeatures\u003c/a\u003e \u0026middot;\n    \u003ca href=\"#support\"\u003eSupport\u003c/a\u003e\n\u003c/p\u003e\n\nPydoll automates Chromium-based browsers (Chrome, Edge) by connecting directly to the Chrome DevTools Protocol over WebSocket. **No WebDriver binary, no `navigator.webdriver` flag, no compatibility issues.**\n\nIt combines a high-level API for stealthy automation with low-level CDP access for fine-grained control over network, fingerprinting, and browser behavior. And with its new **Pydantic-powered extraction engine**, it maps the DOM directly to structured Python objects, delivering an unmatched Developer Experience (DX).\n\n### Top Sponsors\n\n\u003ca href=\"https://substack.thewebscraping.club/p/pydoll-webdriver-scraping?utm_source=github\u0026utm_medium=repo\u0026utm_campaign=pydoll\"\u003e\n    \u003cimg src=\"public/images/banner-the-webscraping-club.png\" alt=\"The Web Scraping Club\" /\u003e\n\u003c/a\u003e\n\n\u003csub\u003eRead a full review of Pydoll on \u003cb\u003e\u003ca href=\"https://substack.thewebscraping.club/p/pydoll-webdriver-scraping?utm_source=github\u0026utm_medium=repo\u0026utm_campaign=pydoll\"\u003eThe Web Scraping Club\u003c/a\u003e\u003c/b\u003e, the #1 newsletter dedicated to web scraping.\u003c/sub\u003e\n\n### Sponsors\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003ca href=\"https://www.thordata.com/?ls=github\u0026lk=pydoll\"\u003e\u003cimg src=\"public/images/Thordata-logo.png\" height=\"30\" alt=\"Thordata\" /\u003e\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://dashboard.capsolver.com/passport/register?inviteCode=WPhTbOsbXEpc\"\u003e\u003cimg src=\"public/images/capsolver-logo.png\" height=\"40\" alt=\"CapSolver\" /\u003e\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://www.testmuai.com/?utm_medium=sponsor\u0026utm_source=pydoll\"\u003e\u003cimg src=\"public/images/logo-lamda-test.svg\" height=\"30\" width=\"130\" alt=\"LambdaTest\" /\u003e\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003csub\u003e[Learn more about our sponsors](SPONSORS.md) \u0026middot; [Become a sponsor](https://github.com/sponsors/thalissonvs)\u003c/sub\u003e\n\n### Why Pydoll\n\n- **Structured extraction**: Define a [Pydantic](https://docs.pydantic.dev/) model, call `tab.extract()`, get typed and validated data back. No manual element-by-element querying.\n- **Async and typed**: Built on `asyncio` from the ground up, 100% type-checked with `mypy`. Full IDE autocompletion and static error checking.\n- **Stealth built in**: Human-like mouse movement, realistic typing, and granular [browser preference](https://pydoll.tech/docs/features/configuration/browser-preferences/) control for fingerprint management.\n- **Network control**: [Intercept](https://pydoll.tech/docs/features/network/interception/) requests to block ads/trackers, [monitor](https://pydoll.tech/docs/features/network/monitoring/) traffic for API discovery, and make [authenticated HTTP requests](https://pydoll.tech/docs/features/network/http-requests/) that inherit the browser session.\n- **Shadow DOM and iframes**: Full support for [shadow roots](https://pydoll.tech/docs/deep-dive/architecture/shadow-dom/) (including closed) and cross-origin iframes. Discover, query, and interact with elements inside them using the same API.\n\n## Installation\n\n```bash\npip install pydoll-python\n```\n\nNo WebDriver binaries or external dependencies required.\n\n## Getting Started\n\n### 1. Stateful Automation \u0026 Evasion\n\nWhen you need to navigate, bypass challenges, or interact with dynamic UI, Pydoll's imperative API handles it with humanized timing by default.\n\n```python\nimport asyncio\nfrom pydoll.browser import Chrome\nfrom pydoll.constants import Key\n\nasync def google_search(query: str):\n    async with Chrome() as browser:\n        tab = await browser.start()\n        await tab.go_to('https://www.google.com')\n\n        # Find elements and interact with human-like timing\n        search_box = await tab.find(tag_name='textarea', name='q')\n        await search_box.insert_text(query)\n        await tab.keyboard.press(Key.ENTER)\n\n        first_result = await tab.find(\n            tag_name='h3',\n            text='autoscrape-labs/pydoll',\n            timeout=10,\n        )\n        await first_result.click()\n        print(f\"Page loaded: {await tab.title}\")\n\nasyncio.run(google_search('pydoll site:github.com'))\n```\n\n### 2. Structured Data Extraction\n\nOnce you reach the target page, switch to the declarative engine. Define what you want with a model, and Pydoll extracts it — typed, validated, and ready to use.\n\n```python\nfrom pydoll.browser.chromium import Chrome\nfrom pydoll.extractor import ExtractionModel, Field\n\nclass Quote(ExtractionModel):\n    text: str = Field(selector='.text', description='The quote text')\n    author: str = Field(selector='.author', description='Who said it')\n    tags: list[str] = Field(selector='.tag', description='Tags')\n    year: int | None = Field(selector='.year', description='Year', default=None)\n\nasync def extract_quotes():\n    async with Chrome() as browser:\n        tab = await browser.start()\n        await tab.go_to('https://quotes.toscrape.com')\n\n        quotes = await tab.extract_all(Quote, scope='.quote', timeout=5)\n\n        for q in quotes:\n            print(f'{q.author}: {q.text}')  # fully typed, IDE autocomplete works\n            print(q.tags)                    # list[str], not a raw element\n            print(q.model_dump_json())       # pydantic serialization built-in\n\nasyncio.run(extract_quotes())\n```\n\nModels support CSS/XPath auto-detection, HTML attribute targeting, custom transforms, and nested models.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eNested models, transforms, and attribute extraction\u003c/b\u003e\u003c/summary\u003e\n\u003cbr\u003e\n\n```python\nfrom datetime import datetime\nfrom pydoll.extractor import ExtractionModel, Field\n\ndef parse_date(raw: str) -\u003e datetime:\n    return datetime.strptime(raw.strip(), '%B %d, %Y')\n\nclass Author(ExtractionModel):\n    name: str = Field(selector='.author-title')\n    born: datetime = Field(\n        selector='.author-born-date',\n        transform=parse_date,\n    )\n\nclass Article(ExtractionModel):\n    title: str = Field(selector='h1')\n    url: str = Field(selector='.source-link', attribute='href')\n    author: Author = Field(selector='.author-card', description='Nested model')\n\narticle = await tab.extract(Article, timeout=5)\narticle.author.born.year  # int — types are preserved all the way down\n```\n\u003c/details\u003e\n\n## Features\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eHumanized Mouse Movement\u003c/b\u003e\u003c/summary\u003e\n\u003cbr\u003e\n\nMouse operations produce human-like cursor movement by default:\n\n- **Bezier curve paths** with asymmetric control points\n- **Fitts's Law timing**: duration scales with distance\n- **Minimum-jerk velocity**: bell-shaped speed profile\n- **Physiological tremor**: Gaussian noise scaled with velocity\n- **Overshoot correction**: ~70% chance on fast movements, then corrects back\n\n```python\nawait tab.mouse.move(500, 300)\nawait tab.mouse.click(500, 300)\nawait tab.mouse.drag(100, 200, 500, 400)\n\nbutton = await tab.find(id='submit')\nawait button.click()\n\n# Opt out when speed matters\nawait tab.mouse.click(500, 300, humanize=False)\n```\n\n[Mouse Control Docs](https://pydoll.tech/docs/features/automation/mouse-control/)\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eShadow DOM Support\u003c/b\u003e\u003c/summary\u003e\n\u003cbr\u003e\n\nFull Shadow DOM support, including closed shadow roots. Because Pydoll operates at the CDP level (below JavaScript), the `closed` mode restriction doesn't apply.\n\n```python\nshadow = await element.get_shadow_root()\nbutton = await shadow.query('.internal-btn')\nawait button.click()\n\n# Discover all shadow roots on the page\nshadow_roots = await tab.find_shadow_roots()\nfor sr in shadow_roots:\n    checkbox = await sr.query('input[type=\"checkbox\"]', raise_exc=False)\n    if checkbox:\n        await checkbox.click()\n```\n\nHighlights:\n- Closed shadow roots work without workarounds\n- `find_shadow_roots()` discovers every shadow root on the page\n- `timeout` parameter for polling until shadow roots appear\n- `deep=True` traverses cross-origin iframes (OOPIFs)\n- Standard `find()`, `query()`, `click()` API inside shadow roots\n\n[Shadow DOM Docs](https://pydoll.tech/docs/deep-dive/architecture/shadow-dom/)\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eHAR Network Recording\u003c/b\u003e\u003c/summary\u003e\n\u003cbr\u003e\n\nRecord network activity during a browser session and export as HAR 1.2. Replay recorded requests to reproduce exact API sequences.\n\n```python\nfrom pydoll.browser.chromium import Chrome\n\nasync with Chrome() as browser:\n    tab = await browser.start()\n\n    async with tab.request.record() as capture:\n        await tab.go_to('https://example.com')\n\n    capture.save('flow.har')\n    print(f'Captured {len(capture.entries)} requests')\n\n    responses = await tab.request.replay('flow.har')\n```\n\n[HAR Recording Docs](https://pydoll.tech/docs/features/network/network-recording/)\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003ePage Bundles\u003c/b\u003e\u003c/summary\u003e\n\u003cbr\u003e\n\nSave the current page and all its assets (CSS, JS, images, fonts) as a `.zip` bundle for offline viewing. Optionally inline everything into a single HTML file.\n\n```python\nawait tab.save_bundle('page.zip')\nawait tab.save_bundle('page-inline.zip', inline_assets=True)\n```\n\n[Screenshots, PDFs \u0026 Bundles Docs](https://pydoll.tech/docs/features/automation/screenshots-and-pdfs/)\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eHybrid Automation (UI + API)\u003c/b\u003e\u003c/summary\u003e\n\u003cbr\u003e\n\nUse UI automation to pass login flows (CAPTCHAs, JS challenges), then switch to `tab.request` for fast API calls that inherit the full browser session: cookies, headers, and all.\n\n```python\n# Log in via UI\nawait tab.go_to('https://my-site.com/login')\nawait (await tab.find(id='username')).type_text('user')\nawait (await tab.find(id='password')).type_text('pass123')\nawait (await tab.find(id='login-btn')).click()\n\n# Make authenticated API calls using the browser session\nresponse = await tab.request.get('https://my-site.com/api/user/profile')\nuser_data = response.json()\n```\n[Hybrid Automation Docs](https://pydoll.tech/docs/features/network/http-requests/)\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eNetwork Interception and Monitoring\u003c/b\u003e\u003c/summary\u003e\n\u003cbr\u003e\n\nMonitor traffic for API discovery or intercept requests to block ads, trackers, and unnecessary resources.\n\n```python\nimport asyncio\nfrom pydoll.browser.chromium import Chrome\nfrom pydoll.protocol.fetch.events import FetchEvent, RequestPausedEvent\nfrom pydoll.protocol.network.types import ErrorReason\n\nasync def block_images():\n    async with Chrome() as browser:\n        tab = await browser.start()\n\n        async def block_resource(event: RequestPausedEvent):\n            request_id = event['params']['requestId']\n            resource_type = event['params']['resourceType']\n\n            if resource_type in ['Image', 'Stylesheet']:\n                await tab.fail_request(request_id, ErrorReason.BLOCKED_BY_CLIENT)\n            else:\n                await tab.continue_request(request_id)\n\n        await tab.enable_fetch_events()\n        await tab.on(FetchEvent.REQUEST_PAUSED, block_resource)\n\n        await tab.go_to('https://example.com')\n        await asyncio.sleep(3)\n        await tab.disable_fetch_events()\n\nasyncio.run(block_images())\n```\n[Network Monitoring](https://pydoll.tech/docs/features/network/monitoring/) | [Request Interception](https://pydoll.tech/docs/features/network/interception/)\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eBrowser Fingerprint Control\u003c/b\u003e\u003c/summary\u003e\n\u003cbr\u003e\n\nGranular control over [browser preferences](https://pydoll.tech/docs/features/configuration/browser-preferences/): hundreds of internal Chrome settings for building consistent fingerprints.\n\n```python\noptions = ChromiumOptions()\n\noptions.browser_preferences = {\n    'profile': {\n        'default_content_setting_values': {\n            'notifications': 2,\n            'geolocation': 2,\n        },\n        'password_manager_enabled': False\n    },\n    'intl': {\n        'accept_languages': 'en-US,en',\n    },\n    'browser': {\n        'check_default_browser': False,\n    }\n}\n```\n[Browser Preferences Guide](https://pydoll.tech/docs/features/configuration/browser-preferences/)\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eConcurrency, Contexts and Remote Connections\u003c/b\u003e\u003c/summary\u003e\n\u003cbr\u003e\n\nManage [multiple tabs](https://pydoll.tech/docs/features/browser-management/tabs/) and [browser contexts](https://pydoll.tech/docs/features/browser-management/contexts/) (isolated sessions) concurrently. Connect to browsers running in Docker or remote servers.\n\n```python\nasync def scrape_page(url, tab):\n    await tab.go_to(url)\n    return await tab.title\n\nasync def concurrent_scraping():\n    async with Chrome() as browser:\n        tab_google = await browser.start()\n        tab_ddg = await browser.new_tab()\n\n        results = await asyncio.gather(\n            scrape_page('https://google.com/', tab_google),\n            scrape_page('https://duckduckgo.com/', tab_ddg)\n        )\n        print(results)\n```\n[Multi-Tab Management](https://pydoll.tech/docs/features/browser-management/tabs/) | [Remote Connections](https://pydoll.tech/docs/features/advanced/remote-connections/)\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eRetry Decorator\u003c/b\u003e\u003c/summary\u003e\n\u003cbr\u003e\n\nThe `@retry` decorator supports custom recovery logic between attempts (e.g., refreshing the page, rotating proxies) and exponential backoff.\n\n```python\nfrom pydoll.decorators import retry\nfrom pydoll.exceptions import ElementNotFound, NetworkError\n\n@retry(\n    max_retries=3,\n    exceptions=[ElementNotFound, NetworkError],\n    on_retry=my_recovery_function,\n    exponential_backoff=True\n)\nasync def scrape_product(self, url: str):\n    # scraping logic\n    ...\n```\n[Retry Decorator Docs](https://pydoll.tech/docs/features/advanced/decorators/)\n\u003c/details\u003e\n\n---\n\n## Contributing\n\nContributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n## Support\n\nIf you find Pydoll useful, consider [sponsoring the project on GitHub](https://github.com/sponsors/thalissonvs).\n\n## License\n\n[MIT License](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautoscrape-labs%2Fpydoll","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautoscrape-labs%2Fpydoll","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautoscrape-labs%2Fpydoll/lists"}