{"id":32521597,"url":"https://github.com/lablnet/stepwright","last_synced_at":"2026-01-20T17:03:11.289Z","repository":{"id":320205334,"uuid":"1081212206","full_name":"lablnet/stepwright","owner":"lablnet","description":"A powerful web scraping library built with Playwright that provides a declarative, step-by-step approach to web automation and data extraction.","archived":false,"fork":false,"pushed_at":"2025-10-31T07:09:17.000Z","size":156,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-31T08:37:46.651Z","etag":null,"topics":["declerative","json","mit-license","scraping","structured"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/stepwright/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lablnet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"patreon":"lablnet"}},"created_at":"2025-10-22T13:16:55.000Z","updated_at":"2025-10-31T07:07:42.000Z","dependencies_parsed_at":"2025-10-31T08:25:20.198Z","dependency_job_id":null,"html_url":"https://github.com/lablnet/stepwright","commit_stats":null,"previous_names":["lablnet/stepwright"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/lablnet/stepwright","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lablnet%2Fstepwright","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lablnet%2Fstepwright/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lablnet%2Fstepwright/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lablnet%2Fstepwright/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lablnet","download_url":"https://codeload.github.com/lablnet/stepwright/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lablnet%2Fstepwright/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28607624,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-20T16:10:39.856Z","status":"ssl_error","status_checked_at":"2026-01-20T16:10:39.493Z","response_time":117,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["declerative","json","mit-license","scraping","structured"],"created_at":"2025-10-28T06:41:52.602Z","updated_at":"2026-01-20T17:03:11.282Z","avatar_url":"https://github.com/lablnet.png","language":"Python","readme":"# StepWright\n\nA powerful web scraping library built with Playwright that provides a declarative, step-by-step approach to web automation and data extraction.\n\n## Features\n\n- 🚀 **Declarative Scraping**: Define scraping workflows using Python dictionaries or dataclasses\n- 🔄 **Pagination Support**: Built-in support for next button and scroll-based pagination\n- 📊 **Data Collection**: Extract text, HTML, values, and files from web pages\n- 🔗 **Multi-tab Support**: Handle multiple tabs and complex navigation flows\n- 📄 **PDF Generation**: Save pages as PDFs or trigger print-to-PDF actions\n- 📥 **File Downloads**: Download files with automatic directory creation\n- 🔁 **Looping \u0026 Iteration**: ForEach loops for processing multiple elements\n- 📡 **Streaming Results**: Real-time result processing with callbacks\n- 🎯 **Error Handling**: Graceful error handling with configurable termination\n- 🔧 **Flexible Selectors**: Support for ID, class, tag, and XPath selectors\n- 🔁 **Retry Logic**: Automatic retry on failure with configurable delays\n- 🎛️ **Conditional Execution**: Skip or execute steps based on JavaScript conditions\n- ⏳ **Smart Waiting**: Wait for selectors before actions with configurable timeouts\n- 🔀 **Fallback Selectors**: Multiple selector fallbacks for increased robustness\n- 🖱️ **Enhanced Clicks**: Double-click, right-click, modifier keys, and force clicks\n- ⌨️ **Input Enhancements**: Clear before input, human-like typing delays\n- 🔍 **Data Transformations**: Regex extraction, JavaScript transformations, default values\n- 🌐 **Page Actions**: Reload, get URL/title, meta tags, cookies, localStorage, viewport\n- 🤖 **Human-like Behavior**: Random delays to mimic human interaction\n- ✅ **Element State Checks**: Require visible/enabled before actions\n\n## Installation\n\n```bash\n# Using pip\npip install stepwright\n\n# Using pip with development dependencies\npip install stepwright[dev]\n\n# From source\ngit clone https://github.com/lablnet/stepwright.git\ncd stepwright\npip install -e .\n```\n\n## Quick Start\n\n### Basic Usage\n\n```python\nimport asyncio\nfrom stepwright import run_scraper, TabTemplate, BaseStep\n\nasync def main():\n    templates = [\n        TabTemplate(\n            tab=\"example\",\n            steps=[\n                BaseStep(\n                    id=\"navigate\",\n                    action=\"navigate\",\n                    value=\"https://example.com\"\n                ),\n                BaseStep(\n                    id=\"get_title\",\n                    action=\"data\",\n                    object_type=\"tag\",\n                    object=\"h1\",\n                    key=\"title\",\n                    data_type=\"text\"\n                )\n            ]\n        )\n    ]\n\n    results = await run_scraper(templates)\n    print(results)\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n## API Reference\n\n### Core Functions\n\n#### `run_scraper(templates, options=None)`\n\nMain function to execute scraping templates.\n\n**Parameters:**\n- `templates`: List of `TabTemplate` objects\n- `options`: Optional `RunOptions` object\n\n**Returns:** `List[Dict[str, Any]]`\n\n```python\nresults = await run_scraper(templates, RunOptions(\n    browser={\"headless\": True}\n))\n```\n\n#### `run_scraper_with_callback(templates, on_result, options=None)`\n\nExecute scraping with streaming results via callback.\n\n**Parameters:**\n- `templates`: List of `TabTemplate` objects\n- `on_result`: Callback function for each result (can be sync or async)\n- `options`: Optional `RunOptions` object\n\n```python\nasync def process_result(result, index):\n    print(f\"Result {index}: {result}\")\n\nawait run_scraper_with_callback(templates, process_result)\n```\n\n### Types\n\n#### `TabTemplate`\n\n```python\n@dataclass\nclass TabTemplate:\n    tab: str\n    initSteps: Optional[List[BaseStep]] = None      # Steps executed once before pagination\n    perPageSteps: Optional[List[BaseStep]] = None   # Steps executed for each page\n    steps: Optional[List[BaseStep]] = None          # Single steps array\n    pagination: Optional[PaginationConfig] = None\n```\n\n#### `BaseStep`\n\n```python\n@dataclass\nclass BaseStep:\n    id: str\n    description: Optional[str] = None\n    object_type: Optional[SelectorType] = None  # 'id' | 'class' | 'tag' | 'xpath'\n    object: Optional[str] = None\n    action: Literal[\n        \"navigate\", \"input\", \"click\", \"data\", \"scroll\", \n        \"eventBaseDownload\", \"foreach\", \"open\", \"savePDF\", \n        \"printToPDF\", \"downloadPDF\", \"downloadFile\",\n        \"reload\", \"getUrl\", \"getTitle\", \"getMeta\", \"getCookies\", \n        \"setCookies\", \"getLocalStorage\", \"setLocalStorage\", \n        \"getSessionStorage\", \"setSessionStorage\", \"getViewportSize\", \n        \"setViewportSize\", \"screenshot\", \"waitForSelector\", \"evaluate\"\n    ] = \"navigate\"\n    value: Optional[str] = None\n    key: Optional[str] = None\n    data_type: Optional[DataType] = None        # 'text' | 'html' | 'value' | 'default' | 'attribute'\n    wait: Optional[int] = None\n    terminateonerror: Optional[bool] = None\n    subSteps: Optional[List[\"BaseStep\"]] = None\n    autoScroll: Optional[bool] = None\n    \n    # Retry configuration\n    retry: Optional[int] = None                  # Number of retries on failure (default: 0)\n    retryDelay: Optional[int] = None            # Delay between retries in ms (default: 1000)\n    \n    # Conditional execution\n    skipIf: Optional[str] = None                 # JavaScript expression - skip step if true\n    onlyIf: Optional[str] = None                # JavaScript expression - execute only if true\n    \n    # Element waiting and state\n    waitForSelector: Optional[str] = None        # Wait for selector before action\n    waitForSelectorTimeout: Optional[int] = None # Timeout for waitForSelector in ms (default: 30000)\n    waitForSelectorState: Optional[Literal[\"visible\", \"hidden\", \"attached\", \"detached\"]] = None\n    \n    # Multiple selector fallbacks\n    fallbackSelectors: Optional[List[Dict[str, str]]] = None  # List of {object_type, object}\n    \n    # Click enhancements\n    clickModifiers: Optional[List[ClickModifier]] = None  # ['Control', 'Meta', 'Shift', 'Alt']\n    doubleClick: Optional[bool] = None            # Perform double click\n    forceClick: Optional[bool] = None            # Force click even if not visible/actionable\n    rightClick: Optional[bool] = None            # Perform right click\n    \n    # Input enhancements\n    clearBeforeInput: Optional[bool] = None      # Clear input before typing (default: True)\n    inputDelay: Optional[int] = None           # Delay between keystrokes in ms\n    \n    # Data extraction enhancements\n    required: Optional[bool] = None             # Raise error if extraction returns None/empty\n    defaultValue: Optional[str] = None          # Default value if extraction fails\n    regex: Optional[str] = None                 # Regex pattern to extract from data\n    regexGroup: Optional[int] = None            # Regex group to extract (default: 0)\n    transform: Optional[str] = None             # JavaScript expression to transform data\n    \n    # Timeout configuration\n    timeout: Optional[int] = None                # Step-specific timeout in ms\n    \n    # Navigation enhancements\n    waitUntil: Optional[Literal[\"load\", \"domcontentloaded\", \"networkidle\", \"commit\"]] = None\n    \n    # Human-like behavior\n    randomDelay: Optional[Dict[str, int]] = None # {min: ms, max: ms} for random delay\n    \n    # Element state checks\n    requireVisible: Optional[bool] = None        # Require element visible (default: True for click)\n    requireEnabled: Optional[bool] = None       # Require element enabled\n    \n    # Skip/continue logic\n    skipOnError: Optional[bool] = None          # Skip step if error occurs (default: False)\n    continueOnEmpty: Optional[bool] = None      # Continue if element not found (default: True)\n```\n\n#### `RunOptions`\n\n```python\n@dataclass\nclass RunOptions:\n    browser: Optional[dict] = None  # Playwright launch options\n    onResult: Optional[Callable] = None\n```\n\n## Step Actions\n\n### Navigate\nNavigate to a URL.\n\n```python\nBaseStep(\n    id=\"go_to_page\",\n    action=\"navigate\",\n    value=\"https://example.com\"\n)\n```\n\n### Input\nFill form fields.\n\n```python\nBaseStep(\n    id=\"search\",\n    action=\"input\",\n    object_type=\"id\",\n    object=\"search-box\",\n    value=\"search term\"\n)\n```\n\n### Click\nClick on elements.\n\n```python\nBaseStep(\n    id=\"submit\",\n    action=\"click\",\n    object_type=\"class\",\n    object=\"submit-button\"\n)\n```\n\n### Data Extraction\nExtract data from elements.\n\n```python\nBaseStep(\n    id=\"get_title\",\n    action=\"data\",\n    object_type=\"tag\",\n    object=\"h1\",\n    key=\"title\",\n    data_type=\"text\"\n)\n```\n\n### ForEach Loop\nProcess multiple elements.\n\n```python\nBaseStep(\n    id=\"process_items\",\n    action=\"foreach\",\n    object_type=\"class\",\n    object=\"item\",\n    subSteps=[\n        BaseStep(\n            id=\"get_item_title\",\n            action=\"data\",\n            object_type=\"tag\",\n            object=\"h2\",\n            key=\"title\",\n            data_type=\"text\"\n        )\n    ]\n)\n```\n\n### File Operations\n\n#### Event-Based Download\n```python\nBaseStep(\n    id=\"download_file\",\n    action=\"eventBaseDownload\",\n    object_type=\"class\",\n    object=\"download-link\",\n    value=\"./downloads/file.pdf\",\n    key=\"downloaded_file\"\n)\n```\n\n#### Download PDF/File\n```python\nBaseStep(\n    id=\"download_pdf\",\n    action=\"downloadPDF\",\n    object_type=\"class\",\n    object=\"pdf-link\",\n    value=\"./output/document.pdf\",\n    key=\"pdf_file\"\n)\n```\n\n#### Save PDF\n```python\nBaseStep(\n    id=\"save_pdf\",\n    action=\"savePDF\",\n    value=\"./output/page.pdf\",\n    key=\"pdf_file\"\n)\n```\n\n## Pagination\n\n### Next Button Pagination\n```python\nPaginationConfig(\n    strategy=\"next\",\n    nextButton=NextButtonConfig(\n        object_type=\"class\",\n        object=\"next-page\",\n        wait=2000\n    ),\n    maxPages=10\n)\n```\n\n### Scroll Pagination\n```python\nPaginationConfig(\n    strategy=\"scroll\",\n    scroll=ScrollConfig(\n        offset=800,\n        delay=1500\n    ),\n    maxPages=5\n)\n```\n\n### Pagination Strategies\n\n#### paginationFirst\nPaginate first, then collect data from each page:\n\n```python\nTabTemplate(\n    tab=\"news\",\n    initSteps=[...],\n    perPageSteps=[...],  # Collect data from each page\n    pagination=PaginationConfig(\n        strategy=\"next\",\n        nextButton=NextButtonConfig(...),\n        paginationFirst=True  # Go to next page before collecting\n    )\n)\n```\n\n#### paginateAllFirst\nPaginate through all pages first, then collect all data at once:\n\n```python\nTabTemplate(\n    tab=\"articles\",\n    initSteps=[...],\n    perPageSteps=[...],  # Collect all data after all pagination\n    pagination=PaginationConfig(\n        strategy=\"next\",\n        nextButton=NextButtonConfig(...),\n        paginateAllFirst=True  # Load all pages first\n    )\n)\n```\n\n## Advanced Features\n\n### Proxy Support\n```python\nfrom stepwright import run_scraper, RunOptions\n\nresults = await run_scraper(templates, RunOptions(\n    browser={\n        \"proxy\": {\n            \"server\": \"http://proxy-server:8080\",\n            \"username\": \"user\",\n            \"password\": \"pass\"\n        }\n    }\n))\n```\n\n### Custom Browser Options\n```python\nresults = await run_scraper(templates, RunOptions(\n    browser={\n        \"headless\": False,\n        \"slow_mo\": 1000,\n        \"args\": [\"--no-sandbox\", \"--disable-setuid-sandbox\"]\n    }\n))\n```\n\n### Streaming Results\n```python\nasync def process_result(result, index):\n    print(f\"Result {index}: {result}\")\n    # Process result immediately (e.g., save to database)\n    await save_to_database(result)\n\nawait run_scraper_with_callback(\n    templates, \n    process_result,\n    RunOptions(browser={\"headless\": True})\n)\n```\n\n### Data Placeholders\nUse collected data in subsequent steps:\n\n```python\nBaseStep(\n    id=\"get_title\",\n    action=\"data\",\n    object_type=\"id\",\n    object=\"page-title\",\n    key=\"page_title\",\n    data_type=\"text\"\n),\nBaseStep(\n    id=\"save_with_title\",\n    action=\"savePDF\",\n    value=\"./output/{{page_title}}.pdf\",  # Uses collected page_title\n    key=\"pdf_file\"\n)\n```\n\n### Index Placeholders\nUse loop index in foreach steps:\n\n```python\nBaseStep(\n    id=\"process_items\",\n    action=\"foreach\",\n    object_type=\"class\",\n    object=\"item\",\n    subSteps=[\n        BaseStep(\n            id=\"save_item\",\n            action=\"savePDF\",\n            value=\"./output/item_{{i}}.pdf\",      # i = 0, 1, 2, ...\n            # or\n            value=\"./output/item_{{i_plus1}}.pdf\" # i_plus1 = 1, 2, 3, ...\n        )\n    ]\n)\n```\n\n## Error Handling\n\nSteps can be configured to terminate on error:\n\n```python\nBaseStep(\n    id=\"critical_step\",\n    action=\"click\",\n    object_type=\"id\",\n    object=\"important-button\",\n    terminateonerror=True  # Stop execution if this fails\n)\n```\n\nWithout `terminateonerror=True`, errors are logged but execution continues.\n\n## Advanced Step Options\n\n### Retry Logic\n\nAutomatically retry failed steps with configurable delays:\n\n```python\nBaseStep(\n    id=\"click_button\",\n    action=\"click\",\n    object_type=\"id\",\n    object=\"flaky-button\",\n    retry=3,              # Retry up to 3 times\n    retryDelay=1000        # Wait 1 second between retries\n)\n```\n\n### Conditional Execution\n\nExecute or skip steps based on JavaScript conditions:\n\n```python\n# Skip step if condition is true\nBaseStep(\n    id=\"optional_click\",\n    action=\"click\",\n    object_type=\"id\",\n    object=\"optional-button\",\n    skipIf=\"document.querySelector('.modal').classList.contains('hidden')\"\n)\n\n# Execute only if condition is true\nBaseStep(\n    id=\"conditional_data\",\n    action=\"data\",\n    object_type=\"id\",\n    object=\"dynamic-content\",\n    key=\"content\",\n    onlyIf=\"document.querySelector('#dynamic-content') !== null\"\n)\n```\n\n### Wait for Selector\n\nWait for elements to appear before performing actions:\n\n```python\nBaseStep(\n    id=\"click_after_load\",\n    action=\"click\",\n    object_type=\"id\",\n    object=\"target-button\",\n    waitForSelector=\"#loading-indicator\",      # Wait for this selector\n    waitForSelectorTimeout=5000,               # Timeout: 5 seconds\n    waitForSelectorState=\"hidden\"              # Wait until hidden\n)\n```\n\n### Fallback Selectors\n\nProvide multiple selector options for increased robustness:\n\n```python\nBaseStep(\n    id=\"click_with_fallback\",\n    action=\"click\",\n    object_type=\"id\",\n    object=\"primary-button\",                   # Try this first\n    fallbackSelectors=[\n        {\"object_type\": \"class\", \"object\": \"btn-primary\"},\n        {\"object_type\": \"class\", \"object\": \"submit-btn\"},\n        {\"object_type\": \"xpath\", \"object\": \"//button[contains(text(), 'Submit')]\"}\n    ]\n)\n```\n\n### Click Enhancements\n\nAdvanced click options for different interaction types:\n\n```python\n# Double click\nBaseStep(\n    id=\"double_click\",\n    action=\"click\",\n    object_type=\"id\",\n    object=\"item\",\n    doubleClick=True\n)\n\n# Right click (context menu)\nBaseStep(\n    id=\"right_click\",\n    action=\"click\",\n    object_type=\"id\",\n    object=\"context-menu-trigger\",\n    rightClick=True\n)\n\n# Click with modifier keys (Ctrl/Cmd+Click)\nBaseStep(\n    id=\"multi_select\",\n    action=\"click\",\n    object_type=\"class\",\n    object=\"item\",\n    clickModifiers=[\"Control\"]  # or [\"Meta\"] for Mac\n)\n\n# Force click (click hidden elements)\nBaseStep(\n    id=\"force_click\",\n    action=\"click\",\n    object_type=\"id\",\n    object=\"hidden-button\",\n    forceClick=True\n)\n```\n\n### Input Enhancements\n\nMore control over input behavior:\n\n```python\n# Clear input before typing (default: True)\nBaseStep(\n    id=\"clear_and_input\",\n    action=\"input\",\n    object_type=\"id\",\n    object=\"search-box\",\n    value=\"new search term\",\n    clearBeforeInput=True  # Clear existing value first\n)\n\n# Human-like typing with delays\nBaseStep(\n    id=\"human_like_input\",\n    action=\"input\",\n    object_type=\"id\",\n    object=\"form-field\",\n    value=\"slowly typed text\",\n    inputDelay=100  # 100ms delay between each character\n)\n```\n\n### Data Extraction Enhancements\n\nAdvanced data extraction and transformation options:\n\n```python\n# Extract with regex\nBaseStep(\n    id=\"extract_price\",\n    action=\"data\",\n    object_type=\"id\",\n    object=\"price\",\n    key=\"price\",\n    regex=r\"\\$(\\d+\\.\\d+)\",        # Extract dollar amount\n    regexGroup=1                   # Get first capture group\n)\n\n# Transform extracted data with JavaScript\nBaseStep(\n    id=\"transform_data\",\n    action=\"data\",\n    object_type=\"id\",\n    object=\"raw-data\",\n    key=\"processed\",\n    transform=\"value.toUpperCase().trim()\"  # JavaScript transformation\n)\n\n# Required field with default value\nBaseStep(\n    id=\"get_required_data\",\n    action=\"data\",\n    object_type=\"id\",\n    object=\"important-field\",\n    key=\"important\",\n    required=True,                 # Raise error if not found\n    defaultValue=\"N/A\"            # Use if extraction fails\n)\n\n# Continue even if element not found\nBaseStep(\n    id=\"optional_data\",\n    action=\"data\",\n    object_type=\"id\",\n    object=\"optional-content\",\n    key=\"optional\",\n    continueOnEmpty=True           # Don't raise error if not found\n)\n```\n\n### Element State Checks\n\nValidate element state before actions:\n\n```python\nBaseStep(\n    id=\"click_visible\",\n    action=\"click\",\n    object_type=\"id\",\n    object=\"button\",\n    requireVisible=True,           # Ensure element is visible\n    requireEnabled=True            # Ensure element is enabled\n)\n```\n\n### Random Delays\n\nAdd human-like random delays to actions:\n\n```python\nBaseStep(\n    id=\"human_like_action\",\n    action=\"click\",\n    object_type=\"id\",\n    object=\"button\",\n    randomDelay={\"min\": 500, \"max\": 2000}  # Random delay between 500-2000ms\n)\n```\n\n### Skip on Error\n\nSkip steps that fail instead of stopping execution:\n\n```python\nBaseStep(\n    id=\"optional_step\",\n    action=\"click\",\n    object_type=\"id\",\n    object=\"optional-button\",\n    skipOnError=True  # Continue even if this step fails\n)\n```\n\n## Page Actions\n\n### Reload Page\n\nReload the current page with optional wait conditions:\n\n```python\nBaseStep(\n    id=\"reload\",\n    action=\"reload\",\n    waitUntil=\"networkidle\"  # Wait for network to be idle\n)\n```\n\n### Get Current URL\n\n```python\nBaseStep(\n    id=\"get_url\",\n    action=\"getUrl\",\n    key=\"current_url\"  # Store in collector\n)\n```\n\n### Get Page Title\n\n```python\nBaseStep(\n    id=\"get_title\",\n    action=\"getTitle\",\n    key=\"page_title\"\n)\n```\n\n### Get Meta Tags\n\n```python\n# Get specific meta tag\nBaseStep(\n    id=\"get_description\",\n    action=\"getMeta\",\n    object=\"description\",  # Meta name or property\n    key=\"meta_description\"\n)\n\n# Get all meta tags\nBaseStep(\n    id=\"get_all_meta\",\n    action=\"getMeta\",\n    key=\"all_meta_tags\"  # Returns dictionary of all meta tags\n)\n```\n\n### Cookies Management\n\n```python\n# Get all cookies\nBaseStep(\n    id=\"get_cookies\",\n    action=\"getCookies\",\n    key=\"cookies\"\n)\n\n# Get specific cookie\nBaseStep(\n    id=\"get_session_cookie\",\n    action=\"getCookies\",\n    object=\"session_id\",\n    key=\"session\"\n)\n\n# Set cookie\nBaseStep(\n    id=\"set_cookie\",\n    action=\"setCookies\",\n    object=\"preference\",\n    value=\"dark_mode\"\n)\n```\n\n### LocalStorage \u0026 SessionStorage\n\n```python\n# Get localStorage value\nBaseStep(\n    id=\"get_storage\",\n    action=\"getLocalStorage\",\n    object=\"user_preference\",\n    key=\"preference\"\n)\n\n# Set localStorage value\nBaseStep(\n    id=\"set_storage\",\n    action=\"setLocalStorage\",\n    object=\"theme\",\n    value=\"dark\"\n)\n\n# Get all localStorage items\nBaseStep(\n    id=\"get_all_storage\",\n    action=\"getLocalStorage\",\n    key=\"all_storage\"\n)\n\n# SessionStorage (same pattern)\nBaseStep(\n    id=\"get_session\",\n    action=\"getSessionStorage\",\n    object=\"temp_data\",\n    key=\"data\"\n)\n```\n\n### Viewport Operations\n\n```python\n# Get viewport size\nBaseStep(\n    id=\"get_viewport\",\n    action=\"getViewportSize\",\n    key=\"viewport\"\n)\n\n# Set viewport size\nBaseStep(\n    id=\"set_viewport\",\n    action=\"setViewportSize\",\n    value=\"1920x1080\"  # or \"1920,1080\" or \"1920 1080\"\n)\n```\n\n### Screenshot\n\n```python\n# Full page screenshot\nBaseStep(\n    id=\"screenshot\",\n    action=\"screenshot\",\n    value=\"./screenshots/page.png\",\n    data_type=\"full\"  # Full page, omit for viewport only\n)\n\n# Element screenshot\nBaseStep(\n    id=\"element_screenshot\",\n    action=\"screenshot\",\n    object_type=\"id\",\n    object=\"content-area\",\n    value=\"./screenshots/element.png\",\n    key=\"screenshot_path\"\n)\n```\n\n### Wait for Selector\n\nExplicit wait for element state:\n\n```python\nBaseStep(\n    id=\"wait_for_element\",\n    action=\"waitForSelector\",\n    object_type=\"id\",\n    object=\"dynamic-content\",\n    value=\"visible\",      # visible, hidden, attached, detached\n    wait=5000,            # Timeout in ms\n    key=\"wait_result\"     # Stores True/False\n)\n```\n\n### Evaluate JavaScript\n\nExecute custom JavaScript:\n\n```python\nBaseStep(\n    id=\"custom_js\",\n    action=\"evaluate\",\n    value=\"() =\u003e document.querySelector('.counter').textContent\",\n    key=\"counter_value\"\n)\n```\n\n## Complete Example\n\n```python\nimport asyncio\nfrom pathlib import Path\nfrom stepwright import (\n    run_scraper,\n    TabTemplate,\n    BaseStep,\n    PaginationConfig,\n    NextButtonConfig,\n    RunOptions\n)\n\nasync def main():\n    templates = [\n        TabTemplate(\n            tab=\"news_scraper\",\n            initSteps=[\n                BaseStep(\n                    id=\"navigate\",\n                    action=\"navigate\",\n                    value=\"https://news-site.com\"\n                ),\n                BaseStep(\n                    id=\"search\",\n                    action=\"input\",\n                    object_type=\"id\",\n                    object=\"search-box\",\n                    value=\"technology\"\n                )\n            ],\n            perPageSteps=[\n                BaseStep(\n                    id=\"collect_articles\",\n                    action=\"foreach\",\n                    object_type=\"class\",\n                    object=\"article\",\n                    subSteps=[\n                        BaseStep(\n                            id=\"get_title\",\n                            action=\"data\",\n                            object_type=\"tag\",\n                            object=\"h2\",\n                            key=\"title\",\n                            data_type=\"text\"\n                        ),\n                        BaseStep(\n                            id=\"get_content\",\n                            action=\"data\",\n                            object_type=\"tag\",\n                            object=\"p\",\n                            key=\"content\",\n                            data_type=\"text\"\n                        ),\n                        BaseStep(\n                            id=\"get_link\",\n                            action=\"data\",\n                            object_type=\"tag\",\n                            object=\"a\",\n                            key=\"link\",\n                            data_type=\"value\"\n                        )\n                    ]\n                )\n            ],\n            pagination=PaginationConfig(\n                strategy=\"next\",\n                nextButton=NextButtonConfig(\n                    object_type=\"id\",\n                    object=\"next-page\",\n                    wait=2000\n                ),\n                maxPages=5\n            )\n        )\n    ]\n\n    # Run scraper\n    results = await run_scraper(templates, RunOptions(\n        browser={\"headless\": True}\n    ))\n\n    # Process results\n    for i, article in enumerate(results):\n        print(f\"\\nArticle {i + 1}:\")\n        print(f\"Title: {article.get('title')}\")\n        print(f\"Content: {article.get('content')[:100]}...\")\n        print(f\"Link: {article.get('link')}\")\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n## Development\n\n### Setup\n\n```bash\n# Clone repository\ngit clone https://github.com/lablnet/stepwright.git\ncd stepwright\n\n# Create virtual environment\npython -m venv venv\nsource venv/bin/activate  # On Windows: venv\\Scripts\\activate\n\n# Install in development mode\npip install -e \".[dev]\"\n\n# Install Playwright browsers\nplaywright install chromium\n```\n\n### Running Tests\n\n```bash\n# Run all tests\npytest\n\n# Run with verbose output\npytest -v\n\n# Run specific test file\npytest tests/test_scraper.py\n\n# Run specific test class\npytest tests/test_scraper.py::TestGetBrowser\n\n# Run specific test\npytest tests/test_scraper.py::TestGetBrowser::test_create_browser_instance\n\n# Run with coverage\npytest --cov=src --cov-report=html\n\n# Run integration tests only\npytest tests/test_integration.py\n```\n\n### Project Structure\n\n```\nstepwright/\n├── src/\n│   ├── __init__.py\n│   ├── step_types.py      # Type definitions and dataclasses\n│   ├── helpers.py         # Utility functions\n│   ├── executor.py        # Core step execution logic\n│   ├── parser.py          # Public API (run_scraper)\n│   ├── scraper.py         # Low-level browser automation\n│   ├── handlers/          # Action-specific handlers\n│   │   ├── __init__.py\n│   │   ├── data_handlers.py      # Data extraction handlers\n│   │   ├── file_handlers.py      # File download/PDF handlers\n│   │   ├── loop_handlers.py      # Foreach/open handlers\n│   │   └── page_actions.py       # Page-related actions (reload, getUrl, etc.)\n│   └── scraper_parser.py  # Backward compatibility\n├── tests/\n│   ├── __init__.py\n│   ├── conftest.py        # Pytest configuration\n│   ├── test_page.html     # Test HTML page\n│   ├── test_page_enhanced.html  # Enhanced test page for new features\n│   ├── test_scraper.py    # Core scraper tests\n│   ├── test_parser.py     # Parser function tests\n│   ├── test_new_features.py  # Tests for new features\n│   └── test_integration.py # Integration tests\n├── pyproject.toml         # Package configuration\n├── setup.py               # Setup script\n├── pytest.ini             # Pytest configuration\n├── README.md              # This file\n└── README_TESTS.md        # Detailed test documentation\n```\n\n### Code Quality\n\n```bash\n# Format code with black\nblack src/ tests/\n\n# Lint with flake8\nflake8 src/ tests/\n\n# Type checking with mypy\nmypy src/\n```\n\n## Module Organization\n\nThe codebase follows separation of concerns:\n\n- **step_types.py**: All type definitions (BaseStep, TabTemplate, etc.)\n- **helpers.py**: Utility functions (placeholder replacement, locator creation, condition evaluation)\n- **executor.py**: Core execution logic (execute steps, handle pagination, retry logic)\n- **parser.py**: Public API (run_scraper, run_scraper_with_callback)\n- **scraper.py**: Low-level Playwright wrapper (navigate, click, get_data)\n- **handlers/**: Action-specific handlers organized by functionality\n  - **data_handlers.py**: Data extraction logic with transformations\n  - **file_handlers.py**: File download and PDF operations\n  - **loop_handlers.py**: Foreach loops and new tab/window handling\n  - **page_actions.py**: Page-related actions (reload, getUrl, cookies, storage, etc.)\n- **scraper_parser.py**: Backward compatibility wrapper\n\nYou can import from the main module or specific submodules:\n\n```python\n# From main module (recommended)\nfrom stepwright import run_scraper, TabTemplate, BaseStep\n\n# From specific modules\nfrom stepwright.step_types import TabTemplate, BaseStep\nfrom stepwright.parser import run_scraper\nfrom stepwright.helpers import replace_data_placeholders\n```\n\n## Testing\n\nSee [README_TESTS.md](README_TESTS.md) for detailed testing documentation.\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Make your changes\n4. Add tests for new functionality\n5. Ensure all tests pass (`pytest`)\n6. Commit your changes (`git commit -m 'Add amazing feature'`)\n7. Push to the branch (`git push origin feature/amazing-feature`)\n8. Open a Pull Request\n\n## License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n## Support\n\n- 🐛 Issues: [GitHub Issues](https://github.com/lablnet/stepwright/issues)\n- 📖 Documentation: [README.md](README.md) and [README_TESTS.md](README_TESTS.md)\n- 💬 Discussions: [GitHub Discussions](https://github.com/lablnet/stepwright/discussions)\n\n## Acknowledgments\n\n- Built with [Playwright](https://playwright.dev/)\n- Inspired by declarative web scraping patterns\n- Original TypeScript version: [framework-Island/stepwright](https://github.com/framework-Island/stepwright)\n\n## Author\n\nMuhammad Umer Farooq ([@lablnet](https://github.com/lablnet))\n\n","funding_links":["https://patreon.com/lablnet"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flablnet%2Fstepwright","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flablnet%2Fstepwright","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flablnet%2Fstepwright/lists"}