{"id":30770391,"url":"https://github.com/0x7f9/xauto","last_synced_at":"2026-05-20T07:01:29.503Z","repository":{"id":309452705,"uuid":"1026980205","full_name":"0x7f9/xauto","owner":"0x7f9","description":"Selenium WebDriver Infrastructure Built in Python","archived":false,"fork":false,"pushed_at":"2025-08-12T00:33:38.000Z","size":146,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-12T02:16:55.951Z","etag":null,"topics":["browser-automation","javascript-api","scraper-python","selenium-framework","selenium-python","selenium-webdriver","webdriver-manager"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/0x7f9.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-27T03:45:05.000Z","updated_at":"2025-08-12T00:33:41.000Z","dependencies_parsed_at":"2025-08-12T02:17:03.054Z","dependency_job_id":null,"html_url":"https://github.com/0x7f9/xauto","commit_stats":null,"previous_names":["0x7f9/xauto"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/0x7f9/xauto","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0x7f9%2Fxauto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0x7f9%2Fxauto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0x7f9%2Fxauto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0x7f9%2Fxauto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/0x7f9","download_url":"https://codeload.github.com/0x7f9/xauto/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0x7f9%2Fxauto/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273685604,"owners_count":25149722,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-04T02:00:08.968Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["browser-automation","javascript-api","scraper-python","selenium-framework","selenium-python","selenium-webdriver","webdriver-manager"],"created_at":"2025-09-04T23:04:37.825Z","updated_at":"2026-05-20T07:01:29.495Z","avatar_url":"https://github.com/0x7f9.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Selenium WebDriver Infrastructure\n\nThis repository provides a Selenium WebDriver infrastructure in Python. It is designed for web automation tasks including scraping, form filling, and browser automation.\n\n## Core Utilities\n\n### JavaScript API Injection\n\n```python\nfrom xauto.utils.injection import ensure_injected\n# core utilities call ensure_injected internally\n# injects internal JS API into the page\nensure_injected(driver)\n```\n\u003e APIs can be extended to support DOM traversal, element extraction, in page URL parsing, etc.\n\nCurrently, the API provides:\n\n- `waitForReady()` - waits for full document readiness and 70% of `fetch()` requests to complete\n- `closePopups()` - attempts to close all tabs in `window.openedWindows` that are not the original tab\n- `stealthPatches()` - masks common browser automation fingerprints such as navigator, plugins, etc\n- Injection state tracking via `data-injected` attribute\n- Safe reinjection on navigation or dynamic DOM reloads\n- Code is frozen and hidden from enumeration\n\n### Driver Pool Management\n```python\nfrom xauto.internal.geckodriver.driver import get_driver_pool\n# create driver pool\ndriver_pool = get_driver_pool(max_size=10, firefox_options=options)\n\n# will call get_driver_with_injection() internally\nacquire_driver_with_pressure_check(driver_pool, \"Worker-name\")\n\n# helpful methods of driver_pool\ndriver_pool.get_driver(timeout=)  \ndriver_pool.get_driver_with_injection(timeout=)\ndriver_pool.return_driver(driver)          \ndriver_pool.mark_driver_failed(driver)      \ndriver_pool.cleanup_idle_drivers(max_idle_time=)\ndriver_pool.should_close_driver_for_pressure(cooldown_seconds=)\n\n# get pool stats\nstats = driver_pool.get_pool_stats()\nprint(f\"Active drivers: {stats['in_use']}\")\nprint(f\"Total created: {stats['created']}\")\nprint(f\"Errors: {stats['errors']}\")\n```\n\n### Resource Monitoring\n```python\nfrom xauto.internal.memory import get_memory_monitor\nmonitor = get_memory_monitor()\nif monitor.is_under_pressure():\n    # handle high resource usage here\n    print(\"System under pressure\")\n\n# get resource stats\nstats = monitor.get_resource_stats()\nprint(f\"Memory usage: {stats.memory_percent}%\")\nprint(f\"CPU usage: {stats.cpu_percent}%\")\n\nfrom xauto.internal.memory import acquire_driver_with_pressure_check\n# only acquire a driver when system is under CPU/MEM thresholds\n# class Worker uses this check before spawning drivers\ndriver = acquire_driver_with_pressure_check(driver_pool, context=\"unknown\")\ndriver.get()\n\nfrom xauto.internal.memory import wait_high_load\n# blocks the threads runtime preventing work under high load \nforced_unblock = wait_high_load(pool, context=\"url_navigation\", allow_timeout=True)\nif forced_unblock:\n  print(\"Unblocked due to max_wait_time\")\n```\n\n### Browser Validation Checks\n```python\nfrom xauto.utils.validation import is_up\n# quick page probe before trying a driver.get()\nup, status_code = is_up(url, driver)\nif not up:\n    print(f\"Page is not up {status_code}\")\n    # handle error here\n# else continue with driver.get()\n\nfrom xauto.utils.validation import is_browser_error_page\nif is_browser_error_page(driver):\n    print(\"Browser error page detected\")\n    # handle error page here\n\nfrom xauto.utils.validation import is_connection_error\nif is_connection_error(error):\n    print(\"Browser connection error detected\")\n    # handle connection error here\n\nfrom xauto.utils.validation import is_bot_page\nif is_bot_page(driver, url):\n    print(\"Browser bot page detected\")\n    # handle bot page error here\n```\n\n### Browser Handling\n```python\nfrom xauto.utils.browser_utils import close_popups\n# closes all tabs in window.openedWindows that are not the original tab\n# for cleaning up popups or new tabs triggered by window.open during automation\n# close_popups is called internally by all page loading functions\nclose_popups(driver)\n\nfrom xauto.utils.browser_utils import send_key\n# sends keys to a form field with retry logic and optional iframe handling\n# check_url=True waits for a URL change after pressing RETURN\nsend_key(driver, field, \"username\", check_url=False, iframe=iframe_element)\nsend_key(driver, field, \"password1\", check_url=True, iframe=iframe_element)\n```\n\n### Browser Page Loading\n```python\nfrom xauto.utils.page_loading import load_page_with_high_load_check\n# wait_for_page_load and wait_high_load are called inside\nif not load_page_with_high_load_check(driver, url, timeout=):\n    print(\"Page failed to load\")\n    return\n\nfrom xauto.utils.page_loading import wait_for_page_load\n# ensure_body_loaded is called inside\nif not wait_for_page_load(driver, wait_for=):\n    print(\"Page failed to load\")\n    # handle page loading error here\n\nfrom xauto.utils.page_loading import ensure_body_loaded\nif not ensure_body_loaded(driver, wait_for=):\n    print(\"Page body failed to load\")\n    # handle page loading error here\n\nfrom xauto.utils.page_loading import explicit_page_load\nif not explicit_page_load(driver, wait_for=):\n    print(\"Page did not load after a explict wait\")\n    # handle page loading error here\n\nfrom xauto.utils.page_loading import wait_for_url_change\n# wait for a URL change after sending RETURN key\nwait_for_url_change(driver, old_url, wait_for=)\n```\n\n### Thread Safe\n```python\nfrom xauto.internal.thread_safe import ThreadSafeList, ThreadSafeDict, AtomicCounter\nsafe_list = ThreadSafeList()\nsafe_dict = ThreadSafeDict()\ncounter = AtomicCounter()\n\nsafe_list.append(item)\nsafe_dict[key] = value\ncounter.increment()\ncounter.decrement()\ncounter.reset()\ncounter.get()\n\nfrom xauto.internal.thread_safe import SafeThread\nthread = SafeThread(\n    target_fn=dummy_function,\n    name=\"DummyName\"\n)\nthread.start()\n```\n\n### Bootstrap\n```python\nfrom xauto.bootstrap.build import bootstrap\nif not bootstrap():\n    print(\"Bootstrap failed\")\n    sys.exit(1)\n```\n\nThe bootstrap system will:\n1. Check python version is compatible \n2. Download GeckoDriver\n3. Create a virtual environment\n4. Install required dependencies\n\n## Features\n\n- **Thread Safety**: Uses SafeThread wrappers to isolate thread crashes from the main event loop\n- **Resource Monitoring**: Real time memory and CPU pressure monitoring\n- **Auto scaling**: Driver pool scales based on system feedback\n- **Graceful Shutdown**: Cleanup and resource management\n- **Configuration Management**: YAML based configuration with runtime freezing\n- **Proxy Support**: Supports proxy rotation, authentication, and SOCKS5 proxies.\n- **Browser Automation**: Firefox/GeckoDriver with some anti detection features\n- **Task Management**: Thread safe task queuing and worker management\n- **Browser Error Detection**: Built in detection of browser error pages\n- **Generic Infrastructure**: Designed for any web automation task, not just scraping\n- **Logging**: Multiple log levels and output files\n- **JavaScript API Injection**: Custom browser API (_xautoAPI)\n- **Bot Detection**: Able to detect pages with bot challenges for handling or bypass logic\n\n## Configuration\n\nThe application is configured via `settings.yaml`. Key configuration sections:\n\n### System Settings\n```yaml\nsystem:\n  driver_limit: auto  # or specific number\n  headless: false\n```\n\n### Resource Management\n```yaml\nresources:\n  driver_autoscaling:\n    scaling_check_interval: 0.5\n    step_up: 2\n    step_down: 1\n    scale_down_cooldown: 5.0\n  memory_tuning:\n    pressure:\n      mem_threshold: 75.0\n      cpu_threshold: 80.0\n```\n\n### Misc \n```yaml\nmisc:\n  logging:\n    log_timer_interval: 5\n\n  thread_monitoring:\n    worker_monitor_loop_interval: 1\n\n  timeouts:\n    # shutdown timeouts \n    program_shutdown_timeout: 10\n    stop_all_workers_timeout: 10\n    stop_worker_timeout: 5\n\n    # function retries\n    max_send_key_retries: 2\n    max_worker_task_retries: 2\n\n   # loading timeouts\n    max_body_load_wait: 10\n    max_url_load_wait: 5\n\n    # driver timeouts\n    driver_recreate_delay: 5\n    driver_slot_wait_delay: 3\n```\n\n### Proxy Configuration\n```yaml\nproxy:\n  enabled: true\n  credentials:\n    enabled: false\n    username: \"\"\n    password: \"\"\n  list: []\n```\n\n## Dependencies\n\nCore dependencies (see `bootstrap/installs.txt`):\n- `selenium==4.33.0` - WebDriver automation\n- `selenium-wire==5.1.0` - Proxy support\n- `requests` - HTTP client\n- `psutil` - System monitoring\n- `pyyaml` - Configuration parsing\n- `termcolor` - Colored output\n- `blinker` - Event signaling\n\n## Disclaimer\n\nThis infrastructure is designed for legitimate web automation tasks. Repository contains only the core infrastructure. The actual automation logic and credential processing components have been removed.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0x7f9%2Fxauto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F0x7f9%2Fxauto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0x7f9%2Fxauto/lists"}