{"id":24527590,"url":"https://github.com/offlinehacker/spynner","last_synced_at":"2025-03-15T16:33:21.316Z","repository":{"id":141497773,"uuid":"2266607","full_name":"offlinehacker/spynner","owner":"offlinehacker","description":"Statefull programmatic web browser module for python.","archived":false,"fork":false,"pushed_at":"2011-08-25T08:04:07.000Z","size":372,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-15T04:46:22.779Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/offlinehacker.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGES.txt","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-08-25T07:59:07.000Z","updated_at":"2014-11-27T00:07:05.000Z","dependencies_parsed_at":"2023-03-13T09:15:36.412Z","dependency_job_id":null,"html_url":"https://github.com/offlinehacker/spynner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/offlinehacker%2Fspynner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/offlinehacker%2Fspynner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/offlinehacker%2Fspynner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/offlinehacker%2Fspynner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/offlinehacker","download_url":"https://codeload.github.com/offlinehacker/spynner/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243760345,"owners_count":20343626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-22T06:19:38.320Z","updated_at":"2025-03-15T16:33:21.296Z","avatar_url":"https://github.com/offlinehacker.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Intro\n=====================\n\n.. contents::\n\nSpynner is a stateful programmatic web browser module for Python. It is based upon `PyQT \u003chttp://www.qtsoftware.com/\u003e`_ and `WebKit \u003chttp://webkit.org/\u003e`_, so it supports Javascript, AJAX, and every other technology that !WebKit is able to handle (Flash, SVG, ...). Spynner takes advantage of `JQuery \u003chttp://jquery.com\u003e`_. a powerful Javascript library that makes the interaction with pages and event simulation really easy.\n\nUsing Spynner you would able to simulate a web browser with no GUI (though a browsing window can be opened for debugging purposes), so it may be used to implement crawlers or acceptance testing tools.\n\nCredits\n========\nCompanies\n---------\n|makinacom|_\n\n  * `Planet Makina Corpus \u003chttp://www.makina-corpus.org\u003e`_\n  * `Contact us \u003cmailto:python@makina-corpus.org\u003e`_\n\n.. |makinacom| image:: http://depot.makina-corpus.org/public/logo.gif\n.. _makinacom:  http://www.makina-corpus.com\n\nAuthors\n------------\n\n- Mathieu Le Marec - Pasquet \u003ckiorky@cryptelium.net\u003e\n- Arnau Sanchez \u003ctokland@gmail.com\u003e\n\nContributors\n-----------------\n\nDependencies\n===================\n\n  * `Python \u003e=26 \u003chttp://www.python.org\u003e`_\n  * `PyQt \u003e 443 \u003chttp://www.riverbankcomputing.co.uk/software/pyqt/download\u003e`_\n  * Libxml2 / Libxslt libraries and includes files for lxml\n\nFeedback\n==============\nOpen an `Issue \u003chttps://github.com/kiorky/spynner/issues\u003e`_ to report a bug or request a new feature. Other comments and suggestions can be directly emailed to the authors_.\n\nInstall\n============\n* Throught regular easy_install / buildout::\n\n    easy_install spynner\n\n* The bleeding edge version is hosted on github::\n\n    git clone http://spynner.googlecode.com/svn/trunk/ spynner\n    cd spynner\n    python setup.py install\n\nAPI\n=====\nhttp://tokland.freehostia.com/googlecode/spynner/api/\n\nYou can generate the API locally (will create docs/api directory)::\n\n    python setup.py gen_doc\n\nUsage\n=========\nA basic example::\n\n    import spynner\n    browser = spynner.Browser()\n    browser.load(\"http://www.wordreference.com\")\n    browser.runjs(\"console.log('I can run Javascript')\")\n    browser.runjs(\"console.log('I can run jQuery: ' + jQuery('a:first').attr('href'))\")\n    browser.select(\"#esen\")\n    browser.wk_fill(\"input[name=enit]\", \"hola\")\n    browser.click(\"input[name=b]\")\n    browser.wait_page_load()\n    print browser.url, browser.html\n    browser.close()\n\nSometimes you'll want to see what is going on::\n\n    browser = spynner.Browser()\n    browser.debug_level = spynner.DEBUG\n    browser.create_webview()\n    browser.show()\n\nSee more examples in the repository: https://github.com/kiorky/spynner/tree/master/examples\n\nInteract with the controls\n============================\n- See the implementation docstrings or examples !\n- You have three levels of control:\n\n  - webkit methods which are recommended to us (wk_fill_*, wk_click_*) which are jquery based\n  - classical methods (fill, click_*) which are jquery based\n  - low level using QT raw events which are not that well  working ATM.\n    At least, you can move the mouse\n\nRunning Javascript\n====================\nSpynner uses jQuery to make Javascript interface easier.\nBy default, two modules are injected to every loaded page:\n\n  * `JQuery core \u003chttp://docs.jquery.com/Downloading_jQuery\u003e`_ Amongst other things, it adds the powerful `JQuery selectors \u003chttp://docs.jquery.com/Selectors\u003e`_, which are used internally by some Spynner methods.\n    Of course you can also use jQuery when you inject your own code into a page.\n\n  * `Simulate \u003chttp://code.google.com/p/jqueryjs/source/browse/trunk/plugins/simulate\u003e`_ jQuery plugin: Makes it possible to simulate mouse and keyboard events (for now spynner uses it only in the _click_ action). Look up the library code to see which kind of events you can fire.\n\nNote that you must use __jQuery(...)_ instead of _jQuery(...)_  or the common shortcut _$(...)_.\nThat prevents name clashing with the jQuery library used by the page.\n\nCook your soup: parsing the HTML\n===================================\nYou can parse the HTML of a webpage with your favorite parsing library `BeautifulSoup \u003chttp://www.crummy.com/software/BeautifulSoup\u003e`_, `lxml \u003chttp://codespeak.net/lxml/\u003e`_ ,..\nSince we are already using Jquery for Javascript, it feels just natural to work with `pyquery \u003chttp://pypi.python.org/pypi/pyquery\u003e`_, its Python counterpart::\n\n    import spynner\n    import pyquery\n    browser = spynner.Browser()\n    ...\n    d = pyquery.Pyquery(browser.html)\n    d.make_links_absolute(browser.get_url())\n    href = d(\"#somelink\").attr(\"href\")\n    browser.download(href, open(\"/path/outputfile\", \"w\"))\n\nRunning Spynner without X11\n====================================\n- Spynner needs a X11 server to run. If you are running it in a server without X11 you must install the virtual `Xvfb server \u003chttp://en.wikipedia.org/wiki/Xvfb\u003e`_.\n  Debian users can use the small wrapper (xvfb-run). If you are not using Debian, you can download it here:\n  http://www.mail-archive.com/debian-x@lists.debian.org/msg69632/x-run ::\n\n    xvfb-run python myscript_using_spynner.py\n\n- You can also use tightvnc.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fofflinehacker%2Fspynner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fofflinehacker%2Fspynner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fofflinehacker%2Fspynner/lists"}