{"id":13464520,"url":"https://github.com/jmcarp/robobrowser","last_synced_at":"2025-05-14T04:07:29.087Z","repository":{"id":13953426,"uuid":"16653581","full_name":"jmcarp/robobrowser","owner":"jmcarp","description":null,"archived":false,"fork":false,"pushed_at":"2020-09-10T18:41:47.000Z","size":576,"stargazers_count":3712,"open_issues_count":59,"forks_count":338,"subscribers_count":109,"default_branch":"master","last_synced_at":"2025-05-13T07:09:06.725Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jmcarp.png","metadata":{"files":{"readme":"README.rst","changelog":"HISTORY.rst","contributing":"docs/contributing.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-02-08T21:29:49.000Z","updated_at":"2025-05-11T08:11:12.000Z","dependencies_parsed_at":"2022-07-12T15:13:18.741Z","dependency_job_id":null,"html_url":"https://github.com/jmcarp/robobrowser","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmcarp%2Frobobrowser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmcarp%2Frobobrowser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmcarp%2Frobobrowser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmcarp%2Frobobrowser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jmcarp","download_url":"https://codeload.github.com/jmcarp/robobrowser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254069479,"owners_count":22009557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T14:00:45.485Z","updated_at":"2025-05-14T04:07:29.028Z","avatar_url":"https://github.com/jmcarp.png","language":"Python","readme":"RoboBrowser: Your friendly neighborhood web scraper\n===============================================\n\n.. image:: https://badge.fury.io/py/robobrowser.png\n    :target: http://badge.fury.io/py/robobrowser\n\n.. image:: https://travis-ci.org/jmcarp/robobrowser.png?branch=master\n        :target: https://travis-ci.org/jmcarp/robobrowser\n\n.. image:: https://coveralls.io/repos/jmcarp/robobrowser/badge.png?branch=master\n        :target: https://coveralls.io/r/jmcarp/robobrowser\n\nHomepage: `http://robobrowser.readthedocs.org/ \u003chttp://robobrowser.readthedocs.org/\u003e`_\n\nRoboBrowser is a simple, Pythonic library for browsing the web without a standalone web browser. RoboBrowser\ncan fetch a page, click on links and buttons, and fill out and submit forms. If you need to interact with web services\nthat don't have APIs, RoboBrowser can help.\n\n.. code-block:: python\n\n    import re\n    from robobrowser import RoboBrowser\n\n    # Browse to Genius\n    browser = RoboBrowser(history=True)\n    browser.open('http://genius.com/')\n\n    # Search for Porcupine Tree\n    form = browser.get_form(action='/search')\n    form                # \u003cRoboForm q=\u003e\n    form['q'].value = 'porcupine tree'\n    browser.submit_form(form)\n\n    # Look up the first song\n    songs = browser.select('.song_link')\n    browser.follow_link(songs[0])\n    lyrics = browser.select('.lyrics')\n    lyrics[0].text      # \\nHear the sound of music ...\n\n    # Back to results page\n    browser.back()\n\n    # Look up my favorite song\n    song_link = browser.get_link('trains')\n    browser.follow_link(song_link)\n\n    # Can also search HTML using regex patterns\n    lyrics = browser.find(class_=re.compile(r'\\blyrics\\b'))\n    lyrics.text         # \\nTrain set and match spied under the blind...\n\nRoboBrowser combines the best of two excellent Python libraries:\n`Requests \u003chttp://docs.python-requests.org/en/latest/\u003e`_ and\n`BeautifulSoup \u003chttp://www.crummy.com/software/BeautifulSoup/\u003e`_.\nRoboBrowser represents browser sessions using Requests and HTML responses\nusing BeautifulSoup, transparently exposing methods of both libraries:\n\n.. code-block:: python\n\n    import re\n    from robobrowser import RoboBrowser\n\n    browser = RoboBrowser(user_agent='a python robot')\n    browser.open('https://github.com/')\n\n    # Inspect the browser session\n    browser.session.cookies['_gh_sess']         # BAh7Bzo...\n    browser.session.headers['User-Agent']       # a python robot\n\n    # Search the parsed HTML\n    browser.select('div.teaser-icon')       # [\u003cdiv class=\"teaser-icon\"\u003e\n                                            # \u003cspan class=\"mega-octicon octicon-checklist\"\u003e\u003c/span\u003e\n                                            # \u003c/div\u003e,\n                                            # ...\n    browser.find(class_=re.compile(r'column', re.I))    # \u003cdiv class=\"one-third column\"\u003e\n                                                        # \u003cdiv class=\"teaser-icon\"\u003e\n                                                        # \u003cspan class=\"mega-octicon octicon-checklist\"\u003e\u003c/span\u003e\n                                                        # ...\n\nYou can also pass a custom `Session` instance for lower-level configuration:\n\n.. code-block:: python\n\n    from requests import Session\n    from robobrowser import RoboBrowser\n\n    session = Session()\n    session.verify = False  # Skip SSL verification\n    session.proxies = {'http': 'http://custom.proxy.com/'}  # Set default proxies\n    browser = RoboBrowser(session=session)\n\nRoboBrowser also includes tools for working with forms, inspired by\n`WebTest \u003chttps://github.com/Pylons/webtest\u003e`_ and `Mechanize \u003chttp://wwwsearch.sourceforge.net/mechanize/\u003e`_.\n\n.. code-block:: python\n\n    from robobrowser import RoboBrowser\n\n    browser = RoboBrowser()\n    browser.open('http://twitter.com')\n\n    # Get the signup form\n    signup_form = browser.get_form(class_='signup')\n    signup_form         # \u003cRoboForm user[name]=, user[email]=, ...\n\n    # Inspect its values\n    signup_form['authenticity_token'].value     # 6d03597 ...\n\n    # Fill it out\n    signup_form['user[name]'].value = 'python-robot'\n    signup_form['user[user_password]'].value = 'secret'\n\n    # Submit the form\n    browser.submit_form(signup_form)\n\nCheckboxes:\n\n.. code-block:: python\n\n    from robobrowser import RoboBrowser\n\n    # Browse to a page with checkbox inputs\n    browser = RoboBrowser()\n    browser.open('http://www.w3schools.com/html/html_forms.asp')\n\n    # Find the form\n    form = browser.get_forms()[3]\n    form                            # \u003cRoboForm vehicle=[]\u003e\n    form['vehicle']                 # \u003crobobrowser.forms.fields.Checkbox...\u003e\n\n    # Checked values can be get and set like lists\n    form['vehicle'].options         # [u'Bike', u'Car']\n    form['vehicle'].value           # []\n    form['vehicle'].value = ['Bike']\n    form['vehicle'].value = ['Bike', 'Car']\n\n    # Values can also be set using input labels\n    form['vehicle'].labels          # [u'I have a bike', u'I have a car \\r\\n']\n    form['vehicle'].value = ['I have a bike']\n    form['vehicle'].value           # [u'Bike']\n\n    # Only values that correspond to checkbox values or labels can be set;\n    # this will raise a `ValueError`\n    form['vehicle'].value = ['Hot Dogs']\n\nUploading files:\n\n.. code-block:: python\n\n    from robobrowser import RoboBrowser\n\n    # Browse to a page with an upload form\n    browser = RoboBrowser()\n    browser.open('http://cgi-lib.berkeley.edu/ex/fup.html')\n\n    # Find the form\n    upload_form = browser.get_form()\n    upload_form                     # \u003cRoboForm upfile=, note=\u003e\n\n    # Choose a file to upload\n    upload_form['upfile']           # \u003crobobrowser.forms.fields.FileInput...\u003e\n    upload_form['upfile'].value = open('path/to/file.txt', 'r')\n\n    # Submit\n    browser.submit(upload_form)\n\nBy default, creating a browser instantiates a new requests `Session`. \n\nRequirements\n------------\n\n- Python \u003e= 2.6 or \u003e= 3.3\n\nLicense\n-------\n\nMIT licensed. See the bundled `LICENSE \u003chttps://github.com/jmcarp/robobrowser/blob/master/LICENSE\u003e`_ file for more details.\n\n","funding_links":[],"categories":["Web Crawling \u0026 Web Scraping","All","Python","资源列表","Web Crawling","HTML 处理","Computer Networking and Security","Web Crawling [🔝](#readme)"],"sub_categories":["HTML 处理"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmcarp%2Frobobrowser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjmcarp%2Frobobrowser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmcarp%2Frobobrowser/lists"}