{"id":19152492,"url":"https://github.com/gesiscss/webbotparser","last_synced_at":"2025-10-07T22:06:42.407Z","repository":{"id":185372740,"uuid":"627806529","full_name":"gesiscss/WebBotParser","owner":"gesiscss","description":"Python package for basic parsing of search engine results scraped with WebBot.","archived":false,"fork":false,"pushed_at":"2023-08-01T16:25:49.000Z","size":14,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-01-03T18:47:11.845Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gesiscss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-14T08:39:11.000Z","updated_at":"2023-08-20T19:36:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"de8a917f-7921-4272-af9c-447468e98a81","html_url":"https://github.com/gesiscss/WebBotParser","commit_stats":null,"previous_names":["gesiscss/webbotparser"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gesiscss%2FWebBotParser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gesiscss%2FWebBotParser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gesiscss%2FWebBotParser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gesiscss%2FWebBotParser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gesiscss","download_url":"https://codeload.github.com/gesiscss/WebBotParser/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240236361,"owners_count":19769580,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T08:18:04.500Z","updated_at":"2025-10-07T22:06:37.369Z","avatar_url":"https://github.com/gesiscss.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WebBotParser\n\nWebBotParser is a Python package that adds basic parsing capabilities for search engine results scraped with our browser extension [WebBot](https://github.com/gesiscss/WebBot).\nWith WebBotParser, you can parse search result pages saved from Google, DuckDuckGo, etc. using WebBot's [download capabilities](https://github.com/gesiscss/WebBot#-saving-search-results), or obtained through a different method for further analysis.\n\nThe following engines and search result types are currently supported out of the box:\n\n|                | Text | News | Images | Videos |\n|----------------|------|------|--------|--------|\n| **Google**     | ✓    | ✓    | ✓      | ✓      |\n| **DuckDuckGo** | ✓    | ✓    | ✓      | ✓      |\n| **Yahoo**      | ✓    | ✓    | ✓\n| **Baidu**      | ✓    | ✓\n\n## Installation\n\nFor basic usage, simply clone this repository, or directly download [webbotparser/webbotparser.py](./webbotparser/webbotparser.py) and add the script to your working directory.\n\n### Install the package with pip\n\nIf you want to use WebBotParser over multiple projects/directories, you can also install it as a Python package. Simply run\n```sh\npip install git+https://github.com/gesiscss/WebBotParser\n```\nThe `webbotparser` package is then available globally in your respective Python installation.\n\n## Usage\n\nFor the search engines and result types supported out of the box, simply run\n```python\nfrom webbotparser import WebBotParser\n```\nand initialize the WebBotParser for the search engine and result type your are investigating, for example\n```python\nparser = WebBotParser(engine = 'DuckDuckGo News')\n```\nThen, you can obtain the search results as a pandas DataFrame and metadata as a Python dictionary with\n```python\nmetadata, results = parser.get_results(file='path/to/the/result_page.html')\n```\nFurthermore, `parser.get_metadata(file)` can be used to only extract the metadata. `parser.get_results_from_dir(dir)` allows to directly extract search results spread over multiple pages, as Google text result are provided for instance.\n\nFor more details and examples also see [WebBot tutorials](https://github.com/gesiscss/WebBot-tutorials).\n\n## Extracting images\n\nWebBot archives images inline in the html file of the search results, i.e., they are neither external files on your drive nor fetched from the original source on viewing the downloaded search results page. This allows us to extract the images directly from the html file for further analysis. The engines and result types supported out of the box with WebBotParser allow for extracting images as well. Simply initialize `WebBotParser` as follows:\n```python\nparser = WebBotParser(engine = 'Google Video', extract_images=True)\n```\nYou can optionally specify `extract_images_prefix`, `extract_images_format`, and `extract_images_to_dir`. See `example.ipynb` for more details, including preview in Jupyter Notebooks.\n\n## Custom result types\n\nWebBotParser out of the box only provides support for some search engines and result types. Even these parsers might stop working if the search engine providers decide to change their layout. However, WebBotParser can still be used in these cases by defining a custom `result_selector`, `queries`, and optionally a `metadata_extractor` function. In this case, a WebBotParser is initiated with these instead of with the `engine` attribute\n```python\nparser = WebBotParser(queries, result_selector, metadata_extractor)\n```\n\nUnder the hood, WebBotParser uses [BeautifulSoup](https://beautiful-soup-4.readthedocs.io/en/latest/index.html) to\n\n1. Parse the search result page's HTML via LXML\n2. Disciminate the individual results on each page using a [CSS selector](https://beautiful-soup-4.readthedocs.io/en/latest/index.html#css-selectors) called `result_selector` that matches a list of search results\n3. For each of those results, extract available information through a list of queries\n\nSee the below example for available types of queries and their usage\n```python\nqueries = [\n    # extract the text from inside a matched element, getting all the text over all its children\n    {'name': 'abc', 'type': 'text', 'selector': 'h3'},\n    \n    # extract the value of an attribute of a matched element\n    {'name': 'def', 'type': 'attribute', 'selector': 'a', 'attribute': 'href'},\n    \n    # whether or not a CSS selector matches, returns a Boolean\n    {'name': 'ghi', 'type': 'exists', 'selector': 'ul'},\n\n    # extract inline images and name them by a title\n    {'name': 'jkl', 'type': 'image', 'selector': 'g-img \u003e img', 'title_selector': 'h3'}\n    \n    # pass a custom query function\n    {'name': 'mno', 'type': 'custom', 'function': my_function},\n]\n```\n\nYou can optionally provide a `metadata_extractor(soup, file)` function to extract metadata alongside the search results, or import one of the existing extractors, e.g. with\n```python\nfrom webbotparser import GoogleParser\nmetadata_extractor = GoogleParser.google_metadata\n```\n\n## Related projects\n\n* [WebBot](https://github.com/gesiscss/WebBot) is a browser extension for retrieving search engine results.\n* [webbotparseR](https://github.com/schochastics/webbotparseR) is a similar R package for parsing WebBot results.\n* [WebBot tutorials](https://github.com/gesiscss/WebBot-tutorials) contains tutorials of WebBot and parsing.\n* [WebSearcher](https://github.com/gitronald/WebSearcher) is a Python package that facilitates obtaining and parsing search results from Google text search. Compared to WebBotParser, it supports parsing more diverse results (ads, knowledge boxes, etc.), but only Google text results (for now).\n\n## Authors\n\n[Georg Ahnert](https://github.com/wanLo), [Jun Sun](https://github.com/yfiua)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgesiscss%2Fwebbotparser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgesiscss%2Fwebbotparser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgesiscss%2Fwebbotparser/lists"}