{"id":13825902,"url":"https://github.com/opsdisk/yagooglesearch","last_synced_at":"2025-04-04T11:11:30.304Z","repository":{"id":44877824,"uuid":"400291486","full_name":"opsdisk/yagooglesearch","owner":"opsdisk","description":"Yet another googlesearch - A Python library for executing intelligent, realistic-looking, and tunable Google searches.","archived":false,"fork":false,"pushed_at":"2024-04-07T13:07:42.000Z","size":182,"stargazers_count":273,"open_issues_count":2,"forks_count":49,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-28T10:06:33.399Z","etag":null,"topics":["google","googlesearch","python","search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opsdisk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-26T20:02:09.000Z","updated_at":"2025-03-27T15:59:49.000Z","dependencies_parsed_at":"2023-12-11T14:34:33.839Z","dependency_job_id":"2a410244-29eb-47c4-9a2e-03a8d37c2cc0","html_url":"https://github.com/opsdisk/yagooglesearch","commit_stats":{"total_commits":32,"total_committers":3,"mean_commits":"10.666666666666666","dds":0.1875,"last_synced_commit":"d1c79356a205cb46e8938d9631b5f997cdaccf95"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opsdisk%2Fyagooglesearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opsdisk%2Fyagooglesearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opsdisk%2Fyagooglesearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opsdisk%2Fyagooglesearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opsdisk","download_url":"https://codeload.github.com/opsdisk/yagooglesearch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247166168,"owners_count":20894654,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google","googlesearch","python","search"],"created_at":"2024-08-04T09:01:28.874Z","updated_at":"2025-04-04T11:11:30.284Z","avatar_url":"https://github.com/opsdisk.png","language":"Python","readme":"# yagooglesearch - Yet another googlesearch\n\n## Overview\n\n`yagooglesearch` is a Python library for executing intelligent, realistic-looking, and tunable Google searches.  It\nsimulates real human Google search behavior to prevent rate limiting by Google (the dreaded HTTP 429 response), and if\nHTTP 429 blocked by Google, logic to back off and continue trying.  The library does not use the Google API and is\nheavily based off the [googlesearch](https://github.com/MarioVilas/googlesearch) library.  The features include:\n\n* Tunable search client attributes mid searching\n* Returning a list of URLs instead of a generator\n* HTTP 429 / rate-limit detection (Google is blocking your IP for making too many search requests) and recovery\n* Randomizing delay times between retrieving paged search results (i.e., clicking on page 2 for more results)\n* HTTP(S) and SOCKS5 proxy support\n* Leveraging `requests` library for HTTP requests and cookie management\n* Adds \"\u0026filter=0\" by default to search URLs to prevent any omission or filtering of search results by Google\n* Console and file logging\n* Python 3.6+\n\n## Terms and Conditions\n\nThis code is supplied as-is and you are fully responsible for how it is used.  Scraping Google Search results may\nviolate their [Terms of Service](https://policies.google.com/terms).  Another Python Google search library had some\ninteresting information/discussion on it:\n\n* [Original issue](https://github.com/aviaryan/python-gsearch/issues/1)\n* [A response](https://github.com/aviaryan/python-gsearch/issues/1#issuecomment-365581431\u003e)\n* Author created a separate [Terms and Conditions](https://github.com/aviaryan/python-gsearch/blob/master/T_AND_C.md)\n* ...that contained link to this [blog](https://benbernardblog.com/web-scraping-and-crawling-are-perfectly-legal-right/)\n\nGoogle's preferred method is to use their [API](https://developers.google.com/custom-search/v1/overview).\n\n## Installation\n\n## pip\n\n```bash\npip install yagooglesearch\n```\n\n## pyproject.toml\n\n```bash\ngit clone https://github.com/opsdisk/yagooglesearch\ncd yagooglesearch\nvirtualenv -p python3 .venv  # If using a virtual environment.\nsource .venv/bin/activate  # If using a virtual environment.\npip install .  # Reads from pyproject.toml\n```\n\n## Usage\n\n```python\nimport yagooglesearch\n\nquery = \"site:github.com\"\n\nclient = yagooglesearch.SearchClient(\n    query,\n    tbs=\"li:1\",\n    max_search_result_urls_to_return=100,\n    http_429_cool_off_time_in_minutes=45,\n    http_429_cool_off_factor=1.5,\n    # proxy=\"socks5h://127.0.0.1:9050\",\n    verbosity=5,\n    verbose_output=True,  # False (only URLs) or True (rank, title, description, and URL)\n)\nclient.assign_random_user_agent()\n\nurls = client.search()\n\nlen(urls)\n\nfor url in urls:\n    print(url)\n```\n\n## Max ~400 results returned\n\nEven though searching Google through the GUI will display a message like \"About 13,000,000 results\", that does not mean\n`yagooglesearch` will find anything close to that.  Testing shows that at most, about 400 results are returned.  If you\nset 400 \u003c `max_search_result_urls_to_return`, a warning message will be printed to the logs.  See\n\u003chttps://github.com/opsdisk/yagooglesearch/issues/28\u003e for the discussion.\n\n## Google is blocking me!\n\nLow and slow is the strategy when executing Google searches using `yagooglesearch`.  If you start getting HTTP 429\nresponses, Google has rightfully detected you as a bot and will block your IP for a set period of time. `yagooglesearch`\nis not able to bypass CAPTCHA, but you can do this manually by performing a Google search from a browser and proving you\nare a human.\n\nThe criteria and thresholds to getting blocked is unknown, but in general, randomizing the user agent, waiting enough\ntime between paged search results (7-17 seconds), and waiting enough time between different Google searches (30-60\nseconds) should suffice.  Your mileage will definitely vary though.  Using this library with Tor will likely get you\nblocked quickly.\n\n## HTTP 429 detection and recovery (optional)\n\nIf `yagooglesearch` detects an HTTP 429 response from Google, it will sleep for `http_429_cool_off_time_in_minutes`\nminutes and then try again.  Each time an HTTP 429 is detected, it increases the wait time by a factor of\n`http_429_cool_off_factor`.\n\nThe goal is to have `yagooglesearch` worry about HTTP 429 detection and recovery and not put the burden on the script\nusing it.\n\nIf you do not want `yagooglesearch` to handle HTTP 429s and would rather handle it yourself, pass\n`yagooglesearch_manages_http_429s=False` when instantiating the yagooglesearch object.  If an HTTP 429 is detected, the\nstring \"HTTP_429_DETECTED\" is added to a list object that will be returned, and it's up to you on what the next step\nshould be.  The list object will contain any URLs found before the HTTP 429 was detected.\n\n```python\nimport yagooglesearch\n\nquery = \"site:twitter.com\"\n\nclient = yagooglesearch.SearchClient(\n    query,\n    tbs=\"li:1\",\n    verbosity=4,\n    num=10,\n    max_search_result_urls_to_return=1000,\n    minimum_delay_between_paged_results_in_seconds=1,\n    yagooglesearch_manages_http_429s=False,  # Add to manage HTTP 429s.\n)\nclient.assign_random_user_agent()\n\nurls = client.search()\n\nif \"HTTP_429_DETECTED\" in urls:\n    print(\"HTTP 429 detected...it's up to you to modify your search.\")\n\n    # Remove HTTP_429_DETECTED from list.\n    urls.remove(\"HTTP_429_DETECTED\")\n\n    print(\"URLs found before HTTP 429 detected...\")\n\n    for url in urls:\n        print(url)\n```\n\n![http429_detection_string_in_returned_list.png](img/http429_detection_string_in_returned_list.png)\n\n## HTTP and SOCKS5 proxy support\n\n`yagooglesearch` supports the use of a proxy.  The provided proxy is used for the entire life cycle of the search to\nmake it look more human, instead of rotating through various proxies for different portions of the search.  The general\nsearch life cycle is:\n\n1) Simulated \"browsing\" to `google.com`\n2) Executing the search and retrieving the first page of results\n3) Simulated clicking through the remaining paged (page 2, page 3, etc.) search results\n\nTo use a proxy, provide a proxy string when initializing a `yagooglesearch.SearchClient` object:\n\n```python\nclient = yagooglesearch.SearchClient(\n    \"site:github.com\",\n    proxy=\"socks5h://127.0.0.1:9050\",\n)\n```\n\nSupported proxy schemes are based off those supported in the Python `requests` library\n(\u003chttps://docs.python-requests.org/en/master/user/advanced/#proxies\u003e):\n\n* `http`\n* `https`\n* `socks5` - \"causes the DNS resolution to happen on the client, rather than on the proxy server.\"  You likely **do\n  not** want this since all DNS lookups would source from where `yagooglesearch` is being run instead of the proxy.\n* `socks5h` - \"If you want to resolve the domains on the proxy server, use socks5h as the scheme.\"  This is the **best**\n  option if you are using SOCKS because the DNS lookup and Google search is sourced from the proxy IP address.\n\n## HTTPS proxies and SSL/TLS certificates\n\nIf you are using a self-signed certificate for an HTTPS proxy, you will likely need to disable SSL/TLS verification when\neither:\n\n1) Instantiating the `yagooglesearch.SearchClient` object:\n\n```python\nimport yagooglesearch\n\nquery = \"site:github.com\"\n\nclient = yagooglesearch.SearchClient(\n    query,\n    proxy=\"http://127.0.0.1:8080\",\n    verify_ssl=False,\n    verbosity=5,\n)\n```\n\n2) or after instantiation:\n\n```python\nquery = \"site:github.com\"\n\nclient = yagooglesearch.SearchClient(\n    query,\n    proxy=\"http://127.0.0.1:8080\",\n    verbosity=5,\n)\n\nclient.verify_ssl = False\n```\n\n## Multiple proxies\n\nIf you want to use multiple proxies, that burden is on the script utilizing the `yagooglesearch` library to instantiate\na new `yagooglesearch.SearchClient` object with the different proxy. Below is an example of looping through a list of\nproxies:\n\n```python\nimport yagooglesearch\n\nproxies = [\n    \"socks5h://127.0.0.1:9050\",\n    \"socks5h://127.0.0.1:9051\",\n    \"http://127.0.0.1:9052\",  # HTTPS proxy with a self-signed SSL/TLS certificate.\n]\n\nsearch_queries = [\n    \"python\",\n    \"site:github.com pagodo\",\n    \"peanut butter toast\",\n    \"are dragons real?\",\n    \"ssh tunneling\",\n]\n\nproxy_rotation_index = 0\n\nfor search_query in search_queries:\n\n    # Rotate through the list of proxies using modulus to ensure the index is in the proxies list.\n    proxy_index = proxy_rotation_index % len(proxies)\n\n    client = yagooglesearch.SearchClient(\n        search_query,\n        proxy=proxies[proxy_index],\n    )\n\n    # Only disable SSL/TLS verification for the HTTPS proxy using a self-signed certificate.\n    if proxies[proxy_index].startswith(\"http://\"):\n        client.verify_ssl = False\n\n    urls_list = client.search()\n\n    print(urls_list)\n\n    proxy_rotation_index += 1\n```\n\n## GOOGLE_ABUSE_EXEMPTION cookie\n\nIf you have a `GOOGLE_ABUSE_EXEMPTION` cookie value, it can be passed into `google_exemption` when instantiating the\n`SearchClient` object.\n\n## \u0026tbs= URL filter clarification\n\nThe `\u0026tbs=` parameter is used to specify either verbatim or time-based filters.\n\n### Verbatim search\n\n```none\n\u0026tbs=li:1\n```\n\n![verbatim.png](./img/verbatim.png)\n\n### time-based filters\n\n| Time filter | \u0026tbs= URL parameter                   | Notes                                 |\n| ----------- | ------------------------------------- | ------------------------------------- |\n| Past hour   | qdr:h                                 |                                       |\n| Past day    | qdr:d                                 | Past 24 hours                         |\n| Past week   | qdr:w                                 |                                       |\n| Past month  | qdr:m                                 |                                       |\n| Past year   | qdr:y                                 |                                       |\n| Custom      | cdr:1,cd_min:1/1/2021,cd_max:6/1/2021 | See yagooglesearch.get_tbs() function |\n\n![time_filters.png](./img/time_filters.png)\n\n## Limitations\n\nCurrently, the `.filter_search_result_urls()` function will remove any url with the word \"google\" in it.  This is to\nprevent the returned search URLs from being polluted with Google URLs.  Note this if you are trying to explicitly search\nfor results that may have \"google\" in the URL, such as `site:google.com computer`\n\n## License\n\nDistributed under the BSD 3-Clause License. See [LICENSE](./LICENSE) for more information.\n\n## Contact\n\n[@opsdisk](https://twitter.com/opsdisk)\n\nProject Link: [https://github.com/opsdisk/yagooglesearch](https://github.com/opsdisk/yagooglesearch)\n\n## Acknowledgements\n\n* [Mario Vilas](https://github.com/MarioVilas) for his amazing work on the original\n  [googlesearch](https://github.com/MarioVilas/googlesearch) library.\n\n## Contributors\n\n* [KennBro](https://github.com/KennBro) - \u003chttps://github.com/opsdisk/yagooglesearch/pull/9\u003e\n* [ArshansGithub](https://github.com/ArshansGithub) - \u003chttps://github.com/opsdisk/yagooglesearch/pull/21\u003e\n* [pguridi](https://github.com/pguridi) - \u003chttps://github.com/opsdisk/yagooglesearch/pull/38\u003e\n* [libenc](https://github.com/libenc) - \u003chttps://github.com/opsdisk/yagooglesearch/pull/42\u003e\n","funding_links":[],"categories":["Python","[](#table-of-contents) Table of contents"],"sub_categories":["[](#filesharing-search-engines)Filesharing Search Engines"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopsdisk%2Fyagooglesearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopsdisk%2Fyagooglesearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopsdisk%2Fyagooglesearch/lists"}