{"id":21077702,"url":"https://github.com/howie6879/magic_google","last_synced_at":"2025-12-14T01:02:17.881Z","repository":{"id":15760309,"uuid":"78717595","full_name":"howie6879/magic_google","owner":"howie6879","description":"Google search results crawler, get google search results that you need","archived":false,"fork":false,"pushed_at":"2023-11-14T10:11:26.000Z","size":40,"stargazers_count":403,"open_issues_count":3,"forks_count":110,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-03-29T08:06:11.272Z","etag":null,"topics":["crawler","google","google-search","spider"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/howie6879.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-01-12T06:55:21.000Z","updated_at":"2025-03-28T07:29:30.000Z","dependencies_parsed_at":"2024-06-19T00:27:15.783Z","dependency_job_id":"2f7d508e-c831-4648-a08b-94856dc4280d","html_url":"https://github.com/howie6879/magic_google","commit_stats":{"total_commits":38,"total_committers":4,"mean_commits":9.5,"dds":0.1842105263157895,"last_synced_commit":"dbfbfef897754520113e893318c1882c5d873b3b"},"previous_names":["howie6879/magicgoogle"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/howie6879%2Fmagic_google","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/howie6879%2Fmagic_google/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/howie6879%2Fmagic_google/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/howie6879%2Fmagic_google/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/howie6879","download_url":"https://codeload.github.com/howie6879/magic_google/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247312077,"owners_count":20918344,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","google","google-search","spider"],"created_at":"2024-11-19T19:37:59.011Z","updated_at":"2025-12-14T01:02:12.836Z","avatar_url":"https://github.com/howie6879.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## magic_google\n\n[![](https://img.shields.io/pypi/v/magic_google.svg)](https://pypi.org/project/magic-google/)\n\n### 1.What's magic_google\nThis is an easy Google Searching crawler that you can get anything you want in the page by using it.\n\nDuring the process of  crawling,you need to pay attention to the limitation from google towards ip address and the warning of exception , so I suggest that you should pause running the program and own the Proxy ip\n\nphp - [MagicGoogle](https://github.com/howie6879/php-google)\n\n### 2.How to Use?\nRun\n``` shell\npip install magic_google\n# Or\npip install git+https://github.com/howie6879/magic_google.git\n# Or\ngit clone https://github.com/howie6879/magic_google.git\ncd magic_google\nvim google_search.py\n# Or \npython setup.py install\n```\nExample\n``` python\nfrom magic_google import MagicGoogle\nimport pprint\n\n# Or PROXIES = None\nPROXIES = [{\n    'http': 'http://192.168.2.207:1080',\n    'https': 'http://192.168.2.207:1080'\n}]\n\n# Or MagicGoogle()\nmg = MagicGoogle(PROXIES)\n\n#  Crawling the whole page\nresult = mg.search_page(query='python')\n\n# Crawling url\nfor url in mg.search_url(query='python'):\n    pprint.pprint(url)\n    \n# Output\n# 'https://www.python.org/'\n# 'https://www.python.org/downloads/'\n# 'https://www.python.org/about/gettingstarted/'\n# 'https://docs.python.org/2/tutorial/'\n# 'https://docs.python.org/'\n# 'https://en.wikipedia.org/wiki/Python_(programming_language)'\n# 'https://www.codecademy.com/courses/introduction-to-python-6WeG3/0?curriculum_id=4f89dab3d788890003000096'\n# 'https://www.codecademy.com/learn/python'\n# 'https://developers.google.com/edu/python/'\n# 'https://learnpythonthehardway.org/book/'\n# 'https://www.continuum.io/downloads'\n\n# Get {'title','url','text'}\nfor i in mg.search(query='python', num=1):\n    pprint.pprint(i)\n    \n# Output\n# {'text': 'The official home of the Python Programming Language.',\n# 'title': 'Welcome to Python .org',\n# 'url': 'https://www.python.org/'}\n```\nYou can see [google_search.py](./examples/google_search.py)\n\n**If  you need a big amount of querie but only having an ip address,I suggest  you can have a time lapse between 5s ~ 30s.**\n\nThe reason that it always return empty might be as follows:\n\n```html\n\u003cHTML\u003e\u003cHEAD\u003e\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003cTITLE\u003e302 Moved\u003c/TITLE\u003e\u003c/HEAD\u003e\u003cBODY\u003e\n\u003cH1\u003e302 Moved\u003c/H1\u003e\nThe document has moved\n\u003cA HREF=\"https://ipv4.google.com/sorry/index?continue=https://www.google.me/s****\"\u003ehere\u003c/A\u003e.\n\u003c/BODY\u003e\u003c/HTML\u003e\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhowie6879%2Fmagic_google","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhowie6879%2Fmagic_google","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhowie6879%2Fmagic_google/lists"}