{"id":20619721,"url":"https://github.com/twtrubiks/google-play-store-spider-selenium","last_synced_at":"2025-04-15T11:55:18.427Z","repository":{"id":84518957,"uuid":"83952084","full_name":"twtrubiks/Google-Play-Store-spider-selenium","owner":"twtrubiks","description":"Google-Play-Store-spider use Selenium +Beautiful Soup on Python ","archived":false,"fork":false,"pushed_at":"2017-03-05T10:37:04.000Z","size":5826,"stargazers_count":8,"open_issues_count":0,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-15T11:55:12.621Z","etag":null,"topics":["beautifulsoup","chrome","crawler","firefox","python","selenium","spider","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/twtrubiks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-05T07:11:18.000Z","updated_at":"2024-07-03T15:01:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"a234765e-a491-485f-b446-bb6d5af000ef","html_url":"https://github.com/twtrubiks/Google-Play-Store-spider-selenium","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twtrubiks%2FGoogle-Play-Store-spider-selenium","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twtrubiks%2FGoogle-Play-Store-spider-selenium/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twtrubiks%2FGoogle-Play-Store-spider-selenium/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twtrubiks%2FGoogle-Play-Store-spider-selenium/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/twtrubiks","download_url":"https://codeload.github.com/twtrubiks/Google-Play-Store-spider-selenium/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249067775,"owners_count":21207395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautifulsoup","chrome","crawler","firefox","python","selenium","spider","sqlite"],"created_at":"2024-11-16T12:12:21.906Z","updated_at":"2025-04-15T11:55:18.421Z","avatar_url":"https://github.com/twtrubiks.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Google-Play-Store-spider-selenium\n抓取 [Google Play Store](https://play.google.com/store/apps/top) 資料 use [Selenium](http://selenium-python.readthedocs.io/index.html) on Python 📝  \n\n並使用 SQLite 儲存 DB\n\n* [Youtube Demo](https://youtu.be/bNTj-CtwX1w)\n \n這個專案和 [Google-Play-Store-spider-bs4-excel](https://github.com/twtrubiks/Google-Play-Store-spider-bs4-excel) 類似，但這專案是使用  [Selenium](http://selenium-python.readthedocs.io/index.html) 結合  [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)。\n\n之前我也介紹過 Selenium 的範例，可參考 [youtube-trends-spider](https://github.com/twtrubiks/youtube-trends-spider)，\n\n因為 [Selenium](http://selenium-python.readthedocs.io/index.html) 有更新加上之前是使用python 2.7，寫法上也有點不同，所以這次使用 python 3.4.3 重新簡單介紹。\n\n## 特色\n* 透過 [Selenium](http://selenium-python.readthedocs.io/index.html) + [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) 抓取  [Google Play Store topselling_new_free ](https://play.google.com/store/apps/category/BOOKS_AND_REFERENCE/collection/topselling_new_free)資料。\n* 使用 SQLITE 儲存資料。\n\n\n\n## 安裝套件 \n確定電腦有安裝 [Python](https://www.python.org/) 之後\n\nclone 我的簡單範例\n\n``` \ngit clone https://github.com/twtrubiks/Google-Play-Store-spider-selenium.git\n```\n\n接著請在  cmd (命令提示字元) 輸入以下指令\n``` \npip install -r requirements.txt\n```\n\n## 使用 Selenium\n建議看一下 [Selenium](http://selenium-python.readthedocs.io/index.html) 官方說明。\n\n首先，必須安裝 [Selenium drivers](http://selenium-python.readthedocs.io/installation.html#drivers) ，請注意 \u003cb\u003e作業系統\u003c/b\u003e 、 \u003cb\u003e位元數\u003c/b\u003e 、 \u003cb\u003e瀏覽器\u003c/b\u003e。\n\n範例是使用 Firefox，需要額外將 [geckodriver.exe](https://github.com/twtrubiks/Google-Play-Store-spider-selenium/blob/master/geckodriver.exe) 這個 drivers 放入路徑底下，否則執行會出現錯誤。\n\n```\nprofile = webdriver.FirefoxProfile()\nprofile.accept_untrusted_certs = True\ndriver = webdriver.Firefox(firefox_profile=profile)\ndriver.get(targetURL)\n```\n\n如果修改成 Chrome，需要額外將 [chromedriver.exe](https://github.com/twtrubiks/Google-Play-Store-spider-selenium/blob/master/chromedriver.exe) 這個 drivers 放入路徑底下，否則執行會出現錯誤。\n\n```\noptions = webdriver.ChromeOptions()\noptions.add_argument('--ignore-certificate-errors')\ndriver = webdriver.Chrome(chrome_options=options)\ndriver.get(targetURL)\n```\n\nIE的部分我一直沒有測試成功，所以這裡我們暫時跳過。\n  \n以上方法是參考  [how-to-deal-with-certificates-using-selenium](http://stackoverflow.com/questions/24507078/how-to-deal-with-certificates-using-selenium)\n\n## 使用方法 以及 執行畫面\n\n``` \npython app.py\n```\n執行畫面\n\n![alt tag](http://i.imgur.com/frCEqu7.jpg)\n\n在執行時，背景會跳出一個瀏覽器，\u003cb\u003e請不要去亂點他(或關閉他)，抓完資料瀏覽器會自動關閉\u003c/b\u003e\n\n![alt tag](http://i.imgur.com/tLXt0zM.jpg)\n\n![alt tag](http://i.imgur.com/x9Tuyf8.jpg)\n\n![alt tag](http://i.imgur.com/bhJJVXl.jpg)\n\n\n執行完畢後，會將資料存在 app.db 裡，可以使用 [SQLiteBrowser](http://sqlitebrowser.org/) 觀看\n\n![alt tag](http://i.imgur.com/PYOkVNN.jpg)\n\n\n\n## 執行環境\n* Python 3.4.3\n* Windows 10\n\n## Reference \n* [Selenium](http://selenium-python.readthedocs.io/index.html)\n* [Flask-Migrate](https://flask-migrate.readthedocs.io/en/latest/) \n* [beautifulsoup4](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n* [requests](http://docs.python-requests.org/en/master/)\n\n\n## License\nMIT license\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwtrubiks%2Fgoogle-play-store-spider-selenium","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftwtrubiks%2Fgoogle-play-store-spider-selenium","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwtrubiks%2Fgoogle-play-store-spider-selenium/lists"}