{"id":18449901,"url":"https://github.com/alincode/learning-web-crawler","last_synced_at":"2026-01-24T04:02:05.361Z","repository":{"id":81411006,"uuid":"82536954","full_name":"alincode/learning-web-crawler","owner":"alincode","description":null,"archived":false,"fork":false,"pushed_at":"2021-02-25T00:52:40.000Z","size":1022,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-18T11:37:36.804Z","etag":null,"topics":["web-crawler"],"latest_commit_sha":null,"homepage":"https://alincode.github.io/learning-web-crawler/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alincode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-20T08:58:10.000Z","updated_at":"2021-02-25T00:53:24.000Z","dependencies_parsed_at":null,"dependency_job_id":"871cbe0f-0b9c-4481-bcbb-f0c93f6e48d3","html_url":"https://github.com/alincode/learning-web-crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/alincode/learning-web-crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alincode%2Flearning-web-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alincode%2Flearning-web-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alincode%2Flearning-web-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alincode%2Flearning-web-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alincode","download_url":"https://codeload.github.com/alincode/learning-web-crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alincode%2Flearning-web-crawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28711453,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-23T23:51:44.727Z","status":"online","status_checked_at":"2026-01-24T02:00:06.909Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["web-crawler"],"created_at":"2024-11-06T07:22:36.547Z","updated_at":"2026-01-24T04:02:05.330Z","avatar_url":"https://github.com/alincode.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# 簡介\n\n紀錄研究網路爬蟲時寫的筆記\n\n## 緒\n\n什麼是研究 Big Data 第一步要做的事？\n\n\u003e 大數據必須藉由計算機對資料進行統計、比對、解析方能得出客觀結果。 \u003e\u003e\u003e [Wiki](https://zh.wikipedia.org/wiki/%E5%A4%A7%E6%95%B8%E6%93%9A)\n\n所以我們得先要有資料，才有所謂的大數據分析，如果你分析的資料可由公司內部的資料庫取出，那非常幸運的你不需要操心這件事，但是往往我們沒這麼幸運，於是我們得辛苦的把這些資料爬回來。我們把這個辛苦的工兵叫[網路爬蟲(Crawler)](https://zh.wikipedia.org/wiki/%E7%B6%B2%E8%B7%AF%E8%9C%98%E8%9B%9B)。\n\n### 作者\n\n劉艾霖\n\n* 軟體技術顧問 / 遠端工作者 / 自由工作者\n* 技術社群講者 / 企業內訓講師 / 大專院校產學合作業師\n* 網站開發家教 (線上一對一教學，有需求者，請來信詢問)\n* alincode@gmail.com\n\n### 作者簡介\n\n是全端工程師、後端工程師、測試開發工程師、企業內訓講師，具有 7~8 年軟體開發相關資歷，專精於網站開發及協助技術導入與培訓。\n\n### 教學 / 講師經歷\n\n* 16/06/25 CTJS - 講者 light talk 淺談 WebdriverIO 測試框架\n* 16/07/20 Trunk studio - 講者 WebdriverIO 起手式\n* 16/08/24-25 Modern Web 2016 - 講者 無痛前端測試\n* 16/10/22-23 JSDC.tw 2016 - 講者 淺談網站自動化測試 - 以 WebdriverIO 框架為例\n* 16/12 正修科技大學 HTML5 TQC 輔導考照\n* 2017 啟德電子 - 企業內訓課程 Bootstrap\n\n### 勘誤通知\n\n如發現內容勘誤，歡迎利用這些管道和我們聯繫：\n\n* 使用 GitHub Issues 回報錯誤或發 pull request。\n* 寄信到 alincode@gmail.com\n\n### 分享的主題\n\n* 透過熟悉爬蟲框架，來瞭解爬蟲應該怎麼寫。\n* 爬蟲框架幫我們做了什麼事？還有有哪些選擇？\n* 如何使用這個爬蟲框架？\n* 一個簡單的範例\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falincode%2Flearning-web-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falincode%2Flearning-web-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falincode%2Flearning-web-crawler/lists"}